中文大模型生成式摘要能力评估

Summary

This study asks how well open-source Chinese large language models handle abstractive summarization of Chinese text. Chinese summaries are hard for machines because the language is dense and summaries often restructure the original sentences. The authors tested three small, widely downloaded models: ChatGLM2-6B, Baichuan2-7B and InternLM-7B. They used two standard benchmarks, NLPCC2017 Task3 and LCSTS, in both zero-shot and few-shot settings.

For zero-shot runs, they tried several Chinese prompts on a data subset, scored the repeated outputs with ROUGE, and kept the best prompt per model. For few-shot runs, they supplied article–summary example pairs in context. Evaluation combined automatic metrics with manual comparison, and the few-shot runs were also examined for output length and inference time.

ChatGLM2-6B and Baichuan2-7B produced fluent, detailed zero-shot summaries that were not concise enough. Few-shot prompting made outputs shorter but caused key content to be dropped. All models showed repetition loops, hallucinations and factual inconsistencies. These results give a reference point for future work on Chinese summarization with open LLMs.

Key points

• Models: ChatGLM2-6B (about 6.24B parameters, pretrained on 1.4T tokens, with RLHF), Baichuan2-7B (about 7.51B parameters, 2.6T tokens) and InternLM-7B (about 7.32B parameters, 8K context window).
• Datasets: NLPCC2017 Task3 contains 50,000 Toutiao news articles, ranging from about 100 to over 20,000 characters, with reference summaries usually under 60 characters and quality labels from 0 to 5. LCSTS is the second benchmark.
• Zero-shot prompt selection: 4 candidate prompts were used for NLPCC2017 and 5 for LCSTS. Each prompt was run repeatedly on the first 100 samples, and the prompt with the highest average ROUGE was then applied to the full dataset.
• Prompt sensitivity: prompts that look almost equivalent still change the resulting summaries, and the models differ in what content they emphasize.
• Few-shot setup: in-context demonstrations were given through a Streamlit chat interface. The resulting summaries were more concise but captured noticeably less key information than zero-shot outputs.