Benchmarking Large Language Models for News Summarization
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2024.tacl-1.3.pdf
Summary
The paper investigates why large language models succeed at news summarization and whether their output matches human writing. The authors ran a human evaluation of ten LLMs that differ in pretraining approach, scale, and prompting setup. They sampled 100 articles each from CNN/DM and XSUM and had annotators rate faithfulness, coherence, and relevance. The main finding is that instruction tuning, not parameter count, drives zero-shot summarization quality. They also found that the benchmark reference summaries were repurposed from website content and are often rated below model outputs. These weak references distort automatic metrics and hurt systems trained or prompted with them. To get a cleaner comparison, the team hired Upwork freelance writers to produce new summaries. The strongest model, Instruct Davinci, copies more and paraphrases less than the writers, yet raters judged it roughly equal overall. The work questions reference-based evaluation on XSUM and suggests single-document news summarization benchmarks may be reaching their limits.
Key points
- Models tested: ten LLMs, including GPT-3 (davinci/curie/ada), InstructGPT variants, OPT 175B, GLM 130B, Cohere xlarge, and Anthropic-LM v4-s3.
- Prompting settings: all models were run five-shot. Zero-shot was tested only for three GPT-3 and three instruction-tuned GPT-3 models.
- Effect of instruction tuning: a 350M-parameter instruction-tuned GPT-3 performed comparably to the 175B base GPT-3.
- Rating scheme and agreement: coherence and relevance used a 1–5 Likert scale, while faithfulness was binary. Pairwise annotator agreement stood at 75% on faithfulness, while coherence reached 81% and relevance 86%.
- Rouge-L and reference quality: with the original XSUM references, Rouge-L correlated negatively with human faithfulness ratings. With the freelance references, the correlation turned clearly positive.
- Released resources: freelance-written summaries and evaluation data covering 18 model settings on two datasets, available on GitHub.
Cited by
- Leveraging Digitized Newspapers to Collect Summarization Data in Low-Resource Languages
- Towards Measuring the Representation of Subjective Global Opinions in Language Models
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
- TrustLLM: Trustworthiness in Large Language Models
- Challenges and Applications of Large Language Models
- Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
- MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents
- Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
- A Survey of Large Language Models
Cite
@article{zhang2024benchmarking,
title = {{Benchmarking Large Language Models for News Summarization}},
author = {Zhang, Tianyi and Ladhak, Faisal and Durmus, Esin and Liang, Percy and McKeown, Kathleen and Hashimoto, Tatsunori B.},
journal = {Transactions of the Association for Computational Linguistics},
year = {2024},
doi = {10.1162/tacl\_a\_00632},
url = {https://doi.org/10.1162/tacl_a_00632}
}References
- OpenAI 175B 6.7B 350M ✗ Brown et al. (2020) find
- InstructGPT davinci v2 InstructGPT curie v1 InstructGPT ada v1 OpenAI 175B 6.7B 350M ✓ Ouyang et al. (2022) find
- OPT 175B Meta 175B ✗ Zhang et al. (2022) find
- GLM Tsinghua University 130B ✗ Du et al. (2021) find
- Cohere xlarge v20220609 Cohere 52.4B ✗ Cohere (2022) find
- Anthropic-LM v4-s3 Anthropic 52B ✓ Bai et al. (2022) find
- GPT-3 davinci v1 GPT-3 curie v1 GPT-3 ada v1 *Table 1: List of large language models we benchmarked with human evaluation.* find