Elamur Research⌕
☰
home / papers / 2023 / Human-like Summarization Evaluation with ChatGPT

Human-like Summarization Evaluation with ChatGPT

Paper · 2023 · Mingqi Gao, Jie Ruan, Renliang Sun et al.

Summary

Automatic metrics for text summarization, such as ROUGE and embedding-based scores, often disagree with human judgment and offer little interpretability. Human evaluation is more reliable but slow and costly. The authors test whether ChatGPT (gpt-3.5-turbo-0301, temperature 0) can stand in for human annotators. Their prompts closely follow the original annotation instructions for four protocols:

  • Likert-scale scoring on SummEval and Newsroom
  • pairwise comparison on TLDR
  • Pyramid-style SCU checking on REALSumm
  • binary sentence-level factuality on QAGS

They compare its outputs with human labels and with ROUGE, BERTScore, MoverScore, BARTScore variants, FactCC and DAE.

ChatGPT handled every protocol with very few failed responses. On SummEval it correlated with humans better than all baseline metrics. On Newsroom, only two BARTScore variants beat it. In the other three protocols its advantage was modest, except on QAGS_XSUM. Results depended strongly on prompt wording. Evaluation was also far cheaper than paying annotators.

This matters because LLM-based evaluation is flexible, reproducible and inexpensive. It can also produce explanations alongside its scores.

Key points

  • On SummEval, ChatGPT's sample-level Spearman correlations were 0.435 for consistency, 0.433 for relevance, 0.419 for fluency and 0.561 for coherence. Its system-level correlation for relevance reached 0.901.
  • Without dimension definitions in the prompt, ChatGPT's fluency and coherence scores were almost identical (correlation 0.960). Adding definitions lowered this to 0.843, meaning the model separated the two dimensions better.
  • Invalid outputs, such as refusals or rewriting the summary, stayed at or below about 1%. The highest rate (0.0106) came with the prompt that added both definitions and step instructions.
  • Scoring all of SummEval cost roughly 3.2 USD, compared with an estimated 60 USD for a human annotator. The authors put human evaluation at about 10–20 times the cost.
  • The explanations ChatGPT gave generally matched its scores. However, the authors note that whether those explanations are factually correct still needs to be verified.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China315 works · 43,268 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksSummarization · 243 works

Cite

Cite as Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang and Xiaojun Wan. 2023. Human-like Summarization Evaluation with ChatGPT. arXiv:2304.02554.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.02554 · arXiv 2304.02554

@misc{gao2023human,
  title = {{Human-like Summarization Evaluation with ChatGPT}},
  author = {Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang and Xiaojun Wan},
  year = {2023},
  doi = {10.48550/arxiv.2304.02554},
  eprint = {2304.02554},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.02554}
}

References

21 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anya Belz, Anastasia Shimorina, Shubham Agarwal, and Ehud Reiter. 2021. The ReproGen shared task on reproducibility of human evaluations in NLG: Overview and results. In Proceedings of the 14th International Conference on Natural Language Generation, pages 249–258, Aberdeen, Scotland, UK. Associatio… find
  2. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359, Online. Association for Computational Linguist…
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  4. Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055– 5070, Online. Association for Computational Lingu…
  5. Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. find
  6. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pa…
  7. Yunjie Ji, Yan Gong, Yiping Peng, Chao Ni, Peiyan Sun, Dongyu Pan, Baochang Ma, and Xiangang Li. 2023. Exploring chatgpt’s ability to rank content: A preliminary study on consistency with human preferences. Computing Research Repository, arXiv:2303.07610. — [arXiv:2303.07610](https://arxiv.org/abs/2303.07610) find
  8. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. Computing Research Repository, arXiv:2302.14520. find
  9. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346, Online. Association for Computational…
  10. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  11. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. Gpteval: Nlg evaluation using gpt-4 with better human alignment.
  12. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for abstractive text summarization. Computing Research Repository, arXiv:2303.15621.
  13. Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145–152, Boston, Massachusetts… find
  14. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  15. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022. Learning to summarize from human feedback. Computing Research Repository, arXiv:2009.01325. Version 3. find
  16. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Computational Linguistics. find
  17. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? a preliminary study. Computing Research Repository, arXiv:2303.04048.
  18. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Computing Research Repository, arXiv:2106.11520. Version 2.
  19. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Computing Research Repository, arXiv:1904.09675. Version 3. find
  20. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2023. Benchmarking large language models for news summarization. Computing Research Repository, arXiv:2301.13848.
  21. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gao-2023-human-like-summarization-evaluation-chatgpt/