Elamur Research⌕
☰
home / papers / 2024 / Large Language Models are Inconsistent and Biased Evaluators

Large Language Models are Inconsistent and Biased Evaluators

Paper · 2024 · Rickard Stureborg, Dimitris Alikaniotis and Yoshi Suhara

Summary

LLM-based judges such as GPT-4 are now widely used to score generated text without references. However, earlier work mostly tuned them to match human ratings and paid little attention to how reliable they are. The authors examine two properties of these judges, bias and consistency, in summary evaluation. They use SummEval as a development set and RoSE as a held-out test set. Across more than 560,000 generated outputs from GPT-3.5 and GPT-4, they find three biases. The models favour low-perplexity text (familiarity bias), they distribute scores unevenly and favour round values, and their judgments on one attribute are anchored by ratings they have already given for other attributes. The judges are also unstable: repeated samples agree with each other less than human annotators do, and the models react to prompt changes that a human would consider irrelevant. From these findings the authors derive a set of configuration recipes. Their resulting evaluator significantly beats the previous state of the art on part of RoSE. The work highlights hidden risks of relying on LLM-as-a-judge metrics.

Key points

  • Data and setup:
    • SummEval provides expert scores for 16 system summaries of each of 100 CNN/Daily Mail articles. It covers Coherence, Consistency, Fluency and Relevance, and agreement with experts is measured with Kendall's τ.
    • RoSE spans CNN/Daily Mail, SAMSum and XSum, with recall-style annotations based on Atomic Content Units (ACU).
  • Models:
    • The analysis uses gpt-3.5-turbo-0301 and gpt-4-0613.
    • The final RoSE comparison runs on GPT-4-Turbo against the authors' own re-implementation of G-Eval.
    • Perplexity is computed with text-davinci-003.
  • Familiarity bias: Mean summary perplexity for GPT-4 Coherence ratings falls sharply as scores rise, from about 8.15 at score 2 to 5.34 at score 5. Expert ratings span a much narrower range (7.03 to 6.36). GPT-4's ratings also correlate negatively with source-document perplexity, while expert ratings show no such link.
  • Score granularity: The authors compare several rating formats: 1–5, 1–5 with "strong/weak" word modifiers, GPA-style float values, 1–10, and 1–100. They also test averaging over multiple samples. LLMs often fail to use these finer scales as instructed.
  • Result and limitations:
    • The recipe-based evaluator significantly outperforms the SOTA on the CNNDM partition of RoSE at 90% confidence. Gains on the other two domains are positive but not significant.
    • Only proprietary OpenAI models were tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Duke UniversityDurham · United States18 works · 14,944 citations🏢NvidiaSanta Clara · United States16 works · 4,820 citations🏢Nvidia (United Kingdom)Reading · United Kingdom13 works · 4,550 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,343 works

Cite

Cite as Rickard Stureborg, Dimitris Alikaniotis and Yoshi Suhara. 2024. Large Language Models are Inconsistent and Biased Evaluators. arXiv:2405.01724.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.01724 · arXiv 2405.01724

@misc{stureborg2024large,
  title = {{Large Language Models are Inconsistent and Biased Evaluators}},
  author = {Rickard Stureborg, Dimitris Alikaniotis and Yoshi Suhara},
  year = {2024},
  doi = {10.48550/arxiv.2405.01724},
  eprint = {2405.01724},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.01724}
}

References

45 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. find
  2. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Ass…
  3. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359, Online. Association for Computational Linguist…
  4. Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt. find
  5. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization. Association for Computational Linguistics. find
  6. Hidehito Honda, Rina Kagawa, and Masaru Shirasuna. 2022. On the round number bias and wisdom of crowds in different response formats for numerical estimation. Scientific Reports, 12(1):8167. find
  7. Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, and Tiejun Zhao. 2024. An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task-specific classifiers. find
  8. Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. find
  9. Zdeněk Kasner and Ondřej Dušek. 2024. Beyond reference-based metrics: Analyzing behaviors of open llms on data-to-text generation. find
  10. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. find
  11. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. find
  12. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2023. Benchmarking cognitive biases in large language models as evaluators. find
  13. Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, and Chongyang Tao. 2024. Leveraging large language models for nlg evaluation: A survey. find
  14. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  15. Yen-Ting Lin and Yun-Nung Chen. 2023. Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models.
  16. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023a. G-eval: Nlg evaluation using gpt-4 with better human alignment. find
  17. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. G-eval: Nlg evaluation using gpt-4 with better human alignment. find
  18. Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. 2023c. LLMs as narcissistic evaluators: When ego inflates evaluation scores. find
  19. Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023d. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. In Proceedings of the 61st Annual Meeting of the…
  20. Annie Louis and Ani Nenkova. 2013. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267– 300.
  21. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for text summarization.
  22. Tomoko Nemoto and David Beglar. 2014. Likert-scale questionnaires. In JALT 2013 conference proceedings, pages 1–8. find
  23. Masanari Ohi, Masahiro Kaneko, Ryuto Koike, Mengsay Loem, and Naoaki Okazaki. 2024. Likelihood-based mitigation of evaluation bias in large language models. find
  24. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. find
  25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  26. Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translation,…
  27. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summarization asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594–6604, Online…
  28. Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La…
  29. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. find
  30. Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023. Large language models are not yet human-level evaluators for abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4215–4233, Singapore. Association for Computational Li…
  31. Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Characterizing the confidence of large language model-based automatic evaluation metrics. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 76… find
  32. Manoj Thomas and Vicki Morwitz. 2009. Heuristics in numerical cognition: Implications for pricing. In Handbook of pricing research in marketing, pages 132–149. Edward Elgar Publishing. find
  33. Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124–1131. find
  34. Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon. 2020. Fill in the BLANC: Human-free quality estimation of document summaries. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 11–20, Online. Association for Computational Linguistics.
  35. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. Is chatgpt a good nlg evaluator? a preliminary study.
  36. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023b. Large language models are not fair evaluators. find
  37. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
  38. Minghao Wu and Alham Fikri Aji. 2023. Style over substance: Evaluation biases for large language models.
  39. Yunshu Wu, Hayate Iso, Pouya Pezeshkpour, Nikita Bhutani, and Estevam Hruschka. 2023. Less is more for long document summary evaluation by llms. find
  40. Robert B Zajonc. 1968. Attitudinal effects of mere exposure. Journal of personality and social psychology, 9(2p2):1. find
  41. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BertScore: Evaluating text generation with bert. find
  42. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International…
  43. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023a. Large language models are not robust multiple choice selectors. find
  44. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023b. Judging llm-as-a-judge with mt-bench and chatbot arena. find
  45. Chloe Qinyu Zhu, Rickard Stureborg, and Bhuwan Dhingra. 2024. Hierarchical multi-label classification of online vaccine concerns. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stureborg-2024-large-language-models-inconsistent-biased/