Elamur Research⌕
☰
home / papers / 2023 / Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?

Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?

Paper · 2023 · Rishav Hada, Varun Gumma, Adrian de Wynter et al.

Summary

Scaling LLM evaluation beyond English is difficult. Benchmarks cover few languages and few realistic tasks, automatic metrics handle rich morphology and unusual scripts poorly, and native-speaker judges are costly and slow. The authors test whether GPT-4, scoring outputs without reference texts, can replace or supplement human raters. They use outputs from a GPT-4-powered text generation product on three tasks (Open Prompt, Continue Writing, Summarize). The outputs come from high-resource languages ranging from English to Czech. The authors compare GPT-4's scores with roughly 20K native-speaker judgments across five quality dimensions and try several prompting setups. Raw agreement with humans looks high. However, GPT-4 consistently gives generous scores, especially when human raters disagree with each other. This gap is larger for non-Latin-script languages (Chinese, Japanese) and for the less-resourced Czech. The study warns against trusting LLM judges uncritically in multilingual settings. It recommends calibrating them against human labels in each language before use.

Key points

  • The five metrics are Linguistic Acceptability (LA), Output Content Quality (OCQ) and Task Quality (TQ), each scored 0–2, plus Problematic Content (PC) and Hallucinations (H), both binary.
  • The abstract says eight languages, but the setup lists nine: En, Fr, De, Es, Zh, Ja, It, Pt-Br and Cs. Following Joshi et al. (2020), most are Class 5, and It, Pt-Br and Cs are Class 4.
  • Each item received three annotations from contracted native speakers, paid about $14–30 USD/hr depending on locale.
  • Scoring one metric per GPT-4 call worked better than scoring all metrics at once. Few-shot examples gave no improvement, and more detailed instructions did not remove the upward score bias.
  • The human-annotated dataset is in-house and unreleased, which limits reproducibility. Code is available at aka.ms/LLM-Eval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🏢MicrosoftRedmond · United States240 works · 123,537 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 1,344 worksEvaluation & benchmarks · 685 worksMultilingual & cross-lingual NLP · 465 works

Cite

Cite as Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali and Sunayana Sitaram. 2023. Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?. arXiv:2309.07462.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.07462 · arXiv 2309.07462

@misc{hada2023are,
  title = {{Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?}},
  author = {Rishav Hada, Varun Gumma, Adrian {de Wynter}, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali and Sunayana Sitaram},
  year = {2023},
  doi = {10.48550/arxiv.2309.07462},
  eprint = {2309.07462},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.07462}
}

References

35 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kabir Ahuja, Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2022. Beyond static models and test sets: Benchmarking the potential of pre-trained models across tasks and languages. find
  2. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023a. MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in…
  3. Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023b. Megaverse: Benchmarking large language models across languages, modalities, models and tasks.
  4. Daman Arora, Himanshu Singh, and Mausam. 2023. Have LLMs advanced enough? a challenging problem solving benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7527–7543, Singapore. Association for Computational Linguistic… find
  5. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. 6282–6293, Online. Association for Computational Lingu…
  6. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  7. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. find
  8. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201.
  9. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109.
  10. Honghua Chen and Nai Ding. 2023. Probing the “creativity” of large language models: Can models produce divergent semantic association? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12881–12888, Singapore. Association for Computational Linguistics. find
  11. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics. find
  12. Barbara Di Eugenio and Michael Glass. 2004. The kappa statistic: A second look. Computational linguistics, 30(1):95–101. find
  13. Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André FT Martins, Graham Neubig, Ankush Garg, Jonathan H Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. arXiv preprint arXiv…
  14. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166. find
  15. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056.
  16. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2023. Benchmarking cognitive biases in large language models as evaluators. find
  17. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  18. Feifan Liu and Yang Liu. 2008. Correlation between ROUGE and human evaluation of extractive meeting summaries. In Proceedings of ACL-08: HLT, Short Papers, pages 201–204, Columbus, Ohio. Association for Computational Linguistics. find
  19. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634. find
  20. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651. find
  21. Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin, and Erik Cambria. 2023. Gpteval: A survey on assessments of chatgpt and gpt-4. arXiv preprint arXiv:2308.12488. find
  22. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Ab…
  23. Ben Naismith, Phoebe Mulcaire, and Jill Burstein. 2023. Automated evaluation of written discourse coherence using GPT-4. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 394–403, Toronto, Canada. Association for Computational Lingui…
  24. Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. arXiv preprint arXiv:2305.10160. find
  25. OpenAI. 2023. Gpt-4 technical report.
  26. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages Kishore Papineni, Salim Roukos, Todd Wa…
  27. Pennsylvania, USA. Association for Computational Linguistics. find
  28. Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393–401.
  29. Chenhui Shen, Liying Cheng, Yang You, and Lidong Bing. 2023. Are large language models good evaluators for abstractive summarization? arXiv preprint arXiv:2305.13091. find
  30. Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Blüth-gen, A. Pareek, Malgorzata Polacin, William Collins, Neera Ahuja, C. Langlotz, Jason Hom, S. Gatidis, John Pauly, and Akshay S Chaudhari. 2023. Clinical text summarization: Adapting large language mode… find
  31. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048.
  32. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023b. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. find
  33. Minghao Wu and Alham Fikri Aji. 2023. Style over substance: Evaluation biases for large language models. arXiv preprint arXiv: 2307.03025.
  34. Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. 2023. Wider and deeper llm networks are fairer llm evaluators. arXiv preprint arXiv:2308.01862.
  35. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hada-2023-large-language-model-based-evaluators/