Elamur Research⌕
☰
home / papers / 2023 / PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations

PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations

Paper · 2023 · Ruosen Li, Teerth Patel and Xinya Du

Summary

A common way to compare LLM answers to open-ended questions is to let one supposedly "strongest" model, usually GPT-4, judge pairs of responses. Such judges are biased: they tend to favour their own outputs (self-enhancement) and to prefer answers in a particular position. Drawing on peer-assessment research in education, the authors propose PRD, a framework with two parts. In Peer Rank (PR), every LLM serves as both contestant and reviewer in pairwise battles. Each reviewer's votes are weighted by how well that model itself performs, and win rates or Elo scores are recomputed iteratively until they converge, producing a global ranking. In Peer Discussion (PD), two reviewer LLMs debate over several turns which of two answers is better until they reach agreement. PR is tested on Vicuna80, and PD on Vicuna80 and LFQA, both of which include human pairwise judgments. Both methods match human preferences better than a single judge and reduce self-enhancement and position bias. PR also yields a reasonable self-ranking even when model names are hidden. The approach offers a cheaper, less biased alternative where human annotation is costly.

Key points

  • Benchmarks and models:
    • PR is evaluated on Vicuna80 using peer reviewers such as GPT-4, Bard and Claude.
    • PD is evaluated on Vicuna80 and LFQA, which covers Human-Machine and Machine-Machine answer pairs.
    • Code is available at github.com/bcdnlp/PRD.
  • Position bias: GPT-3.5 as a reviewer strongly prefers the first answer. When GPT-3's answer was shown first, GPT-3.5 gave it a 73.68% win rate versus 57.89% from human judgments. After discussion, all reviewers moved closer to the human preferences.
  • Discussion Ordering Effect: the reviewer who leads a discussion rarely changes its view. On LFQA disagreements, GPT-4 as leader never changed its opinion.
  • Stronger models hold firmer: GPT-4 kept its opinion in 174 discussions, compared with 94 for Claude and 76 for GPT-3.5.
  • Limitations:
    • PR needs O(N³) reviews for N models. The authors suggest using only K random or top-ranked models as reviewers to reduce this cost.
    • PD itself introduces some position bias, which averaging over both answer orders only partly mitigates.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Ruosen Li, Teerth Patel and Xinya Du. 2023. PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations. arXiv:2307.02762.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.02762 · arXiv 2307.02762

@misc{li2023prd,
  title = {{PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations}},
  author = {Ruosen Li, Teerth Patel and Xinya Du},
  year = {2023},
  doi = {10.48550/arxiv.2307.02762},
  eprint = {2307.02762},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.02762}
}

References

49 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. find
  2. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with language-model-as-an-examiner. arXiv preprint arXiv:2306.04181, 2023.
  3. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shan Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. ArXiv, abs/2308.07201, 2023. URL https://api.semanticscholar.org/CorpusID:260887105. ↗
  4. Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. ArXiv, abs/2309.13007, 2023. URL https://api.semanticscholar.org/CorpusID:262217323. ↗
  5. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, and et.al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, https://lmsys.org/blog/2023-03-30-vicuna/. 2023. ↗
  6. Kwangsu Cho and Charles MacArthur. Learning by reviewing. Journal of educational psychology, 103(1):73, 2011. find
  7. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
  8. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023.
  9. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. ArXiv, abs/2305.14325, 2023. URL https://api.semanticscholar.org/CorpusID:258841118. ↗
  10. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023.
  11. Nouha Dziri, Ehsan Kamalloo, Kory Mathewson, and Osmar R Zaiane. Evaluating coherence in dialogue systems using entailment. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pape… find
  12. Arpad E Elo. The proposed uscf rating system. its development, theory, and applications. Chess Life, 22(8): 242–247, 1967. find
  13. A. R. Fabbri, Wojciech Kryscinski, Bryan McCann, Richard Socher, and Dragomir R. Radev. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9: 391–409, 2020. URL https://api.semanticscholar.org/CorpusID:220768873. ↗
  14. Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. Qafacteval: Improved qa-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  15. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3558–3567, 2019.
  16. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166, 2023a. find
  17. Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.10142, 2023b.
  18. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. arXiv preprint arXiv:2305.14627, 2023.
  19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
  20. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. ArXiv, abs/1506.03340, 2015. URL https://api.semanticscholar.org/CorpusID:6203757. ↗
  21. Karen Sparck Jones and Julia R Galliers. Evaluating natural language processing systems: An analysis and review. 1995. find
  22. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. Hurdles to progress in long-form question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4940–4957, Online, June 2021. Association for Compu…
  23. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9332–9346, 2020.
  24. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. arXiv preprint arXiv:2303.17760, 2023.
  25. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. find
  26. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. ArXiv, abs/2305.19118, 2023. URL https://api.semanticscholar.org/CorpusID:258967540. ↗
  27. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74–81, 2004.
  28. Yen-Ting Lin and Yun-Nung Chen. Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. arXiv preprint arXiv:2305.13711, 2023.
  29. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023.
  30. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-… find
  31. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.
  32. David Nicol, Avril Thomson, and Caroline Breslin. Rethinking feedback practices in higher education: a peer review perspective. Assessment & evaluation in higher education, 39(1):102–122, 2014. find
  33. OpenAI. Webgpt annotation guidelines. 2022. URL https://docs.google.com/document/d/1i0h5dorAZydNNiDJamqq_ZGSpzaPKvLcuJvFaO87lM0/edit. ↗
  34. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
  35. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  36. Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999.
  37. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318, 2002.
  38. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. find
  39. Toby Walsh. The peerrank method for peer assessment. In Proceedings of the Twenty-first European Conference on Artificial Intelligence, pp. 909–914, 2014. find
  40. Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5008–5020, 2020. find
  41. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023a. find
  42. Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hanna Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources. ArXiv, abs/2306.04751, 2023b.
  43. Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. A critical evaluation of evaluations for long-form question answering. In Proceedings of ACL, 2023.
  44. Matthew M Yalch, Erika M Vitale, and J Kevin Ford. Benefits of peer review on students’ writing. Psychology Learning & Teaching, 18(3):317–325, 2019. find
  45. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. ArXiv, abs/2401.10020, 2024. URL https://api.semanticscholar.org/CorpusID:267035293. ↗
  46. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. find
  47. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
  48. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards a unified multi-dimensional evaluator for text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2023–2038, 2022.
  49. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2023-prd-peer-rank-discussion-improve/