Elamur Research⌕
☰
home / papers / 2024 / Self-Preference Bias in LLM-as-a-Judge

Self-Preference Bias in LLM-as-a-Judge

Paper · 2024 · Koki Wataoka, Tsubasa Takahashi and Ryokan Ri

Summary

LLM judges are widely used to score dialogue systems, but they may favour their own answers, which risks amplifying one model's style or policies. Earlier attempts to measure this relied on absolute scoring or reference-based metrics in narrow tasks such as summarization and translation, and they lacked a solid human baseline. The authors treat the judge as a classifier and borrow the Equal Opportunity notion from algorithmic fairness. Bias is the gap in how often the judge picks a human-preferred response, depending on whether that response is its own. Using pairwise human verdicts from the Chatbot Arena dataset, they test eight LLMs and swap answer order to cancel position bias. GPT-4 shows by far the strongest self-preference. To explain this, they relate judgments to the perplexity of the answers. LLM judges reward low-perplexity text much more than humans do, whether or not they wrote it. Self-preference therefore looks like a side effect of preferring familiar text. This matters for anyone who relies on a single model as an evaluator.

Key points

  • Proposed metric: Bias = P(Y′=1 | S=1, Y=1) − P(Y′=1 | S=0, Y=1).
    • A value of 0 means no bias, values near 1 mean strong self-preference, and negative values mean the judge undervalues itself.
  • Data and method: Chatbot Arena supplies 33,000 dialogues, each with a human query, two responses from different LLMs and a human verdict.
    • The judge's score comes from its normalized probabilities for the tokens "A" and "B".
    • Scores are averaged over both answer orderings.
  • Judges tested: GPT-4, GPT-3.5-turbo, Vicuna-13b, Vicuna-7b, Koala-13b, oasst-pythia-12b, dolly-v2-12b and stablelm-tuned-alpha-7b.
    • A Demographic Parity variant of the metric gives GPT-4 0.749, Vicuna-13b 0.382 and GPT-3.5-turbo 0.191.
    • dolly-v2-12b (−0.069) and stablelm-tuned-alpha-7b (−0.032) come out slightly negative.
  • Perplexity findings: Every evaluator has lower average log-perplexity on its own responses.
    • Once perplexity is controlled for, own and foreign texts win at similar rates; dolly-v2-12b and stablelm-tuned-alpha-7b are the exceptions.
    • Llama-2 and later models, including Llama 3.1, also over-reward low-perplexity text compared with humans.
  • Suggested mitigation: Ensembles of judges could down-weight a model on samples where its perplexity is low.
    • The new metric can be used to check whether such fixes actually work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksSafety Research · 221 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,343 works

Cite

Cite as Koki Wataoka, Tsubasa Takahashi and Ryokan Ri. 2024. Self-Preference Bias in LLM-as-a-Judge. arXiv:2410.21819.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.21819 · arXiv 2410.21819

@misc{wataoka2024self,
  title = {{Self-Preference Bias in LLM-as-a-Judge}},
  author = {Koki Wataoka, Tsubasa Takahashi and Ryokan Ri},
  year = {2024},
  doi = {10.48550/arxiv.2410.21819},
  eprint = {2410.21819},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.21819}
}

References

17 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural In… find
  2. Julian’s, Malta, March 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.findings-eacl.61. Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. Transactions of the Association for Computa… ↗
  3. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-… find
  4. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=KuPixIqPiq. ↗
  5. Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openr…
  6. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=0NphYCmgua. ↗
  7. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training…
  8. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=HPuS… ↗
  9. Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Tianxiang Li, Shuai Wang, Cuiyun Gao, and Yang Liu. Split and merge: Aligning position biases in large language model based evaluators, 2024. URL https://openreview.net/forum?id=1hLFLNu4uy. ↗
  10. Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. Verbosity bias in preference labeling by large language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. URL https://openreview.net/forum?id=magEgFpK1y. ↗
  11. Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. Building classifiers with independency constraints. In 2009 IEEE International Conference on Data Mining Workshops, pages 13–18, 2009. doi:10.1109/ICDMW.2009.83. — [doi:10.1109/icdmw.2009.83](https://doi.org/10.1109/icdmw.2009.83) find
  12. Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 3323–3331, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
  13. Achiam Josh, Adler Steven, Agarwal Sandhini, Ahmad Lama, Akkaya Ilge, Leoni Aleman Florencia, Almeida Diogo, Altenschmidt Janko, Altman Sam, and et al. Shyamal, Anadkat. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774. ↗
  14. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/… ↗
  15. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on M… find
  16. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023. URL https://www.databricks.com/blog/2023/04/12/ dolly-first-open-commercially-viable-… ↗
  17. Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL https://bair.berkeley.edu/blog/2023/ 04/03/koala/. et al. Jonathan Tow. Stablelm alpha v2 models, 2023. URL https://huggingface… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wataoka-2024-self-preference-bias-llm-judge/