Elamur Research
☰
home / papers / 2024 / Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge

Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2410.02736

Summary

LLMs are now routinely used to grade other models' outputs, both in benchmarks and as reward signals during training. Their impartiality, however, has been studied only piecemeal, and often with costly, subjective human annotation. Ye et al. propose CALM (Comprehensive Assessment of Language Model Judge Biases), a framework that measures judge bias without human raters.

CALM defines a taxonomy of 12 bias types and probes each one with an attack-and-detect procedure. A second LLM (GPT-4o) makes principle-guided edits to a candidate answer or to the system instruction. Examples include padding a response, inserting a fabricated book citation, or telling the judge that an answer was refined. The judge's verdicts on the original and perturbed inputs are then compared. Tests cover fact-related reasoning, answer refinement and alignment data, in both pairwise-comparison and scoring setups.

Across six popular judge LLMs, the strongest models are fairly robust overall, but notable biases remain for particular bias types and tasks. The work offers a scalable, reproducible auditing tool and warns practitioners against trusting automatic verdicts uncritically.

Key points

  • Bias types covered (12): position, verbosity, compassion-fade (model names vs. anonymous aliases), bandwagon, distraction, fallacy-oversight, authority, sentiment, diversity (identity mentions), CoT, self-preference and refinement-aware bias.
  • Datasets by evaluation aspect:
    • Fact-related: GSM8K, MATH, ScienceQA.
    • Refinement-aware: CommonsenseQA, TruthfulQA.
    • Alignment: OpenOrca, emerton_dpo.
  • Answer generation: candidate answers come from Llama3-8B, Llama3-70B, Mistral-7B and Mixtral-8x22B. GPT-4o injects the biased modifications.
  • Judges evaluated: six LLMs, including ChatGPT, GPT-4o, GLM-4, Claude-3.5-Sonnet and Qwen-72b.
  • Metrics: Robustness Rate (RR), Consistency Rate (CR), Original vs. Hacked Accuracy, and Error Rate. A judge counts as unbiased when its score or preference does not change after the perturbation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

133 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 131 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.02736 · arXiv 2410.02736

@misc{ye2024justice,
  title = {{Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge}},
  author = {Ye, Jiayi and Wang, Yanbo and Huang, Yue and Chen, Dongping and Zhang, Qihui and Moniz, Nuno and Gao, Tian and Geyer, Werner and Huang, Chao and Chen, Pin-Yu and Chawla, Nitesh V and Zhang, Xiangliang},
  year = {2024},
  doi = {10.48550/arxiv.2410.02736},
  eprint = {2410.02736},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.02736}
}

References

67 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL\_CARD.md. ↗
  2. AI@Mistral. Mistral 7b: The best 7b model to date, apache 2.0, 2023. URL https://mistral.ai/news/announcing-mistral-7b/. ↗
  3. AI@Mistral. Cheaper, better, faster, stronger, 2024. mixtral-8x22b/. find
  4. Anthropic. Claude 3.5 sonnet, claude-3-5-sonnet. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuan… find our card
  5. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. find our card
  6. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity, 2023. URL https://arxiv.… ↗ our card
  7. Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark, 2024a. URL https://arxiv.org/abs/2402.04788. ↗
  8. Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, et al. Gui-world: A dataset for gui-oriented multimodal llm-based agents. arXiv preprint arXiv:2406.10819, 2024b. — [arXiv:2406.10819](https://arxiv.org/abs/2406.10819) find
  9. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or llms as the judge? a study on judgement biases, 2024c. URL https://arxiv.org/abs/2402.10669. ↗
  10. Weize Chen, Ziming You, Ran Li, Yitong Guan, Chen Qian, Chenyang Zhao, Cheng Yang, Ruobing Xie, Zhiyuan Liu, and Maosong Sun. Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence. arXiv preprint arXiv:2407.07061, 2024d. — [arXiv:2407.07061](https://arxiv.org/abs/2407.07061) find
  11. Xiuying Chen, Tairan Wang, Taicheng Guo, Kehan Guo, Juexiao Zhou, Haoyang Li, Mingchen Zhuge, Jürgen Schmidhuber, Xin Gao, and Xiangliang Zhang. Scholarchemqa: Unveiling the power of language models in chemical research question answering, 2024e. URL https://arxiv. org/abs/2407.16931. find
  12. URL https://mistral.ai/news/2024. URL https://www.anthropic.com/news/Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word prob… ↗ our card
  13. Jon Durbin. Truthy-dpo-v0.1. https://huggingface.co/datasets/jondurbin/truthy-dpo-v0.1, 2023. Accessed: 2024-07-15. ↗
  14. Jon Durbin. Py-dpo-v0.1. https://huggingface.co/datasets/jondurbin/py-dpo-v0.1, 2024. Accessed: 2024-07-15. ↗
  15. Emilio Ferrara. Should chatgpt be biased? challenges and risks of bias in large language models. First Monday, November 2023. ISSN 1396-0466. doi: 10.5210/fm.v28i11.13346. URL http: //dx.doi.org/10.5210/fm.v28i11.13346. — [doi:10.5210/fm.v28i11.13346](https://doi.org/10.5210/fm.v28i11.13346) find
  16. Chujie Gao, Qihui Zhang, Dongping Chen, Yue Huang, Siyuan Wu, Zhengyan Fu, Yao Wan, Xiangliang Zhang, and Lichao Sun. The best of both worlds: Toward an honest and helpful large language model. arXiv preprint arXiv:2406.00380, 2024. — [arXiv:2406.00380](https://arxiv.org/abs/2406.00380) find
  17. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinka… find our card
  18. Taicheng Guo, kehan Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, and Xiangliang Zhang. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neura… find
  19. Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges, 2024. URL https://arxiv.org/abs/2402.01680. ↗ our card
  20. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874. ↗ our card
  21. Hui Huang, Yingqi Qu, Hongli Zhou, Jing Liu, Muyun Yang, Bing Xu, and Tiejun Zhao. On the limitations of fine-tuned judge models for llm evaluation, 2024a. URL https://arxiv.org/abs/2403.02839. — [arXiv:2403.02839](https://arxiv.org/abs/2403.02839) ↗
  22. Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. Metatool benchmark for large language models: Deciding whether to use tools and which to use. arXiv preprint arXiv:2310.03128, 2023a. find
  23. Yue Huang, Qihui Zhang, Lichao Sun, et al. Trustgpt: A benchmark for trustworthy and responsible large language models. arXiv preprint arXiv:2306.11507, 2023b. — [arXiv:2306.11507](https://arxiv.org/abs/2306.11507) find
  24. Yue Huang, Jingyu Tang, Dongping Chen, Bingda Tang, Yao Wan, Lichao Sun, and Xiangliang Zhang. Obscureprompt: Jailbreaking large language models via obscure input. arXiv preprint arXiv:2406.13662, 2024b. — [arXiv:2406.13662](https://arxiv.org/abs/2406.13662) find
  25. Intel. Orca-dpo-pairs. https://huggingface.co/datasets/Intel/orca\_dpo\_pairs, 2023. Accessed: 2024-07-15. ↗
  26. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. Is chatgpt a good translator? yes with gpt-4 as the engine, 2023. URL https://arxiv.org/abs/2301.08745. ↗ our card
  27. Zdeněk Kasner and Ondřej Dušek. Beyond traditional benchmarks: Analyzing behaviors of open llms on data-to-text generation, 2024. URL https://arxiv.org/abs/2401.10186. ↗
  28. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators, 2023. URL https: //arxiv.org/abs/2309.17012. find
  29. Khiem Le, Zhichun Guo, Kaiwen Dong, Xiaobao Huang, Bozhao Nan, Roshni Iyer, Xiangliang Zhang, Olaf Wiest, Wei Wang, and Nitesh V. Chawla. Molx: Enhancing large language models for molecular learning with a multi-modal extension, 2024. URL https://arxiv.org/abs/2406.06777. — [arXiv:2406.06777](https://arxiv.org/abs/2406.06777) ↗
  30. Y. Leo. Emerton-dpo-pairs-judge. https://huggingface.co/datasets/yleo/emerton\_dpo\_pairs\_judge/viewer, 2024. Accessed: 2024-07-15. ↗
  31. Alice Li and Luanne Sinnamon. Examining query sentiment bias effects on search results in large language models. In The Symposium on Future Directions in Information Access (FDIA) co-located with the 2023 European Summer School on Information Retrieval (ESSIR), 2023. find
  32. Ruosen Li, Teerth Patel, and Xinya Du. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint arXiv:2307.02762, 2023a. find
  33. Yuan Li, Yue Huang, Yuli Lin, Siyuan Wu, Yao Wan, and Lichao Sun. I think, therefore i am: Benchmarking awareness of large language models using awarebench, 2024a. URL https: //arxiv.org/abs/2401.17882. — [arXiv:2401.17882](https://arxiv.org/abs/2401.17882) find
  34. Yuan Li, Yue Huang, Hongyi Wang, Xiangliang Zhang, James Zou, and Lichao Sun. Quantifying ai psychology: A psychometrics benchmark for large language models. arXiv preprint arXiv:2406.17675, 2024b. — [arXiv:2406.17675](https://arxiv.org/abs/2406.17675) find
  35. Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. Split and merge: Aligning position biases in large language model based evaluators, 2023b. URL https://arxiv.org/abs/2310.01432. — [arXiv:2310.01432](https://arxiv.org/abs/2310.01432) ↗
  36. Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL https://arxiv.org/abs/2109.07958. ↗ our card
  37. Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. Alignbench: Benchmarking chinese alignment of large language models. arXiv preprint arXiv:2311.18743, 2023a. find
  38. Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: a survey and guideline for evaluating large language models’ alignment, 2024. URL https://arxiv.org/abs/2308.05374. ↗ our card
  39. Zhengliang Liu, Yue Huang, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Yiwei Li, Peng Shu, et al. Deid-gpt: Zero-shot medical text de-identification by gpt-4. arXiv preprint arXiv:2303.11032, 2023b. — [arXiv:2303.11032](https://arxiv.org/abs/2303.11032) find
  40. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022. URL https://arxiv.org/abs/2209.09513. — [arXiv:2209.09513](https://arxiv.org/abs/2209.09513) ↗
  41. John Macnicol. Age Discrimination: An Historical and Contemporary Analysis. 01 2006. ISBN 9780521847773. doi: 10.1017/CBO9780511550560. — [doi:10.1017/cbo9780511550560](https://doi.org/10.1017/cbo9780511550560) find
  42. Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024. find
  43. OpenAI. Gpt-4 technical report, 2024a. URL https://arxiv.org/abs/2303.08774. ↗ our card
  44. OpenAI. Gpt-3.5-turbo model documentation, 2024b. URL https://platform.openai.com/docs/models. ↗
  45. OpenAI. Hello gpt-4o, 2024c. URL https://openai.com/index/hello-gpt-4o/. ↗
  46. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730– 27744, 2022. find
  47. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. find
  48. Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. Optimization-based prompt injection attack to llm-as-a-judge. arXiv preprint arXiv:2403.17710, 2024a. — [arXiv:2403.17710](https://arxiv.org/abs/2403.17710) find
  49. Lin Shi, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms, 2024b. URL https://arxiv.org/abs/2406.07791. — [arXiv:2406.07791](https://arxiv.org/abs/2406.07791) ↗
  50. Paul Slovic, Melissa Finucane, Ellen Peters, and Donald G. MacGregor. The Affect Heuristic, pp. 397–420. Cambridge University Press, 2002. find
  51. Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. Large language models are inconsistent and biased evaluators, 2024. URL https://arxiv.org/abs/2405.01724. ↗
  52. Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric Xing, Furong Huang, Hao Liu, Heng Ji,… find our card
  53. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019. URL https://arxiv.org/abs/1811.00937. ↗
  54. Toughdata. Quora question answer dataset. https://huggingface.co/datasets/toughdata/quora-question-answer-dataset, 2023. ↗
  55. Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. "kelly is a warm person, joseph is a role model": Gender biases in llm-generated reference letters, 2023. URL https://arxiv.org/abs/2310.09219. ↗
  56. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is chatgpt a good nlg evaluator? a preliminary study, 2023a. URL https://arxiv.org/abs/2303.04048. ↗ our card
  57. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023b. URL https: //arxiv.org/abs/2305.17926. find
  58. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903. ↗ our card
  59. Minghao Wu and Alham Fikri Aji. Style over substance: Evaluation biases for large language models, 2023. URL https://arxiv.org/abs/2307.03025. ↗
  60. Siyuan Wu, Yue Huang, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Xiangliang Zhang, Jianfeng Gao, Chaowei Xiao, et al. Unigen: A unified framework for textual dataset generation using large language models. arXiv preprint arXiv:2406.18966, 2024a. — [arXiv:2406.18966](https://arxiv.org/abs/2406.18966) find
  61. Yuanwei Wu, Yue Huang, Yixin Liu, Xiang Li, Pan Zhou, and Lichao Sun. Can large language models automatically jailbreak gpt-4v? arXiv preprint arXiv:2407.16686, 2024b. xDAN. xdan-sft-dpo-roleplay-nsfw-with-lf. https://huggingface.co/datasets/xDAN2099/xDAN-SFT-DPO-Roleplay-NSFW-with-lf, 2024. Accesse… — [arXiv:2407.16686](https://arxiv.org/abs/2407.16686) ↗
  62. Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Wang. Pride and prejudice: LLM amplifies self-bias in self-refinement. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:… find
  63. Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper llm networks are fairer llm evaluators, 2023. URL https: //arxiv.org/abs/2308.01862. find
  64. Chenyang Zhao, Xueying Jia, Vijay Viswanathan, Tongshuang Wu, and Graham Neubig. Self-guide: Better task-specific instruction following via self-synthetic finetuning. arXiv preprint arXiv:2407.12874, 2024. — [arXiv:2407.12874](https://arxiv.org/abs/2407.12874) find
  65. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Lmsys chat platform. https://chat.lmsys.org/, 2023. ↗
  66. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. find
  67. Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. Exploring ai ethics of chatgpt: A diagnostic analysis. ArXiv, abs/2301.12867, 2023. URL https://api.semanticscholar. org/CorpusID:256390238. ↗