Elamur Research⌕
☰
home / papers / 2026 / Quantifying Metric and Model Agreement in Bias Evaluation of Large Language Models

Quantifying Metric and Model Agreement in Bias Evaluation of Large Language Models

Summary

The bias of large language models is evaluated with dozens of metrics and benchmarks. However, until now there has been no way to check whether these tools agree with one another and whether they rank models consistently. The authors propose two measures. The Metric Agreement Score (MeAS) shows how similarly different metrics rank models. The Model Agreement Score (MoAS) compares the bias profiles of the models themselves. Both measures are computed via Spearman and Pearson correlations, and neither is tied to specific models or metrics. The study covers ten LLMs, seven metrics (among them CAT, LMB, HONEST and classifier-based toxicity metrics) and nine corpora. For each metric, the corpora were merged into a shared pool. It turned out that even metrics from the same category often behave independently. Models of the DeepSeek family systematically reverse the rankings obtained for other models. Consequently, a conclusion based on a single metric may be wrong, and bias mitigation methods will not necessarily work for a different model family. The authors therefore insist on evaluation with multiple metrics that takes their agreement into account.

Key points

  • Scale of the experiment: 241 configurations, 750,399 examples, approximately 629 GPU-days on 4×A40. This exceeds BEATS, FairMT-Bench or ROBBIE.
  • Models: Phi-2, Phi-3-mini, Phi-3.5-mini, Gemma-2-9B, Mistral-7B, Llama-4-Scout-17B, Llama-3.1-8B, Llama-3-8B, DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-Distill-Qwen-7B. The data used include StereoSet, BOLD, RealToxicityPrompts, HolisticBias, WinoBias, CrowS-Pairs, RedditBias and BUG. All benchmarks are in English.
  • Disagreements between metrics: HONEST is almost unrelated to the toxicity metrics, although both evaluate generated text. CAT does not correlate with Language Modeling Bias (LMB).
  • Llama, Gemma and Mistral have similar bias profiles. DeepSeek models, by contrast, often show a negative correlation with them.
  • In the leave-one-dataset-out (LODO) ablation, the MoAS rankings barely change, i.e. the conclusions are robust to the composition of the data. The code is available on GitHub: arashasg/LLM-Bias-Agreement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.769

@inproceedings{asgari2026quantifying,
  title = {{Quantifying Metric and Model Agreement in Bias Evaluation of Large Language Models}},
  author = {Asgari, Arash and Wu, Huan and Naziri, Amirreza and Kolahdouzi, Mojtaba and Seyyed-Kalantari, Laleh},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.769},
  url = {https://doi.org/10.18653/v1/2026.acl-long.769}
}

References

38 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219. find our card
  2. Alok Abhishek, Lisa Erickson, and Tushar Bandopad-hyay. 2025. Beats: Bias evaluation and assessment test suite for large language models. arXiv preprint arXiv:2503.24310. — [arXiv:2503.24310](https://arxiv.org/abs/2503.24310) find
  3. Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021. Redditbias: A real-world resource for bias evaluation and debiasing of conversational language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conf… find
  4. Elisa Bassignana, Valerio Basile, Viviana Patti, and 1 others. 2018. Hurtlex: A multilingual lexicon of words to hurt. In CEUR Workshop proceedings, volume 2253, pages 1–6. CEUR-WS. find
  5. Lina Berrayana, Sean Rooney, Luis Garcés-Erice, and Ioana Giurgiu. 2025. Are bias evaluation methods biased? arXiv preprint arXiv:2506.17111. find
  6. Su Lin Blodgett, Solon Barocas, Hal Daumé Iii, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in nlp. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5454–5476. find our card
  7. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… find
  8. David M. Corey, William P. Dunlap, and Michael J. Burke. 1998. Averaging correlations: Expected values and bias in combined pearson rs and fisher’s z transformations. The Journal of General Psychology, 125(3):245–261. find
  9. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. find our card
  10. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages… find
  11. David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi-Yu, Eleonora Presani, Adina Williams, and Eric Smith. 2023. ROBBIE: Robust bias evaluation of large generative language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan… find
  12. Zhiting Fan, Ruizhe Chen, Tianxiang Hu, and Zuozhu Liu. 2024. Fairmt-bench: Benchmarking fairness for multi-turn dialogue in conversational llms. arXiv preprint arXiv:2410.19317. — [arXiv:2410.19317](https://arxiv.org/abs/2410.19317) find
  13. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, pages 1–79. find our card
  14. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguisti… find our card
  15. Paweł Gmyrek, Christoph Lutz, and Gemma Newlands. 2025. A technological construction of society: Comparing gpt-4 and human respondents for occupational evaluation in the uk. British Journal of Industrial Relations, 63(1):180–208. find
  16. Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metrics do not correlate with application bias. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confere… find
  17. Seraphina Goldfarb-Tarrant, Eddie L Ungless, Esma Balkir, and Su Lin Blodgett. 2023. This prompt is measuring\< mask\>: evaluating bias evaluation in language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2209– 2225. find
  18. Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, and 1 others. 2023. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1(3):3. find
  19. Hyejun Jeong, Shiqing Ma, and Amir Houmansadr. 2025. Bias similarity across large language models. Preprint, arXiv:2410.12010. — [arXiv:2410.12010](https://arxiv.org/abs/2410.12010) find
  20. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  21. Dahyun Jung, Seungyoon Lee, Hyeonseok Moon, Chanjun Park, and Heuiseok Lim. 2025. Flex: A benchmark for evaluating robustness of fairness in large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3606–3620, Albuquerque, New Mexico. Association for Comp… find
  22. Shahar Levy, Koren Lazar, and Gabriel Stanovsky. 2021. Collecting a large-scale gender bias dataset for coreference resolution and machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2470–2480, Punta Cana, Dominican Republic. Association for Computati… find our card
  23. Xinyue Li, Zhenpeng Chen, Jie M Zhang, Yiling Lou, Tianlin Li, Weisong Sun, Yang Liu, and Xu-anzhe Liu. 2024. Benchmarking bias in large language models during role-playing. arXiv preprint arXiv:2411.00585. — [arXiv:2411.00585](https://arxiv.org/abs/2411.00585) find
  24. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, and 1 others. 2022. Holistic evaluation of language models. Transactions on Machine Learning Research. Featured Certification, Expert Certification, Outs… find
  25. Marta Marchiori Manerba, Karolina Stańczak, Riccardo Guidotti, and Isabelle Augenstein. 2023. Social bias probing: Fairness benchmarking for language models. arXiv preprint arXiv:2311.09090. find
  26. Mackraz, Barry-John Theobald, Luca Zappella, and Nicholas Apostoloff. 2025. Is your model fairly certain? uncertainty-aware fairness evaluation for llms. Preprint, arXiv:2505.23996. — [arXiv:2505.23996](https://arxiv.org/abs/2505.23996) find
  27. AI Meta. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai.meta. com/blog/llama-4-multimodal-intelligence/, checked on, 4(7):2025. ↗
  28. Robert Wolfe and Aylin Caliskan. 2022. Markedness in visual semantic ai. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1269–1279. find
  29. Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456. find our card
  30. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or-donez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… find our card
  31. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for… find our card
  32. Debora Nozza, Federico Bianchi, Dirk Hovy, and 1 others. 2021. Honest: Measuring hurtful sentence completion in language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computat… find
  33. Yotam Perlitz, Ariel Gera, Ofir Arviv, Asaf Yehudai, Elron Bandel, Eyal Shnarch, Michal Shmueli-Scheuer, and Leshem Choshen. 2024. Do these llm benchmarks agree? fixing benchmark evaluation with benchbench. arXiv preprint arXiv:2407.13696. — [arXiv:2407.13696](https://arxiv.org/abs/2407.13696) find
  34. Harsh Raj, Domenic Rosati, and Subhabrata Majumdar. 2023. Measuring reliability of large language models through semantic consistency. Preprint, arXiv:2211.05853. — [arXiv:2211.05853](https://arxiv.org/abs/2211.05853) find
  35. Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022. "i’m sorry to hear that": Finding new biases in language models with a holistic descriptor dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages… find
  36. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, and 1 others. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. find
  37. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. find
  38. Yinong Oliver Wang, Nivedha Sivakumar, Falaah Arif Khan, Rin Metcalf Susa, Adam Golinski, Natalie find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/asgari-2026-quantifying-metric-model-agreement-bias/