Elamur Research⌕
☰
home / papers / 2026 / Evaluating LLMs for Detecting Demographic-Targeted Social Bias: A Comprehensive Benchmark Study

Evaluating LLMs for Detecting Demographic-Targeted Social Bias: A Comprehensive Benchmark Study

Paper · 2026 · Ayan Majumdar, Feihao Chen, Jinghui Li et al.

Summary

Regulations such as the EU AI Act increasingly require audits of web-scale training corpora for content that harms demographic groups. Manual review cannot scale, and earlier automatic-detection studies were narrow: they covered one content type, few identity axes, single-target labels and a small set of methods. The authors build an English benchmark that treats bias detection as a multi-label problem. A model must decide whether a text is biased and which of nine demographic axes it targets, including texts that attack several groups at once.

The benchmark harmonises twelve existing annotated datasets covering stereotypes, gender–occupation bias and hate/toxicity. On it, the authors compare zero- and few-shot prompting (with retrieval-selected examples), a guardrail model, and fine-tuning across model sizes. Fine-tuned compact models, especially encoder-only ones, prove the most accurate and scalable detectors and beat prompted large LLMs. However, accuracy differs noticeably between demographic axes, and multi-target texts remain difficult. The work therefore provides a reusable testbed and shows that current tools are not yet equitable auditors.

Key points

  • Taxonomy: nine axes (GEN, SO, DIS, AGE, RAC, NAT, REL, SES, PHY) plus an unbiased label (UNB). Labels can be single-axis (e.g., RAC) or combined (e.g., GEN+RAC).
  • Datasets: the benchmark draws on twelve sources: Winoqueer, Winogender, WinoBias, ToxiGen, StereoSet, SBIC, RedditBias, ImplicitHate, HateXplain, CrowS-pairs, BEC-Pro and BBQ. Sampled subsets range from 240 (Winogender) to 9,087 (RedditBias) texts.
  • Fine-tuning per axis: fine-tuned models beat prompting on every class, with the largest gains on the rarer AGE and SES. RoBERTa and DeBERTa generally outperform GPT-2. NAT is the weakest axis. A reweighted loss improves NAT for all models, though not uniformly; GPT-2-large slips slightly on AGE and SES.
  • Prompting per axis: Llama-3.1-70B leads on nearly all axes, while the much smaller GLM-4-9B edges ahead on GEN. Qwen-2.5-72B, the largest model, lags on low-frequency axes (DIS, AGE, SES).
  • Few-shot ablation:
    • BGE-M3 and BCEmbedding work about equally well for choosing in-context examples.
    • The guardrail Llama-Guard-8B reaches only about 74–75 binary F1, versus about 87–89 for general LLMs. Those general LLMs, however, show high false-positive rates (about 0.55–0.78).
    • GLM-4-9B achieves the best multi-label micro-F1 (about 86–88).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany148 works · 18,287 citations🏢Huawei TechnologiesShenzhen · China75 works · 12,975 citations🔬Max Planck Institute for Software SystemsSaarbrücken · Germany3 works · 38,590 citations

Also on the byline: Huawei Technologies (France).

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksBias, fairness & safety · 517 works

Cite

Cite as Ayan Majumdar, Feihao Chen, Jinghui Li and Xiaozhen Wang. 2026. Evaluating LLMs for Detecting Demographic-Targeted Social Bias: A Comprehensive Benchmark Study. Proceedings of the Second Workshop of Identity Aware AI.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3jejtg2hfj3t

@inproceedings{majumdar2026evaluating,
  title = {{Evaluating LLMs for Detecting Demographic-Targeted Social Bias: A Comprehensive Benchmark Study}},
  author = {Ayan Majumdar, Feihao Chen, Jinghui Li and Xiaozhen Wang},
  booktitle = {Proceedings of the Second Workshop of Identity Aware AI},
  year = {2026},
  doi = {10.63317/3jejtg2hfj3t},
  url = {https://doi.org/10.63317/3jejtg2hfj3t}
}

References

72 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. EU FRA. 2018. Handbook on European Non-Discrimination Law. Publications O"ce of the European Union, Luxembourg. find
  2. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online. Association for Computational Linguistics.
  3. European Commission. 2025. Third draft of the general-purpose ai code of practice. Draft prepared by independent experts under the coordination of the European AI O"ce. find
  4. European Union. 2024. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence. [Accessed: 2025-03-27]. find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Paula Fortuna, Juan Soler, and Leo Wanner. 2020. Toxic, hateful, o!ensive or abusive? what are we really classifying? an empirical analysis of hate speech datasets. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6786–6794. find
  7. Jianfa Chen, Emily Shen, Trupti Bavalatti, Xiaowen Lin, Yongkai Wang, Shuming Hu, Harihar Subramanyam, Ksheeraj Sai Vepuri, Ming Jiang, Ji Qi, et al. 2024a. Class-rag: Real-time content moderation with retrieval augmented generation. arXiv preprint arXiv:2410.14881. — [arXiv:2410.14881](https://arxiv.org/abs/2410.14881) find
  8. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097–1179.
  9. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024b. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216.
  10. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462.
  11. United States Congress. 1964. Title vii of the civil rights act of 1964. Pub. L. No. 88-352, 78 Stat. 241; codified at 42 U.S.C. § 2000e et seq. find
  12. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793.
  13. Jesse Dodge, Maarten Sap, Ana Marasovi\#, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pr… find
  14. Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29.
  15. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:…
  16. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  17. Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. Latent hatred: A benchmark for understanding implicit hate speech. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 345–363.
  18. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.
  19. Bob Hepple et al. 2010. The new single equality act in britain. The Equal Rights Review, 5(1):11–24. find
  20. Christoph Hube and Besnik Fetahu. 2018. Detecting biased statements in wikipedia. In Companion proceedings of the the web conference 2018, pages 1779–1786. find
  21. Edgar Litt. 1961. Jewish ethno-religious involvement and political liberalism. Social Forces, 39(4):328–332. find
  22. Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364.
  23. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674.
  24. Alexandra Luccioni and Joseph Viviano. 2021. What’s in the box? an analysis of undesirable content in the Common Crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume… find
  25. Faisal Kamiran and Toon Calders. 2012. Data preprocessing techniques for classification without discrimination. Knowledge and information systems, 33(1):1–33. find
  26. Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In International conference on machine learning, pages 2564–2572. PMLR. find
  27. Gaurav Maheshwari, Dmitry Ivanov, and Kevin El Haddad. 2024. E"cacy of synthetic data as a benchmark. arXiv preprint arXiv:2409.11968. — [arXiv:2409.11968](https://arxiv.org/abs/2409.11968) find
  28. Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2023. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15009–150…
  29. Alexander Klose, Doris Liebscher, Maria Wersig, and Michael Wrase, editors. 2025. Landesan-tidiskriminierungsgesetz Berlin. Nomos, Germany. find
  30. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2024. Benchmarking cognitive biases in large language models as evaluators. In Findings of the Association for Computational Linguistics ACL 2024, pages 517–545. find
  31. Antonis Maronikolakis, Philip Baader, and Hinrich Schütze. 2022. Analyzing hate speech data along racial, gender and intersectional axes. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 1–7. find
  32. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan Van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. 2022. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Ling…
  33. Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14867– 14875. find
  34. Deepak Kumar, Yousef Anees AbuHashem, and Zakir Durumeric. 2024. Watch your language: Investigating content moderation with large language models. In Proceedings of the International AAAI Conference on Web and Social Media, volume 18, pages 865–878. find
  35. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. Stereoset: Measuring stereotypical bias in pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lon… find
  36. John P Lalor, Yi Yang, Kendall Smith, Nicole Forsgren, and Ahmed Abbasi. 2022. Benchmarking intersectional biases in nlp. In Proceedings of the 2022 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 3598–3609. find
  37. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967.
  38. Yunqi Li, Lanjing Zhang, and Yongfeng Zhang. 2023. Fairness of chatgpt. arXiv preprint arXiv:2305.18569. — [arXiv:2305.18569](https://arxiv.org/abs/2305.18569) find
  39. Roberto Navigli, Simone Conia, and Björn Ross. 2023. Biases in large language models: origins, inventory, and discussion. ACM Journal of Data and Information Quality, 15(2):1–21. find
  40. Trustworthiness in large language models. arXiv preprint arXiv:2401.05561, 3.
  41. Kush R Vashney. 2022. Trustworthy machine learning. Independently published. find
  42. Konstantina Palla, José Luis Redondo García, Claudia Hau!, Francesco Fabbri, Andreas Damianou, Henrik Lindström, Dan Taber, and Mounia Lalmas. 2025. Policy-as-prompt: Rethinking content moderation in the age of large language models. In Proceedings of the 2025 ACM Conference on Fairness, Accountabil… find
  43. Mouna Viprey. 2002. New anti-discrimination law adopted. Eurofound. Published: 3 January 2002. find
  44. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schae!er, et al. 2023. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In NeurIPS.
  45. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. Bbq: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022.
  46. Song Wang, Peng Wang, Tong Zhou, Yushun Dong, Zhen Tan, and Jundong Li. 2024. Ceb: Compositional evaluation benchmark for fairness in large language models. arXiv preprint arXiv:2407.02408. — [arXiv:2407.02408](https://arxiv.org/abs/2407.02408) find
  47. Alec Radford, Je!rey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  48. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — Transformers: State-of-the-Art Natural Language Processing
  49. Shaina Raza, Muskan Garg, Deepak John Reji, Syed Raza Bashir, and Chen Ding. 2024. Nbias: A natural language processing framework for bias identification in text. Expert Systems with Applications, 237:121542. find
  50. Steven Salaita. 2006. Beyond orientalism and islamophobia: 9/11, anti-arab racism, and the mythos of national pride. CR: The New Centennial Review, 6(2):245–266. find
  51. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  52. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490.
  53. NetEase Youdao. 2023. Bcembedding: Bilingual and crosslingual embedding for rag. https://github.com/netease-youdao/ BCEmbedding. ↗
  54. David Schraub. 2019. White jews: an intersectional approach. AJS review, 43(2):379–407. find
  55. Mohammad S Sorower. 2010. A literature survey on algorithms for multi-label learning. Oregon State University, Corvallis, 18(1):25. find
  56. Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. 2017. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th international conference on world wide web, pages 1171–1180. find
  57. Miriah Steiger, Timir J Bharucha, Sukrit Venkatagiri, Martin J Riedl, and Matthew Lease. 2021. The psychological well-being of content moderators: the emotional labor of commercial moderation and avenues for improving support. In Proceedings of the 2021 CHI conference on human factors in computing s… find
  58. Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, et al. 2024. Shieldgemma: Generative ai content moderation based on gemma. arXiv preprint arXiv:2407.21772. — [arXiv:2407.21772](https://arxiv.org/abs/2407.21772) find
  59. Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. 2024. find
  60. Trustllm: Xianyang Zhan, Agam Goyal, Yilun Chen, Eshwar Chandrasekharan, and Koustuv Saha. 2025. Slm-mod: Small language models surpass llms at content moderation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Lan… find
  61. Nangia, Nikita and Vania, Clara and Bhalerao, Rasika and Bowman, Samuel. 2020. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. PID https://huggingface.co/datasets/nyu-mll/crows_pairs. ↗
  62. Parrish, Alicia and Chen, Angelica and Nangia, Nikita and Padmakumar, Vishakh and Phang, Jason and Thompson, Jana and Htut, Phu Mon and Bowman, Samuel. 2022. BBQ: A Hand-Built Bias Benchmark for Question Answering. PID https://github.com/nyu-mll/BBQ. \#\# 8. ↗
  63. Language Resource References Rudinger, Rachel and Naradowsky, Jason and Leonard, Brian and Van Durme, Benjamin. 2018. Gender Bias in Coreference Resolution. PID https://github.com/rudinger/winogender-schemas. ↗
  64. Barikeri, Soumya and Lauscher, Anne and Vuli\#, Ivan and Glava$, Goran. 2021. RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models. PID https://github.com/SoumyaBarikeri/RedditBias. ↗
  65. Sap, Maarten and Gabriel, Saadia and Qin, Lianhui and Jurafsky, Dan and Smith, Noah A and Choi, Yejin. 2020. Social Bias Frames: Reasoning about Social and Power Implications of Language. PID https://maartensap.com/social-bias-frames/. ↗
  66. Bartl, Marion and Nissim, Malvina and Gatt, Albert. 2020. Unmasking Contextual Stereotypes: Measuring and Mitigating BERT’s Gender Bias. PID https://github.com/marionbartl/gender-bias-BERT. ↗
  67. Zhao, Jieyu and Wang, Tianlu and Yatskar, Mark and Ordonez, Vicente and Chang, Kai-Wei. 2018. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. PID https://github.com/uclanlp/corefBias/tree/master. ↗
  68. ElSherief, Mai and Ziems, Caleb and Muchlinski, David and Anupindi, Vaishnavi and Seybolt, Jordyn and De Choudhury, Munmun and Yang, Diyi. 2021. Latent Hatred: A Benchmark for Understanding Implicit Hate Speech. PID https://github.com/SALT-NLP/implicit-hate. ↗
  69. Felkner, Virginia and Chang, Ho-Chun Herbert and Jang, Eugene and May, Jonathan. 2023. Wino-Queer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Models. PID https://github.com/katyfelkner/winoqueer. ↗
  70. Hartvigsen, Thomas and Gabriel, Saadia and Palangi, Hamid and Sap, Maarten and Ray, Dipankar and Kamar, Ece. 2022. ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection. PID https://huggingface.co/datasets/toxigen/toxigen-data. ↗
  71. Mathew, Binny and Saha, Punyajoy and Yimam, Seid Muhie and Biemann, Chris and Goyal, Pawan and Mukherjee, Animesh. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. PID https://github.com/hate-alert/HateXplain. ↗
  72. Nadeem, Moin and Bethke, Anna and Reddy, Siva. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. PID https://huggingface.co/datasets/McGill-NLP/stereoset. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/majumdar-2026-evaluating-llms-detecting-demographic-targeted/