Elamur Research⌕
☰
home / papers / 2026 / Do Large Language Models Reflect Demographic Pluralism in Safety?

Do Large Language Models Reflect Demographic Pluralism in Safety?

Paper · 2026 · Usman Naseem, Gautam Siddharth Kashyap, Sushant Kumar Ray et al.

Summary

Safety judgments about LLM outputs differ across communities, yet widely used alignment resources such as ANTHROPIC-HH and DICES draw on narrow annotator pools. Automated LLMs-as-Judges pipelines inherit the same bias. The authors propose Demo-SafetyBench, which places demographic pluralism in the prompts themselves rather than in model responses.

In Stage I, DICES prompts keep their demographic metadata (gender, race, age, education). Mistral-7B-Instruct-v0.3 relabels them, based on text alone, into 14 multi-label safety domains adapted from BEAVERTAILS. Sparse domains are enlarged with Llama-3.1-8B-Instruct, conditioned on demographics drawn from the empirical distribution. SimHash then removes near-duplicates, giving a 43,050-sample corpus.

In Stage II, Gemma-7B, GPT-4o and LLaMA-2-7B rate the prompts zero-shot. GPT-4o is the most consistent and least demographically sensitive rater. Even so, all three models score prompts differently depending on the attached demographic profile. This indicates that LLM safety calibration remains socially conditioned and needs demographic-aware evaluation.

Key points

  • Corpus construction: DICES prompts are reclassified into 14 BEAVERTAILS-derived domains, such as Hate Speech, Privacy Violation and Self-Harm. Domains with fewer than 100 queries are expanded. The final corpus has 43,050 samples, split 80/10/10, and prompts labeled "None" are excluded from benchmarking.
  • Rater reliability: GPT-4o reaches ICC = 0.87 and DS ≈ 0.12. LLaMA-2-7B reaches ICC = 0.69 and Gemma-7B ICC = 0.55.
  • Efficiency of smaller raters: Gemma-7B and LLaMA-2-7B show similar pluralistic trends at much lower cost: 0.42–0.58 s per query, 12.6–14.8 GB of memory, and at most 1.1 kWh per 1k queries.
  • Demographic effects: Prompts linked to male, White, 40+ and high-school profiles get higher risk scores than female, Asian, 18–25 and graduate profiles. Deviations from the global mean reach up to 0.06. The gap is largest for LLaMA-2-7B and smallest for GPT-4o.
  • Threshold settings: A classification threshold of δ = 0.5 balances errors (about 5.6% false positives and 6.1% false negatives). A SimHash Hamming distance of τ = 10 discards about 6.3% of items while keeping cosine similarity near 0.82.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Macquarie UniversitySydney · Australia42 works · 5,776 citations🎓University of DelhiNew Delhi · India5 works · 15 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Usman Naseem, Gautam Siddharth Kashyap, Sushant Kumar Ray, Rafiq Ali, Ebad Shabbir and Abdullah Mohammad. 2026. Do Large Language Models Reflect Demographic Pluralism in Safety?. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.107

@inproceedings{naseem2026do,
  title = {{Do Large Language Models Reflect Demographic Pluralism in Safety?}},
  author = {Usman Naseem, Gautam Siddharth Kashyap, Sushant Kumar Ray, Rafiq Ali, Ebad Shabbir and Abdullah Mohammad},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.107},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.107}
}

References

40 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Claude sonnet 4.5 system card. find
  2. Lora Aroyo, Alex Taylor, Mark Díaz, Christopher Homan, Alicia Parrish, Gregory Serapio-García, Vinodkumar Prabhakaran, and Ding Wang. 2023. Dices dataset: Diversity in conversational ai evaluation for safety. In Advances in Neural Information Processing Systems, volume 36, pages 53330–53342. Curran… find
  3. Mohammad Anas Azeez, Rafiq Ali, Ebad Shabbir, Zohaib Hasan Siddiqui, Gautam Siddharth Kashyap, Jiechao Gao, and Usman Naseem. 2025. Truth, trust, and trouble: Medical ai on the edge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1017… find
  4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, and 13 others. 2… find
  5. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022b. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
  6. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya… — LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
  7. Abeba Birhane, Sepehr Dehdashtian, Vinay Prabhu, and Vishnu Boddeti. 2024. The dark side of dataset scaling: Evaluating racial classification in multimodal models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1229–1244. find
  8. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 4356–4364.
  9. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… — On the Opportunities and Risks of Foundation Models
  10. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and 1 others. 2024. Culturalbench: a robust, diverse and challenging benchmark on measuring the (lack of) cultural knowledge of llms. CoRR. find
  11. Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 4302– 4310. find
  12. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773.
  13. Anna Dillon, Geraldine Chell, Nusaibah Al Ameri, Nahla Alsayed, Yusra Salem, Moss Turner, and Kay Gallagher. 2024. The use of large language model tools such as chatgpt in academic writing in english medium education postgraduate programs: A grounded theory approach. Journal of Educators Online, 21(… find
  14. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. 2021. Unsolved problems in ML safety. arXiv preprint, abs/2109.13916. find
  15. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36:24678–24704.
  16. Jianchao Ji, Yutong Chen, Mingyu Jin, Wujiang Xu, Wenyue Hua, and Yongfeng Zhang. 2024. Moralbench: Moral evaluation of llms. arXiv preprint, abs/2406.04428. — [arXiv:2406.04428](https://arxiv.org/abs/2406.04428) find
  17. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  18. Tao Jiang, Xu Yuan, Yuan Chen, Ke Cheng, Liangmin Wang, Xiaofeng Chen, and Jianfeng Ma. 2022. Fuzzydedup: Secure fuzzy deduplication for cloud storage. IEEE Transactions on Dependable and Secure Computing, 20(3):2466–2483. find
  19. Gautam Siddharth Kashyap, Mark Dras, and Usman Naseem. 2025. Too helpful, too harmless, too honest or just right? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29711–29722. find
  20. Iason Gabriel. 2020. Artificial intelligence, values, and alignment. Minds and Machines, 30(3):411–437. find
  21. Maurice G. Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1-2):81–93. find
  22. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369.
  23. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 3214–3252.
  24. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120.
  25. Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of opportunity in supervised learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 3323–3331.
  26. Lukas Helff, Felix Friedrich, Manuel Brack, Patrick Schramowski, and Kristian Kersting. 2024. Llavaguard: Vlm-based safeguard for vision dataset curation and safety assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8322–8326. find
  27. Utsav Maskey, Mark Dras, and Usman Naseem. 2025. Should llm safety be more than refusing harmful instructions? arXiv preprint arXiv:2506.02442. — [arXiv:2506.02442](https://arxiv.org/abs/2506.02442) find
  28. Reem Masoud, Ziquan Liu, Martin Ferianc, Philip C Treleaven, and Miguel Rodrigues Rodrigues. 2025. Cultural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions. In Proceedings of the 31st International Conference on Computational Linguistics, pages 847…
  29. Afrozah Nadeem, Mark Dras, and Usman Naseem. 2025. Steering towards fairness: Mitigating political bias in llms. In associated with The 15th International Conference on Recent Advances in Natural Language Processing RANLP’2025, page 52. find
  30. Usman Naseem, Gautam Siddharth Kashyap, Kaixuan Ren, Yiran Zhang, Utsav Maskey, Juan Ren, and Afrozah Nadeem. 2025. Alignment of large language models with human preferences and values. In Proceedings of the 23rd Annual Workshop of the Australasian Language Technology Association, pages 245–245. find
  31. Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, and Oscar Wahltinez. 2024. Shieldgemma: Generative ai content moderation based on gemma. Preprint, arXiv:2407.21772. — [arXiv:2407.21772](https://arxiv.org/abs/2407.21772) find
  32. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  33. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  34. Caitlin Sadowski and Greg Levin. 2007. Simhash: Hash-based similarity detection. Technical report, Technical report, Google. find
  35. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? arXiv preprint, abs/2303.17548.
  36. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  37. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  38. Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. 2010. Mining multi-label data. In Oded Maimon and Lior Rokach, editors, Data Mining and Knowledge Discovery Handbook, Data Mining and Knowledge Discovery Handbook, pages 667–685. Springer. find
  39. Sibo Wang, Xiangkui Cao, Jie Zhang, Zheng Yuan, Shiguang Shan, Xilin Chen, and Wen Gao. 2024. Vlbiasbench: A comprehensive benchmark for evaluating bias in large vision-language model. arXiv preprint arXiv:2406.14194. — [arXiv:2406.14194](https://arxiv.org/abs/2406.14194) find
  40. Yueqin Yin, Shentao Yang, Yujia Xie, Ziyi Yang, Yuting Sun, Hany Awadalla, Weizhu Chen, and Mingyuan Zhou. 2025. Segmenting text and learning their rewards for improved rlhf in language model. Preprint, arXiv:2501.02790. — [arXiv:2501.02790](https://arxiv.org/abs/2501.02790) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/naseem-2026-do-large-language-models-reflect/