Elamur Research⌕
☰
home / papers / 2026 / CrowS-Pairs-NL: A Benchmark to Evaluate Dutch Stereotype Bias in LLMs

CrowS-Pairs-NL: A Benchmark to Evaluate Dutch Stereotype Bias in LLMs

Paper · 2026 · Jens van der Weide, Dong Nguyen, Marianne Schaaphok et al.

Summary

Most stereotype-bias benchmarks for LLMs are English-centric and often reflect American culture, so biases tied to other languages and societies go unmeasured. Dutch is one such case, with few evaluation resources. The authors present CrowS-Pairs-NL, a Dutch version of the CrowS-Pairs minimal-pair benchmark. They built it in two steps. First, they selected a subset of the English data, translated it, and adapted it, using explicit criteria to remove or repair instances with the conceptual flaws identified by Blodgett et al. (2021). Second, they added new sentence pairs written by Dutch-speaking crowdworkers and filtered them manually. They tested six multilingual and Dutch-trained models with two metrics: a pseudo-log-likelihood score adapted for autoregressive models and a prompt-based score. Models trained specifically on Dutch showed stronger preferences for stereotypical sentences, a pattern previously reported for French and Filipino. This suggests that language-specific training adds language-specific bias, which English-only testing would miss. The work argues for culturally grounded bias evaluation of Dutch LLMs.

Key points

  • CrowS-Pairs-NL spans nine social axes, among them nationality, physical appearance, age, gender, ethnicity, sexual orientation, faith, disability and socioeconomic standing. New pairs were crowdsourced via Prolific, paying £12/hr for writing and £9/hr for validation.
  • Filtering is stricter than in the earlier Dutch adaptation by Strazda and Spanakis (2025), which dropped only 45 instances. The likelihood metric follows the autoregressive adaptation by Gamboa and Lee (2025).
  • The prompt metric uses three paraphrased templates and reports their mean and standard deviation to capture sensitivity to wording.
  • Dutch-trained models such as GEITje-7B-Ultra and EuroLLM-9B-Instruct scored highest on both metrics. The prompt metric produced a narrower range of scores (0.48–0.58) than the likelihood metric (0.46–0.69).
  • Reported limitations:
    • Disability, sexual orientation and religion are underrepresented.
    • The likelihood score depends on where the group term appears in the sentence.
    • The annotator pool was 62% male and 80% white.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Utrecht UniversityUtrecht · Netherlands39 works · 1,603 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksBias, fairness & safety · 517 works

Cite

Cite as Jens van der Weide, Dong Nguyen, Marianne Schaaphok and Roos M. Bakker. 2026. CrowS-Pairs-NL: A Benchmark to Evaluate Dutch Stereotype Bias in LLMs. Proceedings of the 1st Workshop on Stereotypes Across Cultures in Language Technologies (StereACuLT 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.stereacult-1.1

@inproceedings{weide2026crows,
  title = {{CrowS-Pairs-NL: A Benchmark to Evaluate Dutch Stereotype Bias in LLMs}},
  author = {Jens {van der Weide}, Dong Nguyen, Marianne Schaaphok and Roos M. Bakker},
  booktitle = {Proceedings of the 1st Workshop on Stereotypes Across Cultures in Language Technologies (StereACuLT 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.stereacult-1.1},
  url = {https://doi.org/10.18653/v1/2026.stereacult-1.1}
}

References

34 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gerrit Bloothooft. 2021. gratieachtergrond. Voornamen met een mi-CBS. 2024. Hoeveel inwoners hebben een herkomst buiten Nederland. Last Accessed: June 2025. find
  2. J. R. Landis and G. G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. ↗
  3. Dejian Yang Haowei Zhang Junxiao Song Ruoyu Zhang et al. DeepSeek-AI, Daya Guo. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.
  4. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Alexandra Sasha Luccioni, Alexander M. Rush, Stella Biderman, Margaret Mitchell, Victor Sanh, Colin Raffel, and et al. 2023. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. ↗
  5. Maria Eriksson, Erasmo Purificato, Arman Noroozian, Joao Vinagre, Guillaume Chaslot, Emilia Gomez, and David Fernandez-Llorca. 2025. Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation. ArXiv:2502.06559 [cs]. ↗
  6. Louis Lippens. 2024. Computer says ’no’: Exploring systemic bias in ChatGPT using an audit approach. Computers in Human Behavior: Artificial Humans, 2(1):100054. ↗
  7. Kathleen Fraser, Svetlana Kiritchenko, and Isar Nejadgholi. 2024. How Does Stereotype Content Differ across Data Sources? In Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024), pages 18–34, Mexico City, Mexico. Association for Computational Linguistics. find
  8. Yang Liu. 2024. Quantifying Stereotypes in Language. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1223– 1240, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  9. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. Bias and Fairness in Large Language Models: A Survey. Computational Linguistics, 50(3):1097– 1179. Place: Cambridge, MA Publisher: MIT Press. ↗
  10. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. EuroLLM: Multilingual Language Models for Europe. ArXiv:2409.16235 [cs]. ↗
  11. Lance Calvin Lim Gamboa and Mark Lee. 2025. Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 123–134, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  12. Margaret Mitchell, Giuseppe Attanasio, Ioana Baldini, Miruna Clinciu, Pieter Delobelle, Manan Dey, Sil Hamilton, and et al. 2025. SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 11995–12041, Albuquerque, New Mexico. Association for Computational Linguistics. find
  13. Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, and Su Lin Blodgett. 2023. This prompt is measuring <mask>: Evaluating bias evaluation in language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2209– 2225, Toronto, Canada. Association for Computational Linguistics. find
  14. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual Generalization through Multitask Finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics. ↗
  15. Rebekka Görge, Michael Mock, and Héctor Allende-Cid. 2025. Detecting Linguistic Indicators for Stereotype Assessment with Large Language Models. ArXiv:2502.19160 [cs]. ↗
  16. Sa-kiera Tiarra Jolynn Hudson, Annalisa Myer, and Elyssa Christine Berney. 2024. Stereotyping, prejudice, and discrimination at the intersection of race and gender: An intersectional theory primer. Social and Personality Psychology Compass, 18(2):e12939. ↗
  17. Meertens Instituut. Nederlandse Voornamenbank - Top-namen land Nederland 2000. find
  18. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5356–5371, Online. Association for Computational Linguistics.
  19. Jiho Jin, Jiseon Kim, Nayeon Lee, Haneul Yoo, Alice Oh, and Hwaran Lee. 2024. KoBBQ: Korean Bias Benchmark for Question Answering. Transactions of the Association for Computational Linguistics, 12:507–524. Place: Cambridge, MA Publisher: MIT Press. ↗
  20. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for Computational Linguistics. ↗
  21. Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Deepak Tunuguntla, and Oskar Van Der Wal. 2022. You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings. In Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models, pages 26–41, virtual+Dublin. Association for Computational Linguistics.
  22. Vera Neplenbroek, Arianna Bisazza, and Raquel Fernández. 2024. Mbbq: A dataset for cross-lingual comparison of stereotypes in generative llms. In Proceedings of COLM 2024. ↗
  23. Llama Team. 2024. The Llama 3 Herd of Models | Research - AI at Meta. find
  24. Bram Vanroy. 2024. GEITje 7B Ultra: A Conversational Model for Dutch. ↗
  25. Aurélie Névéol, Yoann Dupont, Julien Bezançon, and Karën Fort. 2022. French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8521–8531, Dublin, Ireland. Association for Computational Linguistics.
  26. Albert Webson and Ellie Pavlick. 2022. Do Prompt-Based Models Really Understand the Meaning of Their Prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344. Association for Computational Linguistics. ↗
  27. Ruby Ostrow and Adam Lopez. 2025. LLMs Reproduce Stereotypes of Sexual and Gender Minorities. ArXiv:2501.05926 [cs]. ↗
  28. Hitomi Yanaka, Namgi Han, Ryoma Kumon, Jie Lu, Masashi Takeshita, Ryo Sekizawa, Taisei Kato, and Hiromi Arai. 2025. Jbbq: Japanese bias benchmark for analyzing social biases in large language models. In Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 1–17. Association for Computational Linguistics. find
  29. Flor Miriam Plaza-del Arco, Amanda Cercas Curry, Alba Curry, Gavin Abercrombie, and Dirk Hovy. 2024. Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attribution. arXiv. Version Number: 3. ↗
  30. Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang. 2019. Examining Gender Bias in Languages with Grammatical Gender. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5276–5284, Hong Kong, China. Association for Computational Linguistics. ↗
  31. Manon Reusens, Philipp Borchert, Margot Mieskes, Jochen De Weerdt, and Bart Baesens. 2023. Investigating Bias in Multilingual Language Models: Cross-Lingual Transfer of Debiasing Techniques. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2887– 2896, Singapore. Association for Computational Linguistics. find
  32. Edwin Rijgersberg and Bob Lucassen. 2023. GEITje: een groot open Nederlands taalmodel. find
  33. Muitze Zulaika and Xabier Saralegi. 2025. BasqBBQ: A QA Benchmark for Assessing Social Biases in LLMs for Basque, a Low-Resource Language. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4753–4767, Abu Dhabi, UAE. Association for Computational Linguistics.
  34. Nihar Sahoo, Pranamya Kulkarni, Arif Ahmad, Tanu Goyal, Narjis Asad, Aparna Garimella, and Pushpak Bhattacharyya. 2024. IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8786–8806, Mexico City, Mexico. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/weide-2026-crows-pairs-nl-benchmark-evaluate/