Elamur Research⌕
☰
home / papers / 2026 / WinoQueer-NL: Assessing Bias in Dutch Language Models toward LGBTQ+ Identities

WinoQueer-NL: Assessing Bias in Dutch Language Models toward LGBTQ+ Identities

Paper · 2026 · Jiska Beuk and Gerasimos Spanakis

Summary

Studies of anti-queer bias in language models have mostly targeted English, so Dutch models have gone largely untested. The authors rebuilt the English WinoQueer benchmark for Dutch. They split it into its parts (templates, first names, identity labels, stereotype predicates), translated and checked each part separately, replaced the names with Dutch ones and then reassembled the sentences. Four LGBTQ+ native speakers reviewed the translated predicates. An anonymous survey of 43 Dutch queer respondents then judged which stereotypes are relevant and how harmful they are, and also proposed new ones. The released WinoQueer-NL contains 42,906 sentence pairs. In each pair, a stereotypical sentence and its counterfactual differ only in the identity term. Dutch and multilingual MLMs and ARLMs were scored by how often they assign higher log-likelihood to the stereotypical sentence. Averaged over everything, the scores look neutral, around 50%. Per identity, however, transgender and non-binary pairs get the most biased scores, up to 97%, while gay pairs can be as low as 6%. The work shows that a single aggregate number can hide harm aimed at specific groups and that benchmarks need cultural grounding.

Key points

  • Source benchmark: English WinoQueer, with 45,540 pairs built as a Cartesian product of 11 templates, 60 names, 9 identity labels and 177 predicates.
  • Translation and cleanup:
    • Reviewers removed six predicates, either as near-duplicates or because they had no natural Dutch rendering or were ungrammatical, which left 171.
    • Survey respondents confirmed 145 of these 171 as culturally relevant and contributed 22 new stereotypes.
    • The final set has 167 stereotypes.
  • Counterfactual swaps:
    • Transgender/Nonbinary are swapped with Cis/Cisgender.
    • Gay, lesbian, bisexual, pansexual and asexual are swapped with straight/heterosexual.
    • The umbrella labels are paired with all of these.
  • Thirteen models were evaluated, including BERTje, RobBERT, mBERT, XLM-RoBERTa, GPT2 Dutch, Schaapje 2B, Fietje-2, Llama-3.2 and Phi-4. Identities that are overrepresented in usage-weighted scores tended to receive lower bias scores.
  • Human harmfulness ratings hardly relate to the model's bias gap. All correlations satisfy |r| < 0.12, and the strongest (XLM-RoBERTa Base, r ≈ 0.119) accounts for less than 1.5% of variance.

Interesting ideas

  • I1. Break a template dataset into independent slots, translate and check each slot on its own, then recombine; this keeps a pair's only difference exactly where you want it.
  • I2. Minimal pairs that differ in a single token isolate the effect of that token on a model's preference.
  • I3. Reporting an overall average next to per-group scores exposes cancellation, where an average of about 50% hides extremes of 6% and 97%.
  • I4. Correlate per-item human judgments with per-item model deltas to check whether an assumed driver (here, harmfulness) really explains model behaviour.
  • I5. Have the affected community validate items and add new ones, and drop items that do not carry over culturally instead of forcing them through.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Translation through the meta-world could be tested on template-generated sentence families, so that a mistranslation can be traced to one slot rather than to a whole sentence.
  • I2 (learn): Evaluation with negative controls could swap one entity or role word and confirm that frame or answer outputs change only where they should.
  • I3 (learn): Evaluation reports could always break scores down by construction type or verb class, so that an aggregate LAS or accuracy cannot mask a failing subgroup.
  • I4 (learn): Witness-parser disagreement or teacher-tier confidence could be correlated with actual error rates to test whether these signals deserve their weight.
  • I5 (answer): Fable and school-test QA sets adapted from other sources could be reviewed by target-audience readers, with culture-specific items removed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Maastricht UniversityMaastricht · Netherlands6 works · 605 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksPsychology · 1,552 worksSafety Research · 543 worksSocial Psychology · 306 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Jiska Beuk and Gerasimos Spanakis. 2026. WinoQueer-NL: Assessing Bias in Dutch Language Models toward LGBTQ+ Identities. arXiv:2609.02651.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2609.02651 · arXiv 2609.02651

@misc{beuk2026winoqueer,
  title = {{WinoQueer-NL: Assessing Bias in Dutch Language Models toward LGBTQ+ Identities}},
  author = {Jiska Beuk and Gerasimos Spanakis},
  year = {2026},
  doi = {10.48550/arxiv.2609.02651},
  eprint = {2609.02651},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2609.02651}
}

References

46 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. 2025. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743. ↗
  2. Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021. RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1941–1955, Online. Association for Computational Linguistics. find
  3. Marion Bartl, Malvina Nissim, and Albert Gatt. 2020. Unmasking contextual stereotypes: Measuring and mitigating BERT’s gender bias. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 1–16, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  4. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. ↗
  5. Rodrigo Alejandro Chávez Mulsa and Gerasimos Spanakis. 2020. Evaluating bias in Dutch word embeddings. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 56–71, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  7. Wietse de Vries and Malvina Nissim. 2021. As good as new. how to successfully recycle English GPT-2 to make models for other languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 836–846, Online. Association for Computational Linguistics.
  8. Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTje: A Dutch BERT Model. arXiv:1912.09582. ↗
  9. Pieter Delobelle, Ewoenam Tokpo, Toon Calders, and Bettina Berendt. 2022. Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1693–1706, Seattle, United States. Association for Computational Linguistics. find
  10. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097– 1179. ↗
  11. Vagrant Gautam, Arjun Subramonian, Anne Lauscher, and Os Keyes. 2024. Stop! in the name of flaws: Disentangling personal names and sociodemographic attributes in NLP. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 323–337, Bangkok, Thailand. Association for Computational Linguistics. find
  12. Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. RobBERT: a Dutch RoBERTa-based Language Model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3255–3265, Online. Association for Computational Linguistics.
  13. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguistics. ↗
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  15. Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metrics do not correlate with application bias. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1926–1940, Online. Association for Computational Linguistics.
  16. Harnoor Dhingra, Preetiha Jayashanker, Sayali Moghe, and Emma Strubell. 2023. Queer people are people first: Deconstructing sexual identity stereotypes in large language models. arXiv preprint arXiv:2307.00101. ↗
  17. Yeb Havinga. 2022. Gpt2-large-dutch. https://huggingface.co/yhavinga/gpt2-large-dutch. GPT-2 large model trained from scratch on cleaned Dutch mC4 dataset. ↗
  18. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73. find
  19. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  20. Allard R Feddes and Kai J Jonas. 2020. Associations between dutch lgbt hate crime experience, well-being, trust in the police and future hate crime reporting. Social Psychology. ↗
  21. Virginia Felkner, Ho-Chun Herbert Chang, Eugene Jang, and Jonathan May. 2023. WinoQueer: A community-in-the-loop benchmark for anti-LGBTQ+ bias in large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9126– 9140, Toronto, Canada. Association for Computational Linguistics. find
  22. Gert Hekma and Jan Willem Duyvendak. 2011. Queer netherlands: A puzzling example. Sexualities, 14(6):625–631. ↗
  23. Akshita Jha, Aida Davani, Chandan K. Reddy, Shachi Dave, Vinodkumar Prabhakaran, and Sunipa Dev. 2023. SeeGULL: A stereotype benchmark with broad geo-cultural coverage leveraging generative models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9851–9870, Toronto, Canada. Association for Computational Linguistics. ↗
  24. Jiho Jin, Jiseon Kim, Nayeon Lee, Haneul Yoo, Alice Oh, and Hwaran Lee. 2024. KoBBQ: Korean bias benchmark for question answering. Transactions of the Association for Computational Linguistics, 12:507–524. ↗
  25. Faisal Kamiran and Toon Calders. 2012. Data preprocessing techniques for classification without discrimination. Knowledge and information systems, 33(1):1– 33. ↗
  26. Masahiro Kaneko, Aizhan Imankulova, Danushka Bollegala, and Naoaki Okazaki. 2022. Gender bias in masked language models for multiple languages. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2740–2750, Seattle, United States. Association for Computational Linguistics. ↗
  27. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108. ↗
  28. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166–172, Florence, Italy. Association for Computational Linguistics.
  29. Robin Smits. 2025. Schaapje-2B-Pretrained. https://huggingface.co/robinsmits/ Schaapje-2B-Pretrained. Continual pretraining on Dutch data based on IBM Granite 3.0 2B Instruct model. ↗
  30. Anne Lauscher, Archie Crowley, and Dirk Hovy. 2022. Welcome to the modern world of pronouns: Identity-inclusive natural language processing beyond gender. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1221– 1232, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. ↗
  31. Sharon Levy, Neha John, Ling Liu, Yogarshi Vyas, Jie Ma, Yoshinari Fujinuma, Miguel Ballesteros, Vittorio Castelli, and Dan Roth. 2023. Comparing biases and the impact of multilingual training across multiple languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10260–10280, Singapore. Association for Computational Linguistics. ↗
  32. Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020. Towards debiasing sentence representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5502–5515, Online. Association for Computational Linguistics. ↗
  33. Tyler J Loftus, Jeremy A Balch, Kenneth L Abbott, Die Hu, Matthew M Ruppert, Benjamin Shickel, Tezcan Ozrazgat-Baslanti, Philip A Efron, Patrick J Tighe, William R Hogan, et al. 2024. Community-engaged artificial intelligence research: A scoping review. PLOS Digital Health, 3(8):e0000561. ↗
  34. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for Computational Linguistics. ↗
  35. Vera Neplenbroek, Arianna Bisazza, and Raquel Fernández. 2024. Mbbq: A dataset for cross-lingual comparison of stereotypes in generative llms. arXiv preprint arXiv:2406.07243. ↗
  36. Aurélie Névéol, Yoann Dupont, Julien Bezançon, and Karën Fort. 2022. French CrowS-pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8521–8531, Dublin, Ireland. Association for Computational Linguistics.
  37. Mae Sosto and Alberto Barrón-Cedeño. 2024. Queerbench: Quantifying discrimination in language models toward queer identities. arXiv preprint arXiv:2406.12399. ↗
  38. Elza Strazda and Gerasimos Spanakis. 2025. Dutch CrowS-pairs: Adapting a challenge dataset for measuring social biases in language models for Dutch. In Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing -Natural Language Processing in the Generative AI Era, pages 1195–1204, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. find
  39. Kunsheng Tang, Wenbo Zhou, Jie Zhang, Aishan Liu, Gelei Deng, Shuai Li, Peigui Qi, Weiming Zhang, Tianwei Zhang, and Nenghai Yu. 2024. Gender-CARE: A comprehensive framework for assessing and reducing gender bias in large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1196–1210. find
  40. Nenad Tomasev, Kevin R McKee, Jackie Kay, and Shakir Mohamed. 2021. Fairness for unobserved characteristics: Insights from technological impacts on queer communities. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 254–265. find
  41. UNESCO and International Research Centre on Artificial Intelligence (IRCAI). 2024. Challenging systematic prejudices: An investigation into bias against women and girls in large language models. Technical report, UNESCO and IRCAI. find
  42. Jens van der Weide, Dong Nguyen, Marianne Schaaphok, and Roos M. Bakker. 2026. CrowS-pairs-NL: A benchmark to evaluate Dutch stereotype bias in LLMs. In Proceedings of the 1st Workshop on Stereotypes Across Cultures in Language Technologies (StereACuLT 2026), pages 1–12, San Diego, California, United States. Association for Computational Linguistics.
  43. Bram Vanroy. 2024. Fietje: An open, efficient LLM for Dutch. Preprint, arXiv:2412.15450. ↗
  44. Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335– 340. find
  45. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  46. Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang. 2019. Examining gender bias in languages with grammatical gender. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5276–5284, Hong Kong, China. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/beuk-2026-winoqueer-nl-assessing-bias-dutch/