Elamur Research⌕
☰
home / papers / 2026 / Are Social Biases in LLMs Consistent across Generative Tasks? A Case Study for Basque

Are Social Biases in LLMs Consistent across Generative Tasks? A Case Study for Basque

Summary

Most evaluations of social bias in LLMs use multiple-choice formats. These miss the subtler stereotypes that can appear when models write freely, and the problem is worse for low-resource languages such as Basque, which have few culturally adapted test resources. The authors first reworked the BasqBBQ dataset with social-research experts into BasqBBQ-Exp. They updated categories, localized the stereotypes and added a Feminism dimension. They then turned its ambiguous-context, negative-question items into prompts for twelve generative tasks, which together form BasqBBG. Experts annotated outputs from four LLMs on story continuation and generative QA, and these labels were used to validate an LLM-as-a-Judge setup. The judge agreed closely with the humans. It was then applied to all twelve tasks and five Basque-capable models. Bias levels differed considerably between tasks and depended on model family more than on model size. Llama-based systems were the most biased and least consistent, while GPT-4o and Kimu-9B stayed much fairer. The study argues that bias auditing should account for both the task and the language.

Key points

  • BasqBBQ-Exp contains 243 templates spread over eight categories: Feminism, Gender Identity, Age, Physical Appearance, Ability and Health, SES, Sexual Orientation, and Origin and Ethnicity. Gender Identity is the largest category (56 templates) and Origin and Ethnicity is second (41).
  • BasqBBG covers twelve tasks, including:
    • educational objectives, headlines and moral judgment
    • NLI, playwriting, poetry and reasoning QA
    • roleplay, social media posts and summarization
    • story continuation and generative QA
  • Agreement with expert annotators: the LLM-as-a-Judge reached AC1 = 0.78 for whether bias is present and AC1 = 0.92 for its direction (positive or negative stereotype).
  • Bias rates by model: Llama-based models showed roughly 45–50% biased outputs. GPT-4o and the Gemma-based Kimu-9B stayed at or below 20%.
  • Availability: the benchmark, prompts and evaluation scripts are released at github.com/orai-nlp/BasqBBG.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/52zk8uyjrw5k

@inproceedings{zulaika2026are,
  title = {{Are Social Biases in LLMs Consistent across Generative Tasks? A Case Study for Basque}},
  author = {Zulaika, Muitze and Saralegi, Xabier and Shershneva, Julia and Gonzalez, Lia and Fullaondo, Arkaitz},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/52zk8uyjrw5k},
  url = {https://doi.org/10.63317/52zk8uyjrw5k}
}

References

34 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. find our card
  2. Elizabeth Badinter. 1981. Existe el amor maternal. find
  3. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. find
  4. Zhibo Chu, Zichong Wang, and Wenbin Zhang. 2024. Fairness in large language models: A taxonomic survey. ACM SIGKDD explorations newsletter, 26(1):34–48. find
  5. Ander Corral, Ixak Sarasua, and Xabier Saralegi. 2024. Pipeline analysis for developing instruct llms in low-resource languages: A case study on basque. arXiv preprint arXiv:2412.13922. find
  6. Susan Faludi. 1993. Reacción. La guerra no declarada contra la mujer moderna. Editorial Anagrama, Barcelona. find
  7. Susan Faludi. 2009. La pesadilla terrorista. Miedo y fantasía en Estados Unidos después del 11-S. Editorial Anagrama, Barcelona. find
  8. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097–1179. find our card
  9. Leo Gao, Jonathan Tow, Stella Biderman, Shawn Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jasmine Hsu, Kyle McDonell, Niklas Muennighoff, et al. 2021. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 10:8–9. find
  10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  11. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge. find
  12. Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48. find
  13. Arlie Russell Hochschild. 2008. La mercantilización de la vida íntima: Apuntes de la casa y el trabajo, volume 3045. Katz editores. find
  14. Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in nlp models as barriers for persons with disabilities. arXiv preprint arXiv:2005.00813. find our card
  15. Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807. — [arXiv:1609.05807](https://arxiv.org/abs/1609.05807) find
  16. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. arXiv preprint arXiv:1906.07337. find our card
  17. Marcela Lagarde et al. 2016. Los cautiverios de las mujeres: madresposas, monjas, putas, presas y locas. Siglo XXI Editores México. find
  18. Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896. find
  19. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744. find
  20. Oscar Sainz, Naiara Perez, Julen Etxaniz, Joseba Fernandez de Landa, Itziar Aldabe, Iker García-Ferrero, Aimar Zabala, Ekhi Azurmendi, German Rigau, Eneko Agirre, Mikel Artetxe, and Aitor Soroa. 2025. Instructing large language models for low-resource languages: A systematic study for basque. find
  21. Ixak Sarasua, Ander Corral, and Xabier Saralegi. 2025. Diploma: Efficient adaptation of instructed llms to low-resource languages via post-training delta merging. In Findings of the Association for Computational Linguistics: EMNLP 2025. find
  22. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  23. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2024. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. arXiv preprint arXiv:2406.12624. find
  24. Monique Wittig. 1992. El pensamiento heterosexual y otros ensayos.(2006) madrid: Egales. Traducción Javier Sáez y Paco Vidarte. find
  25. Naomi Wolf and Cristina Reynoso. 1992. El mito de la belleza. Debate feminista, 5:209–219. find
  26. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595– 46623. 8. find
  27. Language Resource References Dhamala, Jwala and Sun, Tony and Kumar, Varun and Krishna, Satyapriya and Pruksachatkun, Yada and Chang, Kai-Wei and Gupta, Rahul. 2021. BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation. ACM, FAccT ’21. find
  28. Yufei Huang and Deyi Xiong. 2023. CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Models. find
  29. Jin, Jiho and Kang, Woosung and Myung, Junho and Oh, Alice. 2025. Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluations. Association for Computational Linguistics. find
  30. Jin, Jiho and Kim, Jiseon and Lee, Nayeon and Yoo, Haneul and Oh, Alice and Lee, Hwaran. 2024. KoBBQ: Korean bias benchmark for question answering. MIT Press One Broadway, 12th Floor, Cambridge, Massachusetts 02142, USA . . . . find
  31. Li, Tao and Khot, Tushar and Khashabi, Daniel and Sabharwal, Ashish and Srikumar, Vivek. 2020. UNQOVERing stereotyping biases via underspecified questions. find our card
  32. Nangia, Nikita and Vania, Clara and Bhalerao, Rasika and Bowman, Samuel R. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. find our card
  33. Parrish, Alicia and Chen, Angelica and Nangia, Nikita and Padmakumar, Vishakh and Phang, Jason and Thompson, Jana and Htut, Phu Mon and Bowman, Samuel. 2022. BBQ: A hand-built bias benchmark for question answering. Association for Computational Linguistics. find our card
  34. Zulaika, Muitze and Saralegi, Xabier. 2025. BasqBBQ: A QA Benchmark for Assessing Social Biases in LLMs for Basque, a Low-Resource Language. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zulaika-2026-are-social-biases-llms-consistent/