Elamur Research⌕
☰
home / papers / 2025 / BasqBBQ: A QA Benchmark for Assessing Social Biases in LLMs for Basque, a Low-Resource Language

BasqBBQ: A QA Benchmark for Assessing Social Biases in LLMs for Basque, a Low-Resource Language

Paper · 2025 · Muitze Zulaika and Xabier Saralegi

Summary

Tools for measuring social bias in language models exist mostly for English and a few other major languages, so there has been no way to check whether LLMs reproduce stereotypes when working in Basque. The authors build BasqBBQ, a Basque multiple-choice QA test set derived from the English BBQ benchmark (Parrish et al., 2022). They translated its templates and adjusted them to the cultural setting of the Basque Country. The set covers eight bias categories, each with ambiguous items (where "unknown" is correct) and disambiguated items. They test open autoregressive LLMs, both multilingual ones and models continually pre-trained on Basque, in a 4-shot setup. They also compare Llama-3.1-8B with its Basque-adapted derivative Llama-eus-8B on English and Basque versions of U.S.-context and Basque-context data. Larger models answer more accurately, but ambiguous items remain hard for every system. Strong negative bias persists for Disability Status, Age and Physical Appearance. Basque adaptation shifts the bias inherited from the base model only slightly. The resource offers a starting point for auditing fairness in a low-resource language.

Key points

  • BasqBBQ is described as the first Basque bias benchmark. It draws on BBQ (58,000 English questions spanning 9 categories) and narrows the scope to 8 categories: Sexual Orientation, Race/Ethnicity, Gender Identity, Age, Socio-Economy Status, Nationality, Disability Status and Physical Appearance. It is released at github.com/orai-nlp/BasqBBQ.
  • Each template yields four items: an ambiguous and a disambiguated context, each paired with a negative and a positive question. Every item has three answer options: the stereotyped group, the other group, and "unknown".
  • Evaluation is 4-shot multiple-choice within the LM Harness framework. Cross-lingual and cross-cultural pairs (BBQ/BBQ2eu, BasqBBQ2en/BasqBBQ) separate the effect of language from the effect of cultural context.
  • Average bias scores for Llama-3.1-8B on BasqBBQ2en are −17.53 on ambiguous items and −4.85 on unambiguous items. For Llama-eus-8B on BasqBBQ they are −8.56 and −3.12. Average accuracy is 0.36/0.88 versus 0.35/0.71, which reflects the weaker NLU of the adapted model.
  • In the Basque-context data, the lower bias of Llama-eus-8B appears to come from the base model's own behaviour when operating in Basque, not from continual pre-training. The authors note two limitations: no Basque-specific bias dimensions were added, and multiple-choice QA may not reveal all latent bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 5Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksQuestion answering · 441 worksBias, fairness & safety · 517 worksLow-resource languages · 267 works

Cite

Cite as Muitze Zulaika and Xabier Saralegi. 2025. BasqBBQ: A QA Benchmark for Assessing Social Biases in LLMs for Basque, a Low-Resource Language. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zulaika2025basqbbq,
  title = {{BasqBBQ: A QA Benchmark for Assessing Social Biases in LLMs for Basque, a Low-Resource Language}},
  author = {Muitze Zulaika and Xabier Saralegi},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.318/}
}

References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual M…
  3. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. find
  4. Ander Corral, Ixak Sarasua, and Xabier Saralegi. 2024. Llama-eus-8b, a foundational sub-10 billion parameter llm for basque. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  6. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862–872… find
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  8. Julen Etxaniz, Oscar Sainz, Naiara Miguel, Itziar Aldabe, German Rigau, Eneko Agirre, Aitor Ormazabal, Mikel Artetxe, and Aitor Soroa. 2024. Latxa: An open language model and evaluation suite for Basque. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu… find
  9. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, pages 1–79.
  10. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… find
  11. Yufei Huang and Deyi Xiong. 2024. Cbbq: A chinese bias benchmark dataset curated with human-ai collaboration for large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2917–2929. find
  12. Jiho Jin, Jiseon Kim, Nayeon Lee, Haneul Yoo, Alice Oh, and Hwaran Lee. 2024. Kobbq: Korean bias benchmark for question answering. Preprint, arXiv:2307.16778.
  13. Hele-Andra Kuulmets, Taido Purason, Agnes Luhtaru, and Mark Fishel. 2024. Teaching llama a new language through cross-lingual knowledge transfer. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3309–3325. find
  14. Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. 2020. UNQOVERing stereotyping biases via underspecified questions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3475–3489, Online. Association for Computational Linguistics.
  15. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long…
  16. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for…
  17. Vera Neplenbroek, Arianna Bisazza, and Raquel Fernández. 2024. Mbbq: A dataset for cross-lingual comparison of stereotypes in generative llms. arXiv preprint arXiv:2406.07243. — [arXiv:2406.07243](https://arxiv.org/abs/2406.07243) find
  18. Aurélie Névéol, Yoann Dupont, Julien Bezançon, and Karën Fort. 2022. French CrowS-pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol…
  19. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland. Asso…
  20. A Radford. 2018. Improving language understanding by generative pre-training. find
  21. Manon Reusens, Philipp Borchert, Margot Mieskes, Jochen De Weerdt, and Bart Baesens. 2023. Investigating bias in multilingual language models: Cross-lingual transfer of debiasing techniques. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2887–2896, S… find
  22. Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llama beyond english: An empirical study on language capability transfer. arXiv preprint arXiv:2401.01055. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zulaika-2025-basqbbq-qa-benchmark-assessing-social/