Elamur Research⌕
☰
home / papers / 2026 / Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili

Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili

Paper · 2026 · Ruolei Zhang, Teddy Njuguna and Yue Feng

Summary

Bias audits and safety tuning of large language models are carried out mostly in English, on the assumption that the results also hold in other languages. This paper tests that assumption for Swahili. Swahili is a lingua franca of East Africa with over 100 million speakers, but it makes up only about 0.01% of Common Crawl, against 42.6% for English. Using the HolisticBias descriptor-template approach, the authors built 4,900 parallel English–Swahili sentence-completion prompts covering nine demographic axes. The Swahili versions came from the Google Translate API and were checked by native speakers. GPT-5.2 and Gemini 2.5 Flash completed every prompt with greedy decoding. Claude Sonnet 4.5 then judged each output for stereotypes and sentiment, and judged each language pair for semantic similarity. Both models behaved noticeably differently across the two languages in stereotype rates, sentiment, refusals and meaning. A logit-lens probe on Qwen-2.5-3B suggests the two languages are processed along separate internal routes. The practical lesson is that English-only audits leave multilingual deployments insufficiently tested.

Key points

  • Dataset: 50 templates were filled with descriptors from nine axes: race/ethnicity, religion, gender, age, disability, socioeconomic status, education, migration status and nationality. This gave 4,900 prompt pairs and 19,600 completions (9,800 per model, under 30 tokens each, temperature 0).
  • Refusals: GPT-5.2 declined 169 English prompts and none of their Swahili counterparts. The authors note that this gap does not depend on translation quality.
  • Divergence: Stereotype rates differed by up to 12 percentage points on some axes, and Gemini's share of neutral sentiment doubled in Swahili. More than 55% of prompt pairs yielded semantically dissimilar outputs, with the largest gaps on culturally specific axes.
  • Validation: A native annotator reviewed 200 similarity judgments and agreed with the judge 95% of the time. Translation errors were found, such as "Sikh" rendered as Msikh instead of Mkalasinga. The authors argue these errors make the results a conservative lower bound.
  • Mechanistic probe: In Qwen-2.5-3B, Jensen-Shannon divergence between the languages rose from about 0.035 in early layers to 0.15–0.24 in late layers. When the model processed Swahili inputs, the English descriptor tokens ranked on average beyond 40,000th of about 150,000 vocabulary tokens.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of BirminghamBirmingham · United Kingdom27 works · 62,564 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEnglish · 2,379 worksMultilingual & cross-lingual NLP · 882 worksBias, fairness & safety · 587 works

Cite

Cite as Ruolei Zhang, Teddy Njuguna and Yue Feng. 2026. Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili. Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.mellm-1.17

@inproceedings{zhang2026cross,
  title = {{Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili}},
  author = {Ruolei Zhang, Teddy Njuguna and Yue Feng},
  booktitle = {Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.mellm-1.17},
  url = {https://doi.org/10.18653/v1/2026.mellm-1.17}
}

References

23 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. nostalgebraist. 2020. interpreting gpt: the logit lens. find
  2. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4623–4637.
  3. Josh Achiam OpenAI, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2024. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774, 2(6):2. ↗
  4. Matei-Iulian Cocu, Răzvan-Cosmin Cristia, and Adrian Marius Dumitran. 2025. A cross-lingual analysis of bias in large language models using romanian history. arXiv preprint arXiv:2510.02362. — [arXiv:2510.02362](https://arxiv.org/abs/2510.02362) find
  5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  6. Common Crawl. 2026. Statistics of common crawl monthly archives: Distribution of languages. https://commoncrawl.github.io/ cc-crawl-statistics/plots/languages. ↗
  7. Lance Calvin Lim Gamboa, Yue Feng, and Mark Lee. 2025. Social bias in multilingual language models: A survey. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 27845–27868. find
  8. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, and 1 others. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858.
  9. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguisti…
  10. Willem Huijzer and Jieying Chen. 2025. Discrimination by llms: Cross-lingual bias assessment and mitigation in decision-making and summarisation. arXiv preprint arXiv:2509.09735. find
  11. Rudolf Laine, Bilal Chughtai, Jan Betley, Kaivalya Hariharan, Jeremy Scheurer, Mikita Balesni, Marius Hobbhahn, Alexander Meinke, and Owain Evans. 2024. Me, myself, and ai: The situational awareness dataset (sad) for llms. Advances in Neural Information Processing Systems, 37:64010–64118. find
  12. Yuxuan Liang. 2025. Cross-language bias examination in large language models. arXiv preprint arXiv:2512.16029. — [arXiv:2512.16029](https://arxiv.org/abs/2512.16029) find
  13. Vera Neplenbroek, Arianna Bisazza, and Raquel Fernández. 2024. Mbbq: A dataset for cross-lingual comparison of stereotypes in generative llms. arXiv preprint arXiv:2406.07243. — [arXiv:2406.07243](https://arxiv.org/abs/2406.07243) find
  14. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  15. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. Bbq: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105.
  16. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  17. Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022. "i’m sorry to hear that": Finding new biases in language models with a holistic descriptor dataset. In Proceedings of the 2022 conference on empirical methods in natural language processing, pages 9180–92…
  18. J Ben Tamo, Daniel Carlander-Reuterfelt, Jonathan Rubin, Dezhi Hong, Mingxian Wang, and Oleg Polian-nikov. 2026. Linguamap: Which layers of llms speak your language and how to tune them? arXiv preprint arXiv:2601.20009. — [arXiv:2601.20009](https://arxiv.org/abs/2601.20009) find
  19. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  21. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366–15394, Bangkok, T…
  22. Yuemei Xu, Ling Hu, Jiayi Zhao, Zihan Qiu, Kexin Xu, Yuqi Ye, and Hanwen Gu. 2025. A survey on multilingual large language models: Corpora, alignment, and bias. Frontiers of Computer Science, 19(11):1911362. find
  23. Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-cross-lingual-bias-large-language/