Elamur Research⌕
☰
home / papers / 2025 / On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena

On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena

Paper · 2025 · Tarek Naous and Wei Xu

Summary

Earlier work found that multilingual LMs handle Western entities better than Arab ones when working in Arabic. This paper asks where that gap comes from. The authors built CAMeL-2, a parallel Arabic-English benchmark. It has 58,086 culturally annotated entities across seven types and 367 masked natural contexts. Entities came from Wikipedia country categories and from OpenStreetMap, were labelled by native speakers, and were paired with English forms. The contexts were taken from X/Twitter and translated by hand. The authors tested causal LMs (Llama-3.3, Aya-23, Qwen-2.5, AceGPT, JAIS) and encoders (XLM-R, ARBERT, MARBERT, CAMeLBERT, AraBERT) on text-infilling, extractive QA and NER.

The cultural gap is much smaller in English than in Arabic. Much of the bias traces to language-internal factors, not only to missing data:

  • Arab entities that are very frequent in pre-training often also have an everyday word sense (polysemy).
  • Some overlap lexically with other Arabic-script languages.
  • Frequency-based tokenizers often turn them into a single ambiguous token, and this gets worse with larger Arabic vocabularies.

This matters because better data balance alone will not remove these failures.

Key points

  • Dataset size: CAMeL-2 has 58,086 entities, 2.86× the original CAMeL's 20,249. Author entities grew 11.05×. Locations reach 35,200, including 23,765 Arab locations from OSM, compared with only 1,061 Arab locations in CAMeL.
  • Annotation and translation: two native Arabic annotators reached Cohen's Kappa 0.825, with an adjudication step. English counterparts were found automatically for 88.34% of entities. The rest were transliterated manually using locally preferred spellings, for example Tunisian "Mdhilla".
  • Contexts: the 367 contexts combine 250 short CAMeL infilling contexts with 117 new, longer QA contexts in which the entity is mentioned only indirectly. All were translated by hand into English.
  • Performance pattern: models do worst on Arab entities that are very frequent in Arabic data, are polysemous (e.g., Makloube also means "flipped"), or overlap with languages like Farsi and Urdu.
  • Tokenization: single-token polysemous entities score lowest, and multi-token entities do better. Very large Arabic vocabularies hurt, and models with medium-sized vocabularies perform best.

Interesting ideas

  • I1. Run the same task on parallel text in two languages and compare the error gaps, so language-specific causes are separated from knowledge gaps.
  • I2. Swap one culturally marked entity for a single-sense counterpart in an identical context (the Makloube vs Lasagna pair) to isolate polysemy as the cause of failure.
  • I3. Analyse errors by whether an entity name collides with a common word sense, rather than by frequency alone.
  • I4. Link the number of tokens per entity to accuracy, which shows that a single frequent token is the riskiest case.
  • I5. Harvest typed entity lists through recurring Wikipedia category patterns ("[country] cuisine") and through OSM bilingual place names.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): build parallel negative-control pairs (same frame, different surface entity) to check whether a parser error comes from the entity word or from the structure.
  • I2 (answer): create minimal-pair test items for fables and word problems where only an ambiguous noun changes, to show whether the reader truly resolves the referent.
  • I3 (tokens): flag lemmas that are both proper names and common words (like "Ben" or "Bill") so the tagger considers the name reading explicitly instead of trusting frequency.
  • I4 (tokens): log errors by how entities are segmented and treat single-token ambiguous names as a known weak spot for targeted repair rules.
  • I5 (world): fill the ontology and gazetteer with typed, culture-tagged entities cheaply, with bilingual labels useful for translation through the meta-world.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgia Institute of TechnologyAtlanta · United States133 works · 24,497 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksBias, fairness & safety · 517 works

Cite

Cite as Tarek Naous and Wei Xu. 2025. On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.naacl-long.326

@inproceedings{naous2025origin,
  title = {{On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena}},
  author = {Tarek Naous and Wei Xu},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.326},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.326}
}

References

58 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. arXiv preprint arXiv:2101.01785. ↗
  2. Haozhe An and Rachel Rudinger. 2023. Nichelle and Nancy: The influence of demographic attributes and tokenization length on first name biases. arXiv preprint arXiv:2305.16577. ↗
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for arabic language understanding. In LREC Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  4. Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee, Mohit Iyyer, and Eunsol Choi. 2024. CaLMQA: Exploring culturally specific long-form question answering across 23 languages. arXiv preprint arXiv:2406.17761. ↗
  5. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Kelly Marchisio, Sebastian Ruder, et al. 2024. Aya 23: Open weight releases to further multilingual progress. arXiv preprint arXiv:2405.15032. ↗
  6. Yassine Benajiba, Paolo Rosso, and José Miguel Benedíruiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In Computational Linguistics and Intelligent Text Processing: 8th International Conference, CICLing 2007, Mexico City, Mexico, February 18-24, 2007. Proceedings 8, pages 143–153. Springer. find
  7. Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah A. Smith, and Luke Zettlemoyer. 2024. Breaking the curse of multilinguality with cross-lingual expert language models. arXiv preprint arXiv:2401.10440. ↗
  8. Kaj Bostrom and Greg Durrett. 2020. Byte Pair Encoding is suboptimal for language model pretraining. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617–4624, Online. Association for Computational Linguistics.
  9. Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Ashutosh Dwivedi, Alham Fikri Aji, Jacki O’Neill, Ashutosh Modi, and Monojit Choudhury. 2024. Towards measuring and modeling “culture”’ in LLMs: A survey. arXiv preprint arXiv:2403.15412. ↗
  10. Yu Ying Chiu, Liwei Jiang, Maria Antoniak, Chan Young Park, Shuyue Stella Li, Mehar Bhatia, Sahithya Ravi, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2024. CulturalTeaming: AI-assisted interactive red-teaming for challenging LLMs’(lack of) multicultural knowledge. arXiv preprint arXiv:2404.06664. ↗
  11. Badr AlKhamissi, Muhammad ElNokrashy, Mai AlKhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. arXiv preprint arXiv:2402.13231. ↗
  12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  13. Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate Arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1070–1074.
  14. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  15. Yi Fung, Tuhin Chakrabarty, Hao Guo, Owen Rambow, Smaranda Muresan, and Heng Ji. 2023. NORM-SAGE: Multi-lingual multi-cultural norm discovery from conversations on-the-fly. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15217–15230. find
  16. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, pages 1–79. ↗
  17. Vagrant Gautam, Arjun Subramonian, Anne Lauscher, and Os Keyes. 2024. Stop! In the name of flaws: Disentangling personal names and sociodemographic attributes in NLP. arXiv preprint arXiv:2405.17159. ↗
  18. Gene Gragg. 2019. Semitic and Afro-Asiatic. In The Semitic Languages, pages 22–48. Routledge. ↗
  19. Jing Huang and Diyi Yang. 2023. Culturally aware natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7591–7609. ↗
  20. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678. ↗
  21. Sullam Jeoung, Jana Diesner, and Halil Kilicoglu. 2023. Examining the causal effect of first names on language models: The case of social commonsense reasoning. arXiv preprint arXiv:2306.01117. ↗
  22. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pages 15696–15707. PMLR. ↗
  23. Amr Keleg and Walid Magdy. 2023. DLAMA: A framework for curating culturally diverse facts for probing the knowledge of pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6245–6266. ↗
  24. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
  25. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
  26. Huihan Li, Arnav Goel, Keyu He, and Xiang Ren. 2024a. Attributing culture-conditioned generations to pre-training corpora. Preprint, arXiv:2412.20760. ↗
  27. Wenyan Li, Xinyu Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders Søgaard, Daniel Hershcovich, and Desmond Elliott. 2024b. FoodieQA: A multimodal dataset for fine-grained understanding of chinese food culture. arXiv preprint arXiv:2406.11030. ↗
  28. Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang. 2020. Bond: BERT-assisted open-domain named entity recognition with distant supervision. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1054– 1064. find
  29. Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023. Xlm-v: Overcoming the vocabulary bottleneck in multilingual masked language models. arXiv preprint arXiv:2301.10472. ↗
  30. Juhao Liang, Zhenyang Cai, Jianqing Zhu, Huang Huang, Kewei Zong, Bang An, Mosen Alharthi, Juncai He, Lian Zhang, Haizhou Li, et al. 2024. Alignment at pre-training! towards native alignment for Arabic llms. arXiv preprint arXiv:2412.03253. ↗
  31. Chen Cecilia Liu, Iryna Gurevych, and Anna Korhonen. 2024. Culturally aware and adapted NLP: A taxonomy and a survey of the state of the art. arXiv preprint arXiv:2406.03930. ↗
  32. Reem I Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. 2023. Cultural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions. arXiv preprint arXiv:2309.12342. ↗
  33. Anjishnu Mukherjee, Aylin Caliskan, Ziwei Zhu, and Antonios Anastasopoulos. 2024. Global gallery: The fine art of painting culture portraits through multilingual instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6398–6415. find
  34. Tarek Naous, Michael Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393.
  35. Huy Nghiem, John Prindle, Jieyu Zhao, and Hal Daumé III. 2024. "you gotta be a doctor, Lin": An investigation of name-based bias of large language models in employment recommendations. arXiv preprint arXiv:2406.12232. ↗
  36. Shramay Palta and Rachel Rudinger. 2023. FORK: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9952–9962. ↗
  37. Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D Manning. 2018. Universal dependency parsing from scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 160–170.
  38. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67. ↗
  39. Michael Ramscar. 2019. Source codes in human communication. arXiv preprint arXiv:1904.03991. ↗
  40. Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2024. Normad: A benchmark for measuring the cultural adaptability of large language models. arXiv preprint arXiv:2404.12464. ↗
  41. Dwight F Reynolds. 2015. The Cambridge companion to modern Arab culture. Cambridge University Press. find
  42. Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. ↗
  43. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, et al. 2023. JAIS and JAIS-Chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗
  44. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5668–5680. find
  45. Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Raya Horesh, Rogério Abreu de Paula, Diyi Yang, et al. 2024. Culturebank: An online community-driven knowledge base towards culturally aware language technologies. arXiv preprint arXiv:2404.15238. ↗
  46. Shivalika Singh, Freddie Vargus, Daniel D’souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O’Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Chien, Sebastian Ruder, Surya Guthikonda, Emad Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. 2024. Aya dataset: An open-access collection for multilingual instruction tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11521–11567, Bangkok, Thailand. Association for Computational Linguistics.
  47. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast WordPiece tokenization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2089–2103, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  48. Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. 2024. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint arXiv:2402.07827. ↗
  49. Willy Van Langendonck. 2007. Theory and typology of proper names. Mouton de Gruyter. ↗
  50. Kees Versteegh. 2014. Arabic language. Edinburgh University Press. find
  51. Robert Wolfe and Aylin Caliskan. 2021. Low frequency names exhibit bias and overfitting in contextualizing language models. arXiv preprint arXiv:2110.00672. ↗
  52. Agrima Seth, Sanchit Ahuja, Kalika Bali, and Sunayana Sitaram. 2024. DOSA: A dataset of social artifacts from different indian geographical subcultures. arXiv preprint arXiv:2403.14651. ↗
  53. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498. ↗
  54. Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Soujanya Poria, and Rada Mihalcea. 2024. Understanding the capabilities and limitations of large language models for cultural commonsense. ↗
  55. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. ↗
  56. Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. 2022. GeoM-LAMA: Geo-diverse commonsense probing on multilingual pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2039–2055. find
  57. Li Zhou, Taelin Karidi, Nicolas Garneau, Yong Cao, Wanlong Liu, Wenyu Chen, and Daniel Hershcovich. 2024. Does mapo tofu contain coffee? probing llms for food-related cultural knowledge. arXiv preprint arXiv:2404.06833. ↗
  58. Jianqing Zhu, Huang Huang, Zhihang Lin, Juhao Liang, Zhengyang Tang, Khalid Almubarak, Abdulmohsen Alharthik, Bang An, Juncai He, Xiangbo Wu, et al. 2024. Second language (Arabic) acquisition of llms via progressive vocabulary expansion. arXiv preprint arXiv:2412.12310. Entity Type Wikipedia Categories Arabic English Authors  . KA¿ ð H. AJ» [adjective] HAJ [adjective] àñJ K@ð P [adjective] writers [adjective] authors Beverage [adjective] qJ.¢Ó [adjective] cuisine Food [adjective] qJ.¢Ó [adjective] cuisine [adjective] àñJ ƒAJ ƒ [adjective] àñJ “AK P [adjective] àñÊJÜØ  . KA¿ ð H. AJ» [adjective] HAJ [adjective] àñJ K@ð P [adjective] politicians [adjective] athletes [adjective] actors [adjective] writers [adjective] authors Religious [country] ú ¯ Yg. A‚Ó [country] ú ¯  A J» mosques in [country] churches in [country] Sports Clubs [country] ú ¯ ÐY ¯  èQ» éK  Y K @ football clubs in [country] Names CAMeL-2: Additional Details ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/naous-2025-origin-cultural-biases-language-models/