Elamur Research⌕
☰
home / papers / 2026 / Cultural and Knowledge Biases in LLMs through the Lens of Entity-Aware Machine Translation

Cultural and Knowledge Biases in LLMs through the Lens of Entity-Aware Machine Translation

Summary

LLMs translate fluently, but they know far less about entities rooted in underrepresented cultures. Earlier cultural MT work also treated all "cultural" entities as one group. The authors study this gap through entity-aware machine translation (EA-MT) and propose a three-tier scale of cultural specificity:

  • Culturally Agnostic: recognisable through direct lexical mapping.
  • Culturally Sensitive: identifiable with general knowledge of both cultures.
  • Culturally Local: requires insider knowledge.

On the XCTranslate benchmark (English into 10 languages), they test 11 closed, open-weight and open-data LLMs, with and without Wikidata entity information. That information is supplied either through the Relik linker or from gold annotations. Outputs are scored with m-ETA and COMET. Over 500 Traditional Chinese and Italian entities were manually labelled by level. Baseline accuracy falls as entities become more local, and external knowledge helps most for these entities (m-ETA gains up to 70%). Agnostic entities barely benefit. The study shows that cultural embeddedness should guide knowledge augmentation in MT. It also reveals that mismatches between Wikidata labels and benchmark references distort evaluation.

Key points

  • Experiments cover 11 LLMs and 10 languages from XCTranslate, translating from English only. Cultural-level annotation was done for Traditional Chinese and Italian.
  • The released annotation resource contains 500+ entities. Inter-annotator agreement is κ = 0.81 (Chinese) and κ = 0.74 (Italian). The code and data are on GitHub as SapienzaNLP/cultural-ea-mt.
  • Knowledge injection (Relik-based or gold) raises m-ETA by as much as 70% for Culturally Local entities. Smaller models depend on it more.
  • Closed and open-weight models improve in both entity accuracy (m-ETA) and overall quality (COMET). Open-data models gain entity accuracy but lose COMET because they follow instructions poorly, for example by adding unrequested text.
  • Error analysis with GPT-5-mini shows that Wikidata labels often diverge from XCTranslate gold mentions. Some gold labels are themselves wrong, which unfairly penalises correct outputs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3jxgnspt4srr

@inproceedings{xu2026cultural,
  title = {{Cultural and Knowledge Biases in LLMs through the Lens of Entity-Aware Machine Translation}},
  author = {Xu, Lu and Moroni, Luca and Navigli, Roberto},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3jxgnspt4srr},
  url = {https://doi.org/10.63317/3jxgnspt4srr}
}

References

38 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Meltem Aksoy. 2025. Whose morality do they speak? unraveling cultural bias in multilingual language models. Natural Language Processing Journal, page 100172. find
  2. Tuka Alhanai, Adam Kasumovic, Mohammad M Ghassemi, Aven Zitzelberger, Jessica M Lundin, and Guillaume Chabot-Couture. 2025. Bridging the gap: enhancing llm performance for low-resource african languages with new benchmarks, fine-tuning, and cultural adjustments. In Proceedings of the AAAI Conference… find
  3. Sahar Araghi and Alfons Palangkaraya. 2024. The link between translation difficulty and the quality of machine translation: a literature review and empirical investigation. Language Resources and Evaluation, 58(4):1093–1114. find
  4. Arnav Arora, Lucie-Aimée Kaffee, and Isabelle Augenstein. 2023. Probing pre-trained language models for cross-cultural differences in values. Cross-Cultural Considerations in NLP@ EACL, page 114. find our card
  5. Bojana Budimir. 2025. The challenge of translating culture-specific items: Evaluating MT and LLMs compared to human translators. In Proceedings of Machine Translation Summit XX: Volume 1, pages 455–467, Geneva, Switzerland. European Association for Machine Translation. find
  6. Niccolò Campolungo, Federico Martelli, Francesco Saina, and Roberto Navigli. 2022. DiBiMT: A novel benchmark for measuring Word Sense Disambiguation biases in Machine Translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page… find
  7. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 53–67. find our card
  8. Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. Are multilingual LLMs culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics:… find
  9. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, et al. Culturalbench: a robust, diverse and challenging benchmark on measuring (the lack of) cultural knowledge of llms. find
  10. Simone Conia, Daniel Lee, Min Li, Umar Farooq Minhas, Saloni Potdar, and Yunyao Li. 2024. Towards cross-cultural machine translation with retrieval-augmented generation from multilingual knowledge graphs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, page… find
  11. Simone Conia, Min Li, Roberto Navigli, and Saloni Potdar. 2025. Semeval-2025 task 2: Entity-aware machine translation. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), pages 2535–2557. find
  12. Shekoufeh Daghoughi and Mahmood Hashemian. 2016. Analysis of culture-specific items and translation strategies applied in translating jalal al-ahmad’s" by the pen". English language teaching, 9(4):171–185. find
  13. Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lacalle, and Mikel Artetxe. 2024. Bertaqa: How much do language models know about local culture? Advances in Neural Information Processing Systems, 37:34077–34097. find
  14. Aitor Gonzalez-Agirre, Marc Pàmies, Joan Llop, Irene Baucells, Severino Da Dalt, Daniel Tamayo, José Javier Saiz, Ferran Espuña, Jaume Prats, Javier Aula-Blasco, Mario Mina, Iñigo Pikabea, Adrián Rubio, Alexander Shvets, Anna Sallés, Iñaki Lacunza, Jorge Palomar, Júlia Falcão, Lucía Tormo, Luis Vasq… find
  15. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao,… find
  16. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp… find our card
  17. Huihan Li, Liwei Jiang, Nouha Dziri, Xiang Ren, and Yejin Choi. 2024. CULTURE-GEN: Revealing global cultural perception in language models through natural language prompting. In First Conference on Language Modeling. find
  18. Zheng Wei Lim, Ekaterina Vylomova, Charles Kemp, and Trevor Cohn. 2024. Predicting human translation difficulty with neural machine translation. Transactions of the Association for Computational Linguistics, 12:1479–1496. find
  19. Federico Martelli, Stefano Perrella, Niccolò Campolungo, Tina Munda, Svetla Koeva, Carole Tiberius, and Roberto Navigli. 2025. Dibimt: A gold evaluation benchmark for studying lexical ambiguity in machine translation. Computational Linguistics, 51(2):343–413. find our card
  20. Pedro Henrique Martins, João Alves, Patrick Fernandes, Nuno M. Guerreiro, Ricardo Rei, Amin Farajian, Mateusz Klimaszewski, Duarte M. Alves, José Pombal, Nicolas Boizard, Manuel Faysse, Pierre Colombo, François Yvon, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2025.… find
  21. Domenico Meconi, Simone Stirpe, Federico Martelli, Leonardo Lavalle, and Roberto Navigli. 2025. Do large language models understand word senses? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33897–33916, Suzhou, China. Association for Computational… find our card
  22. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, et al. 2024. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. Advances in Neural Information Processing Systems, 37:… find our card
  23. Millicent Ochieng, Varun Gumma, Sunayana Sitaram, Jindong Wang, Vishrav Chaudhary, Keshet Ronen, Kalika Bali, and Jacki O’Neill. 2024. Beyond metrics: evaluating llms’ effectiveness in culturally nuanced, low-resource real-world scenarios. arXiv preprint arXiv:2406.00343. find
  24. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… find our card
  25. Riccardo Orlando, Pere-Lluís Huguet Cabot, Edoardo Barba, and Roberto Navigli. 2024a. Relik: Retrieve and link, fast and accurate entity linking and relation extraction on an academic budget. In Findings of the Association for Computational Linguistics ACL 2024, pages 14114– 14132. find
  26. Riccardo Orlando, Luca Moroni, Pere-Lluís Huguet Cabot, Simone Conia, Edoardo Barba, Sergio Orlandini, Giuseppe Fiameni, and Roberto Navigli. 2024b. Minerva LLMs: The first family of large language models trained from scratch on Italian data. In Proceedings of the 10th Italian Conference on Computat… find
  27. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… find
  28. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  29. Matiss Rikters and Makoto Miwa. 2024. Entity-aware multi-task training helps rare word machine translation. In Proceedings of the 17th International Natural Language Generation Conference, pages 47–54, Tokyo, Japan. Association for Computational Linguistics. find
  30. Nikit Srivastava, Aleksandr Perevalov, Denis Kuchelev, Diego Moussallem, Axel-Cyrille Ngonga Ngomo, and Andreas Both. 2023. Lingua franca–entity-aware machine translation approach for question answering over knowledge graphs. In Proceedings of the 12th Knowledge Capture Conference 2023, pages 122–13… find
  31. Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS nexus, 3(9):pgae346. find
  32. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazari… find our card
  33. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  34. Jiaan Wang, Fandong Meng, Yingxue Zhang, and Jie Zhou. 2024. Retrieval-augmented machine translation with unstructured knowledge. arXiv preprint arXiv:2412.04342. find
  35. Shufang Xie, Yingce Xia, Lijun Wu, Yiqing Huang, Yang Fan, and Tao Qin. 2022. End-to-end entity-aware neural machine translation. Machine Learning, 111(3):1181–1203. find
  36. Lu Xu. 2025. Wikidata-driven entity-aware translation: Boosting LLMs with external knowledge. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), pages 1802–1809, Vienna, Austria. Association for Computational Linguistics. find
  37. Binwei Yao, Ming Jiang, Tara Bobinac, Diyi Yang, and Junjie Hu. 2024. Benchmarking machine translation with cultural awareness. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13078–13096. find our card
  38. Min Zhang, Limin Liu, Zhao Yanqing, Xiaosong Qiao, Su Chang, Xiaofeng Zhao, Junhao Zhu, Ming Zhu, Song Peng, Yinglu Li, et al. 2023. Leveraging multilingual knowledge graph to boost domain-specific entity translation of chatgpt. In Proceedings of Machine Translation Summit XIX, Vol. 2: Users Track,… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-cultural-knowledge-biases-llms-through/