Elamur Research⌕
☰
home / papers / 2026 / Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation

Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation

Summary

LLMs translate general text well, but little is known about how they handle culturally loaded content. The authors build CanMT, a parallel benchmark drawn from classic novels and their professional translations. It covers 12 directions among English, Chinese, Japanese, Spanish and Russian.

To build it, they aligned sentences with Vecalign, then extracted culture-specific items (CSIs) with two LLMs and filtered the resulting lexicon. They chose diverse sentences via LaBSE embeddings and hierarchical clustering, and finished with human checks. Outputs are scored by a single LLM judge with a rubric covering five dimensions taken from translation theory.

Testing many LLMs and MT systems, the authors find that quality rises with model scale and with test-time reasoning. Asking for communicative translation improves fluency and function, while semantic translation preserves meaning more faithfully, and unconstrained models lean semantic by default. Models often know a CSI's meaning yet still render it poorly. Reference translations make the LLM judge agree better with humans.

The work offers a reusable diagnostic resource and points to concrete weaknesses in cultural transfer.

Key points

  • Source novels: "Adventures of Huckleberry Finn", "The Great Gatsby", 《活着》 (To Live), «Анна Каренина», 『古都』 (The Old Capital) and «Don Quijote de la Mancha».
  • Size: each direction has roughly 95–166 samples and 138–368 CSIs. The pipeline used LLaMA-3.1-70B-Instruct and Qwen2.5-72B-Instruct (union of outputs) to find CSIs, with DeepSeek filtering the lexicon.
  • Evaluation dimensions: the rubric rates Naturalness and Fidelity, along with Cultural Adaptation, Contextual Accuracy and Functional Equivalence. These draw on Halliday, Venuti, Nida and Newmark.
  • CSI difficulty: geographic and ecological terms are easiest, while linguistic-symbolic items are hardest. A "knowledge-application gap" persists even when models answer CSI probing questions correctly.
  • Judge reliability: Kendall's τ between the judge and humans is 0.39–0.47 with references versus 0.36–0.45 without. For comparison, human–human agreement is 0.42–0.49.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1367

@inproceedings{yuan2026culture,
  title = {{Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation}},
  author = {Yuan, Zekun and Ye, Yangfan and Feng, Xiaocheng and Li, Baohang and Hong, Qichen and Lu, Yunfei and Tu, Dandan and Qin, Bing},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1367},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1367}
}

References

39 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗ our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473. arXiv preprint Yong Cao, Yova Kementchedjhieva, Ruixiang Cui, Antonia Karamolegkou, Li Zhou, Megan Dare, Lucia Donatelli, and Daniel Hershcovich. 2023. Cultural adaptation of recipes. Preprint, arXiv:2310.17353. ↗ our card
  3. Andong Chen, Lianzhang Lou, Kehai Chen, Xuefeng Bai, Yang Xiang, Muyun Yang, Tiejun Zhao, and Min Zhang. 2025. Benchmarking LLMs for translating classical Chinese poetry: Evaluating adequacy, fluency, and elegance. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33007–33024, Suzhou, China. Association for Computational Linguistics. find our card
  4. Shanbo Cheng, Yu Bao, Qian Cao, Luyang Huang, Liyan Kang, Zhicheng Liu, Yu Lu, Wenhao Zhu, Jingwen Chen, Zhichao Huang, Tao Li, Yifu Li, Huiying Lin, Sitong Liu, Ningxin Peng, Shuaijie She, Lu Xu, Nuo Xu, Sen Yang, and 7 others. 2025. Seed-x: Building strong multilingual translation llm with 7b parameters. Preprint, arXiv:2507.13618. ↗
  5. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, and 1 others. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. ↗
  6. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. ↗ our card
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407. find
  8. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic bert sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891. find our card
  9. Zhaopeng Feng, Jiayuan Su, Jiamei Zheng, Jiahan Ren, Yan Zhang, Jian Wu, Hongwei Wang, and Zuozhu Liu. 2025. M-MAD: Multidimensional multi-agent debate for advanced machine translation evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7084–7107, Vienna, Austria. Association for Computational Linguistics. find our card
  10. Baban Gain, Dibyanayan Bandyopadhyay, and Asif Ekbal. 2025. Bridging the linguistic divide: A survey on leveraging large language models for machine translation. CoRR, abs/2504.01919. ↗
  11. M.A.K. Halliday. 1978. Language as Social Semiotic: The Social Interpretation of Language and Meaning. Open University set book. Edward Arnold. find
  12. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. ArXiv, abs/2302.09210. ↗
  13. Tianyi Hu, Maria Maistro, and Daniel Hershcovich. 2024. Bridging cultures in the kitchen: A framework and benchmark for cross-cultural recipe retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1068–1080, Miami, Florida, USA. Association for Computational Linguistics. find our card
  14. Yichong Huang, Baohang Li, Xiaocheng Feng, Wenshuai Huo, Chengpeng Fu, Ting Liu, and Bing Qin. 2024. Aligning translation-specific understanding to general understanding in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5028–5041, Miami, Florida, USA. Association for Computational Linguistics. find
  15. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, and 7 others. 2024. Mixtral of experts. ArXiv, abs/2401.04088. ↗
  16. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. Is chatgpt a good translator? yes with gpt-4 as the engine. Preprint, arXiv:2301.08745. ↗ our card
  17. Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556. ↗ our card
  18. Yinquan Lu, Wenhao Zhu, Lei Li, Yu Qiao, and Fei Yuan. 2024. LLaMAX: Scaling linguistic horizons of LLM by enhancing translation capabilities beyond 100 languages. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10748–10772, Miami, Florida, USA. Association for Computational Linguistics. find
  19. P. Newmark. 1981. Approaches to Translation. Language teaching methodology series. Pergamon Press. find
  20. Peter Newmark. 1988. A textbook of translation, volume 66. Prentice hall New York. find
  21. E.A. Nida. 1964. Toward a Science of Translating: With Special Reference to Principles and Procedures Involved in Bible Translating. E.J. Brill. find
  22. Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orasan, Tharindu Ranasinghe, and Fred Blain. 2024. What do large language models need for machine translation evaluation? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3660–3674. ↗
  23. Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. 2024. Multilingual large language model: A survey of resources, taxonomy and frontiers. arXiv preprint arXiv:2404.04925. ↗
  24. Tom Kocmi and Christian Federmann. 2023a. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Proceedings of the Eighth Conference on Machine Translation, pages 768–775, Singapore. Association for Computational Linguistics. find our card
  25. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. Qwen2.5 Technical Report. arXiv preprint. ArXiv:2412.15115 [cs]. ↗
  26. Tom Kocmi and Christian Federmann. 2023b. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203, Tampere, Finland. European Association for Machine Translation. ↗
  27. Pushpdeep Singh, Mayur Patidar, and Lovekesh Vig. 2024. Translating across cultures: LLMs for intralingual cultural adaptation. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 400–418, Miami, FL, USA. Association for Computational Linguistics. find
  28. Shuang Li, Jiangjie Chen, Siyu Yuan, Xinyi Wu, Hao Yang, Shimin Tao, and Yanghua Xiao. 2023. Translate meanings, not just words: Idiomkb’s role in optimizing idiomatic translation with language models. Preprint, arXiv:2308.13961. ↗
  29. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗ our card
  30. Brian Thompson and Philipp Koehn. 2019. Vecalign: Improved sentence alignment in linear time and space. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 1342–1348. find
  31. L. Venuti. 2008. The Translator’s Invisibility: A History of Translation. The Translator’s Invisibility: A History of Translation. Routledge. find
  32. Minghan Wang, Viet-Thanh Pham, Farhad Moghimifar, and Thuy-Trang Vu. 2025. Proverbs run in pairs: Evaluating proverb translation capability of large language model. Preprint, arXiv:2501.11953. ↗
  33. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. ↗ our card
  34. Binwei Yao, Ming Jiang, Tara Bobinac, Diyi Yang, and Junjie Hu. 2024. Benchmarking machine translation with cultural awareness. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13078–13096, Miami, Florida, USA. Association for Computational Linguistics. find our card
  35. Yangfan Ye, Xiachong Feng, Xiaocheng Feng, Weitao Ma, Libo Qin, Dongliang Xu, Qing Yang, Hongtao Liu, and Bing Qin. 2024a. GlobeSumm: A challenging benchmark towards unifying multi-lingual, cross-lingual and multi-document news summarization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10803–10821, Miami, Florida, USA. Association for Computational Linguistics. ↗
  36. Yangfan Ye, Xiaocheng Feng, Xiachong Feng, Lei Huang, Weitao Ma, Qichen Hong, Yunfei Lu, Duyu Tang, Dandan Tu, and Bing Qin. 2025a. Langgps: Language separability guided data pre-selection for joint multilingual instruction tuning. arXiv preprint arXiv:2511.10229. ↗
  37. Yangfan Ye, Xiaocheng Feng, Xiachong Feng, Libo Qin, Yichong Huang, Lei Huang, Weitao Ma, Qichen Hong, Zhirui Zhang, Yunfei Lu, and 1 others. 2024b. Exploring cross-lingual latent transplantation: Mutual opportunities and open challenges. arXiv preprint arXiv:2412.12686. ↗
  38. Zhonghe Zhang, Xiaoyu He, Vivek Iyer, and Alexandra Birch. 2024. Cultural adaptation of menus: A fine-grained approach. In Proceedings of the Ninth Conference on Machine Translation, pages 1258–1271, Miami, Florida, USA. Association for Computational Linguistics. find
  39. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine translation with large language models: Empirical results and analysis. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2765–2781, Mexico City, Mexico. Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/yuan-2026-culture-aware-machine-translation-large/