Elamur Research⌕
☰
home / papers / 2025 / Can LLMs Translate Cultural Nuance in Dialects? A Case Study on Lebanese Arabic

Can LLMs Translate Cultural Nuance in Dialects? A Case Study on Lebanese Arabic

Paper · 2025 · Silvana Yakhni and Ali Chehab

Summary

The paper asks whether large language models can carry the cultural meaning of Lebanese Arabic into English, or whether they just render the words. The authors argue that existing Lebanese–English parallel resources hide this weakness. Those resources are mostly translated from English or MSA, so they contain little native idiom or culture. To fix this, they built LW, a professionally translated corpus drawn from the Language Wave podcast. They compare it with OS, MADAR, and FLoRes on three things: sentence length, domain vocabulary, and a lexicon-based Cultural Awareness Score (CAS).

They then test Arabic-focused and general LLMs (Jais, AceGPT, Cohere's Command-R+, GPT-4o) against NMT systems (NLLB-200, Google Translate). On the English-origin sets, both families score about the same. On LW, the LLMs pull ahead, and Command-R+ performs close to GPT-4o. The authors also check xCOMET against human ratings and find a strong correlation, which supports using it for dialect-to-English evaluation. The work shows that translated benchmarks can overstate MT quality for dialects.

Key points

  • LW dataset: 95 podcast episodes yielding 2,947 Lebanese sentences, each with a professional English translation, in a colloquial everyday register.
  • Weaknesses of the compared corpora:
    • OS has 120,600 subtitle sentences, routed through MSA.
    • MADAR has 2,000 BTEC travel sentences in 26 city dialects, expanded to 10,000 for major cities such as Beirut.
    • FLoRes has 3,001 formal Wikimedia sentences.
    • The Zbib et al. Levantine/Egyptian corpus has 3.5M tokens but sits behind LDC access.
  • CAS definition: for each domain, CAS = (f_A − f_W)/(f_A + f_W), ranging over [−1, 1]. Arab and Western terms are counted by exact string match across 8 domains (arts, cuisine, heritage, geography, language, news, socioeconomic life, travel). LW scores positive throughout, especially on personal names and locations.
  • Model comparison: LLMs and NMT tie on non-native data, while LLMs are better on the authentic LW content.
  • Human check: annotators rated 150 translations on a five-level rubric running from Very Good to Incomprehensible. xCOMET (XL 3.5B to XXL 10.7B) tracked these human judgments closely, unlike surface metrics such as BLEU.

Interesting ideas

  • I1. Build test sets from text originally written in the target variety rather than translated into it, because translated sources hide the hard cases; with such data, a gap between model families appeared that translated benchmarks concealed.
  • I2. A cheap signed ratio of two contrasting term lexicons per domain gives a transparent, interpretable score for how a corpus is skewed, and it can be extended with expert-supplied word lists.
  • I3. The paper's motivating failure is an idiom translated word by word into a literal scene instead of its figurative meaning (chaos), which shows that literal composition is the default error mode.
  • I4. The authors collected examples whose correct reading depends on context: a single verb form that can mean "I wrote" or "she wrote", and a possessive that can mean "his leg" or "its leg".
  • I5. The authors check a learned metric against a small set of human ratings guided by a rubric with worked examples before relying on it.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation suites should include natively written English (fables, school texts) next to converted or translated material, so that apparent parity is not an artefact of easy source text.
  • I2 (world): The same ratio over Wikidata/WordNet-backed entity classes could profile which domains and entity types a training corpus actually covers, before trusting the knowledge compiled from it.
  • I3 (frames): Add an idiom/proverb check that flags multiword expressions whose literal frame clashes with the context, and routes them to a figurative reading before frame-building.
  • I4 (tokens): Keep such morphological ambiguity unresolved in the analysis and let later stages choose, rather than having the tagger commit early.
  • I5 (learn): Before adopting any automatic score for generated or translated output, compare it once against a small rubric-guided human sample.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 works

Cite

Cite as Silvana Yakhni and Ali Chehab. 2025. Can LLMs Translate Cultural Nuance in Dialects? A Case Study on Lebanese Arabic. Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yakhni2025can,
  title = {{Can LLMs Translate Cultural Nuance in Dialects? A Case Study on Lebanese Arabic}},
  author = {Silvana Yakhni and Ali Chehab},
  booktitle = {Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script},
  year = {2025},
  url = {https://aclanthology.org/2025.abjadnlp-1.13/}
}

References

39 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Firoj Alam, Shammur Absar Chowdhury, Sabri Boughorbel, and Maram Hasanain. 2024. LLMs for low resource languages in multilingual, multimodal and dialectal settings. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts, pages 27–33, St. Julian’s, Malta. Association for Computational Linguistics. find
  2. Fakhraddin Alwajih, Gagan Bhatia, and Muhammad Abdul-Mageed. 2024. Dallah: A dialect-aware multimodal large language model for arabic. ArXiv, abs/2407.18129. ↗
  3. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan AlRashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abdelali, Jeril Kuriakose, Abdalghani Abujabal, Nora Al-Twairesh, Areeb Alowisheq, and Haidar Khan. 2024. Allam: Large language models for arabic and english. Preprint, arXiv:2407.15390. ↗
  4. Saikat Barua. 2024. Exploring autonomous agents through the lens of large language models: A review. ArXiv, abs/2404.04442. ↗
  5. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The madar arabic dialect corpus and lexicon. In International Conference on Language Resources and Evaluation. find
  6. Zhaopeng Feng, Yan Zhang, Hao Li, Wenqiang Liu, Jun Lang, Yang Feng, Jian Wu, and Zuozhu Liu. 2024. Improving llm-based machine translation with systematic self-correction. ArXiv, abs/2402.16379. ↗
  7. Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. 2023. Dictionary-based phrase-level prompting of large language models for machine translation. ArXiv, abs/2302.07856. ↗
  8. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luísa Coheur, Pierre Colombo, and André Martins. 2023. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995. ↗
  9. Karim El Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi A. Zaraket. 2022. Curras + baladi: Towards a levantine corpus. In International Conference on Language Resources and Evaluation. ↗
  10. Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. 2023. Exploring human-like translation strategy with large language models. Transactions of the Association for Computational Linguistics, 12:229–246. ↗
  11. Amr Hendy, Mohamed Gomaa Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. ArXiv, abs/2302.09210. ↗
  12. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024. AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics. ↗
  13. Joy Ishaku, Muhammad Mustapha, and Muhammad Bello. 2020. Contrastive analysis of lexical and structural ambiguity between hausa and english languages. 2:19–34. find
  14. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? yes with gpt-4 as the engine.
  15. Karima Kadaoui, Samar M. Magdy, Abdul Waheed, Md Tawkat Islam Khondaker, Ahmed Oumar El-Shangiti, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Tarjamat: Evaluation of bard and chatgpt on machine translation of ten arabic varieties. Preprint, arXiv:2308.03051. Md. ↗
  16. Tawkat Islam Khondaker, Numaan Naeem, Fatimah Khan, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2024. Benchmarking llama-3 on arabic language generation tasks. In ARABICNLP.
  17. Tom Kocmi, Vilém Zouhar, Christian Federmann, and Matt Post. 2024. Navigating the metrics maze: Reconciling score magnitudes and accuracies. In Annual Meeting of the Association for Computational Linguistics. ↗
  18. Mateusz Krubiński, Hashem Sellat, Shadi Saleh, Adam Pospíil, Petr Zemánek, and Pavel Pecina. 2023. Multi-parallel corpus of north levantine arabic. In ARABICNLP.
  19. Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In WMT@ACL.
  20. Seungjun Lee, Jungseob Lee, Hyeonseok Moon, Chanjun Park, Jaehyung Seo, Sugyeong Eo, Seonmin Koo, and Heu-Jeoung Lim. 2023. A survey on evaluation metrics for machine translation. Mathematics. ↗
  21. Juhao Liang, Zhenyang Cai, Jianqing Zhu, Huang Huang, Kewei Zong, Bang An, Mosen Alharthi, Juncai He, Lian Zhang, Haizhou Li, Benyou Wang, and Jinchao Xu. 2024. Alignment at pre-training! towards native alignment for arabic LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  22. Chenyang Lyu, Jitao Xu, Longyue Wang, and Minghao Wu. 2023. A paradigm shift: The future of machine translation lies with large language models. In International Conference on Language Resources and Evaluation. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. find
  23. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur A. Chowdhury, and Firoj Alam. 2024. Aradice: Benchmarks for dialectal and cultural capabilities in llms. ArXiv, abs/2409.11404. ↗
  24. Tarek Naous, Michael Joseph Ryan, and Wei Xu. 2023. Having beer after prayer? measuring cultural bias in large language models. In Annual Meeting of the Association for Computational Linguistics.
  25. Fallou Ngom, Daivi Rodima-Taylor, and David Robinson. 2023. ʿajamī literacies of africa: The hausa, fula, mandinka, and wolof traditions. Islamic Africa, 14(2):119 – 143. ↗
  26. Viktória Ondrejová and Marek Šuppa. 2024. Can LLMs handle low-resource dialects? a case study on translation and common sense reasoning in šariš. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 130–139, Mexico City, Mexico. Association for Computational Linguistics. find
  27. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics.
  28. Maja Popovic. 2015. chrf: character n-gram f-score for automatic mt evaluation. In WMT@EMNLP.
  29. Ricardo Rei, José G. C. de Souza, Duarte M. Alves, Chrysoula Zerva, Ana C. Farinha, T. Glushkova, Alon Lavie, Luísa Coheur, and André F. T. Martins. 2022. Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Conference on Machine Translation.
  30. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. Preprint, arXiv:2308.16149. ↗
  31. Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223– 231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas.
  32. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. 2023. A benchmark for learning to translate a new ArXiv, language from one grammar book. abs/2309.16575. ↗
  33. Nllb team, Marta Ruiz Costa-jussà, James Cross, Onur cCelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Alison Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon L. Spruit, C. Tran, Pierre Yves Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm’an, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. ArXiv, abs/2207.04672. ↗
  34. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and ef-ArXiv, ficient foundation language models. abs/2302.13971. ↗
  35. Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2023. A paradigm shift in machine translation: Boosting translation performance of large language models. ArXiv, abs/2309.11674. ↗
  36. Binwei Yao, Ming Jiang, Tara Bobinac, Diyi Yang, and Junjie Hu. 2024. Benchmarking machine translation with cultural awareness. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13078–13096, Miami, Florida, USA. Association for Computational Linguistics.
  37. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyridon Matsoukas, Richard M. Schwartz, John Makhoul, Omar Zaidan, and Chris Callison-Burch. 2012. Machine translation of arabic dialects. In North American Chapter of the Association for Computational Linguistics.
  38. Chen Zhang, Xiao Liu, Jiuheng Lin, and Yansong Feng. 2024. Teaching large language models an unseen language on the fly. ArXiv, abs/2402.19167. ↗
  39. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675. A Translation Models Jais: MBZUAI introduced the largest openly available Arabic language models, known as Jais ranging from 590M to 70B (Sengupta et al., 2023), which quickly captured the attention of the Arabic research community. These models were built based on the GPT architecture and pre-trained using a blend of English and Arabic datasets, making them ideal candidates for the translation task. However, Jais’s primary limitation lies in its heavy reliance on translated datasets, driven by the scarcity of high-quality Arabic datasets. Notably, this reliance on translated data can introduce ”localization issues,” potentially undermining the reliability and applicability of the models in native contexts, specifically in the translation of cultural content. (Huang et al., 2024) have observed an apparent bias in Jais, and showed that Jais produced outputs with a notable inclination toward English-centric content, frequently emphasizing terms associated with Christianity, for instance. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yakhni-2025-can-llms-translate-cultural-nuance/