Elamur Research⌕
☰
home / papers / 2026 / Current state of LLMs for Arabic dialectal machine translation

Current state of LLMs for Arabic dialectal machine translation

Paper · 2026 · Josef Jon, Rawan Bondok and Ondřej Bojar

Summary

Jon, Bondok and Bojar ask how well current large language models translate between English and spoken Arabic varieties, as opposed to Modern Standard Arabic. They test 16 LLMs on the MADAR test data (AraBench version), covering 16 dialects plus MSA, in both directions. The models fall into three groups: Arabic-focused models (Jais-2, Nile-Chat, Command-R7B-Arabic), general multilingual models (Gemma-3, Aya Expanse, Command-R, Mistral, Qwen3, Llama, EuroLLM) and GPT-4.1 APIs.

For English→Arabic, each model gets two prompts: a generic "translate into Arabic" one and one that names the target country's dialect. Outputs are scored with BLEU, ChrF, XCOMET-XL and a reference-free GEMBA judge that penalizes the wrong dialect. A small human study and LLM-based dialect identification complete the evaluation.

GPT-4.1-mini and Jais-2-70B-Chat come out on top. Many multilingual models fall back to MSA even when asked for a dialect. Naming the dialect in the prompt usually helps. The authors also find that reference-based metrics undervalue valid dialect variants, and COMET tends to reward MSA-like output. The paper works as a model-selection guide, but the authors warn that the test data may have leaked into the models' training sets.

Key points

  • In the system-level ranking, GPT-4.1-mini wins 16 times (average rank 2.1) and Jais-2-70B-Chat wins 14 times (average rank 2.9). Most other models often score under 10 BLEU on dialects, and Algerian, Iraqi, Yemeni and Libyan stay hard even for the top systems.

  • In the manual check of 100 sentences from madar.test.nil.1.eg, each output got an accuracy score and a dialectalness score:

    ModelAccuracyDialectalness
    Jais-2-70B94.698.5
    GPT-4.1-mini96.396.0
    Command-r-08-202493.486.8
    EuroLLM-9B93.036.4

  • GEMBA was set up as direct assessment on a 0–100 scale without a reference, with up to 50 points deducted for the wrong dialect.

  • Preprocessing before scoring:

    • Runaway outputs (more than 5× the source length in tokens or characters) were cut back to the input length.
    • Both references and outputs were normalized with CAMeL Tools (Alef, Alef Maksura, Teh Marbuta, Hamza), and Arabic numerals were converted to Latin ones.

  • Four LLMs were used as dialect classifiers: Jais-2-70B-Chat, gemma-3-27b-it, aya-expanse-32b and Llama-3.3-70B-Instruct. Command-r-08-2024 stood out for resolving gender-ambiguous inputs with feminine forms.

Interesting ideas

  • I1. Before using an LLM classifier to judge system outputs, run it first on the human references to confirm it recognizes the variety at all. This calibrates the judge against known-good data.
  • I2. Human raters gave two independent scores, one for meaning accuracy and one for whether the requested variety was produced, since a fluent output in the wrong form can still be accurate.
  • I3. Comparing a generic prompt with a variety-specific one shows whether a model actually follows the condition or ignores it.
  • I4. A simple length-ratio guard (output more than 5× the source) catches degenerate repetition loops before they distort scores.
  • I5. Normalizing spelling variants on both sides before computing string metrics keeps orthographic noise from being counted as errors.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Run each LLM teacher tier on gold UD sentences first, and trust its silver verdicts only where it matches gold.
  • I2 (back): Score generation and the text → graph → text compressor separately for preserved meaning and for register or style, so that a paraphrase is not mistaken for a failure.
  • I3 (learn): Give a teacher the same sentence with and without an explicit annotation guideline (e.g., a UD dialect convention), and keep only teachers whose outputs shift as intended.
  • I4 (learn): Add a cheap sanity filter that rejects LLM-teacher outputs with absurd length or token-repeat patterns before they enter the silver data.
  • I5 (tokens): Map spelling variants (e.g., British vs. American) to a shared form before comparing lemmas and tags across witness parsers, so disagreements reflect analysis rather than spelling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany109 works · 35,920 citations🎓Charles UniversityPrague · Czechia133 works · 20,686 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksSemitic languages · 2,908 works

Cite

Cite as Josef Jon, Rawan Bondok and Ondřej Bojar. 2026. Current state of LLMs for Arabic dialectal machine translation. Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.abjadnlp-1.41

@inproceedings{jon2026current,
  title = {{Current state of LLMs for Arabic dialectal machine translation}},
  author = {Josef Jon, Rawan Bondok and Ondřej Bojar},
  booktitle = {Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script},
  year = {2026},
  doi = {10.18653/v1/2026.abjadnlp-1.41},
  url = {https://doi.org/10.18653/v1/2026.abjadnlp-1.41}
}

References

39 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Idris Abdulmumin, Victor Agostinelli, Tanel Alumäe, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Claudia Borg, Fethi Bougares, Roldano Cattoni, Mauro Cettolo, Lizhong Chen, William Chen, Raj Dabre, Yannick Estève, Marcello Federico, Mark Fishel, Marco Gaido, Dávid Javorský, Marek Kasztelnik, and 33 others. 2025. Findings of the IWSLT 2025 evaluation campaign. In Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025), pages 412–481, Vienna, Austria (in-person and online). Association for Computational Linguistics. Ibrahim Said Ahmad, Antonios Anastasopoulos, Ondřej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, William Chen, Qianqian Dong, Marcello Federico, Barry Haddow, Dávid Javorský, Mateusz Krubiński, Tsz Kin Lam, Xutai Ma, Prashant Mathur, Evgeny Matusov, Chandresh Maurya, John P. McCrae, and 25 others. 2024. find
  2. FIND-INGS OF THE IWSLT 2024 EVALUATION CAM-PAIGN. In Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024), pages 1–11, Bangkok, Thailand (in-person and online). Association for Computational Linguistics. find
  3. Amjad Almahairi, Kyunghyun Cho, Nizar Habash, and Aaron Courville. 2016. First result on arabic neural machine translation. Preprint, arXiv:1606.02680. ↗
  4. Yazeed Alnumay, Alexandre Barbet, Anna Bialas, William Darling, Shaan Desai, Joan Devassy, Kyle Duffy, Stephanie Howe, Olivia Lasche, Justin Lee, Anirudh Shrinivason, and Jennifer Tracey. 2025. Command r7b arabic: A small, enterprise focused, multilingual, and culturally aware arabic llm. Preprint, arXiv:2503.14603. ↗
  5. Mohamed Anwar, Abdelhakim Freihat, George Ibrahim, Mostafa Awad, Abdelrahman Atef Mohamed Ali Sadallah, Gurpreet Gosal, Gokul Ramakrishnan, Biswajit Mishra, Sarath Chandran, Ahmed Frikha, Laith H. Baniata, Se-Young Park, and Seong-Bae Park. 2018. A neural machine translation model for arabic dialects that utilises multitask learning (mtl). find
  6. Computational Intelligence and Neuroscience, 2018. find
  7. Waad Ben Kheder, Josef Jon, André Beyer, Abdel Messaoudi, Rabea Affan, Claude Barras, Maxim Ty-chonov, and Jean-Luc Gauvain. 2024. ALADAN at IWSLT24 low-resource Arabic dialectal speech translation task. In Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024), pages 192–202, Bangkok, Thailand (in-person and online). Association for Computational Linguistics.
  8. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  9. Cohere Labs. 2024a. c4ai-command-r-08-2024. find
  10. Cohere Labs. 2024b. c4ai-command-r-v01. find
  11. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, and 26 others. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261. ↗
  12. Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and Stephan Vogel. 2016. QCRI’s machine translation systems for IWSLT’16. In Proceedings of the 13th International Conference on Spoken Language Translation, Seattle, Washington D.C. International Workshop on Spoken Language Translation. find
  13. Wael Farhan, Bashar Talafha, Analle Abuammar, Ruba Jaikat, Mahmoud Al-Ayyoub, {Ahmad Bisher} Tarakji, and Anas Toma. 2020. Unsupervised dialectal neural machine translation. Information Processing and Management, 57(3). Publisher Copyright: © 2019 Elsevier Ltd. find
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  15. Nizar Habash and Jun Hu. 2009. Improving Arabic-Chinese statistical machine translation using English as pivot language. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 173– 181, Athens, Greece. Association for Computational Linguistics. find
  16. N.Y. Habash. 2010. Introduction to Arabic Natural Language Processing. Synthesis digital library of engineering and computer science. Morgan & Claypool Publishers. ↗
  17. Ahmed Heakl, Youssef Zaghloul, Mennatullah Ali, Rania Hossam, and Walid Gomaa. 2024. Arzen-llm: Code-switched egyptian arabic-english translation and speech recognition using llms. Procedia Computer Science, 244:113–120. 6th International Conference on AI in Computational Linguistics. find
  18. Tom Kocmi, Arkady Arkhangorodsky, Alexandre Berard, Phil Blunsom, Samuel Cahyawijaya, Théo Dehaze, Marzieh Fadaee, Nicholas Frosst, Matthias Galle, Aidan Gomez, Nithya Govindarajan, Wei-Yin Ko, Julia Kreutzer, Kelly Marchisio, Ahmet Üstün, Sebastian Vincent, and Ivan Zhang. 2025a. Command-a-translate: Raising the bar of machine translation with difficulty filtering. In Proceedings of the Tenth Conference on Machine Translation, pages 789–799, Suzhou, China. Association for Computational Linguistics. Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstantin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Howard Lakougna, Jessica Lundin, Christof Monz, Kenton Murray, and 10 others. 2025b. Findings of the WMT25 general machine translation shared task: Time to stop evaluating on easy test sets. In Proceedings of the Tenth Conference on Machine Translation, pages 355–413, Suzhou, China. Association for Computational Linguistics.
  19. Tom Kocmi and Christian Federmann. 2023a. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Proceedings of the Eighth Conference on Machine Translation, pages 768–775, Singapore. Association for Computational Linguistics.
  20. Tom Kocmi and Christian Federmann. 2023b. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203, Tampere, Finland. European Association for Machine Translation. ↗
  21. Pedro Henrique Martins, João Alves, Patrick Fernandes, Nuno M. Guerreiro, Ricardo Rei, Amin Farajian, Mateusz Klimaszewski, Duarte M. Alves, José Pombal, Nicolas Boizard, Manuel Faysse, Pierre Colombo, François Yvon, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2025. Eurollm-9b: Technical report. Preprint, arXiv:2506.04079. ↗
  22. Mistral-Team. 2025. Introducing mistral 3. find
  23. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2021. Investigating code-mixed Modern Standard Arabic-Egyptian to English machine translation. In Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching, pages 56–64, Online. Association for Computational Linguistics.
  24. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022a. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  25. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022b. TURJUMAN: A public toolkit for neural Arabic machine translation. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 1–11, Marseille, France. European Language Resources Association. ↗
  26. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  27. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. ↗
  28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  29. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. ↗
  30. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics. ↗
  31. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
  32. Aziz Mohammed Abdo Saeed. 2025. Machine translation evaluation between arabic and english during 2020 to 2024: A review study. Arts for Linguistic & Literary Studies, 7(2):665–678. ↗
  33. Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi. 2020. AraBench: Benchmarking dialectal Arabic-English machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5094–5107, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  34. Hassan Sajjad, Kareem Darwish, and Yonatan Belinkov. 2013. Translating dialectal Arabic to English. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–6, Sofia, Bulgaria. Association for Computational Linguistics. find
  35. Wael Salloum and Nizar Habash. 2013. Dialectal Arabic to English machine translation: Pivoting through Modern Standard Arabic. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 348–358, Atlanta, Georgia. Association for Computational Linguistics.
  36. Guokan Shang, Hadi Abdine, Ahmad Chamma, Amr Mohamed, Mohamed Anwar, Abdelaziz Bounhar, Omar El Herraoui, Preslav Nakov, Michalis Vazirgiannis, and Eric P. Xing. 2025. Nile-chat: Egyptian language models for Arabic and Latin scripts. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 306–322, Suzhou, China. Association for Computational Linguistics.
  37. Gemma Team. 2025. Gemma 3. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗
  38. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F. Zaidan, and Chris Callison-Burch. 2012. Machine translation of Arabic dialects. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 49–59, Montréal, Canada. Association for Computational Linguistics. Model Size BLEU CHRF Arabic-Specialized Jais-2-70B-Chat Jais-2-8B-Chat Nile-Chat-12B c4ai-command-r7b-arabic-02-2025 70B 8B 12B 7B 25.4 14.5 7.3 8.1 51.8 41.6 34.1 34.5
  39. EuroLLM-9B-Instruct 9B Llama-3.3-70B-Instruct 70B Mistral-Small-3.2-24B-Instruct-2506 24B Qwen3-4B-Instruct-2507 4B aya-expanse-32b 32B aya-expanse-8b 8B c4ai-command-r-08-2024 32B c4ai-command-r-v01 35B command-a-translate-08-2025 111B gemma-3-27b-it 27B gemma-3-4b-it 4B 7.0 8.6 6.0 4.6 9.9 6.5 11.1 5.4 8.7 10.5 7.1 33.3 35.3 28.4 27.8 38.0 33.2 39.1 30.2 37.7 39.2 32.7 15.0 12.4 44.8 42.1 Multilingual Commercial API gpt-4.1-mini gpt-4.1-nano N/A N/A *Table 5: Overall BLEU and ChrF scores computed on the concatenation of all test sets translated using the dialect-specific prompts.* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jon-2026-current-state-llms-arabic-dialectal/