Elamur Research⌕
☰
home / papers / 2025 / Arabizi vs LLMs: Can the Genie Understand the Language of Aladdin?

Arabizi vs LLMs: Can the Genie Understand the Language of Aladdin?

Summary

The paper asks whether general-purpose LLMs can translate Arabizi well enough for gisting without first converting it to Arabic script. Arabizi is the informal, Latin-letters-plus-digits way of writing spoken Arabic dialects, and it is common in chat and social media. The study came from a client request received by iguanodon.ai. The authors built AladdinBench, a set of genuine WhatsApp messages in Lebanese, Egyptian and Algerian Arabizi. Thirty-one native speakers donated the messages, which were then anonymised, and a professional translator produced reference translations into MSA and English. Several instruction-tuned LLMs were then run through a grid of settings: target language, prompt language, and zero-, one- or two-shot prompting. Outputs were scored with n-gram metrics, embedding-based metrics, human raters and an LLM judge. Egyptian was the easiest dialect and Algerian the hardest. GPT-4o was the strongest model, and prompts written in English worked better. The LLM judge agreed with the expert raters. The authors conclude that off-the-shelf LLMs are usable for gisting Arabizi, though far from perfect, and that the LLM judge can serve as a quality check.

Key points

  • AladdinBench is published on Hugging Face (palmaoui/AladdinBench). It contains WhatsApp chats from speakers aged 20–35, collected from two regions per country: north/south Lebanon, Cairo/Luxor and Algiers/Constantine. For example, the Lebanon North subset has 127 segments and Lebanon South has 141.

  • The models tested were GPT-4o (2024-08-06), Llama 3.3 70B-Instruct, Claude, Gemini, Gemma (27B), Mistral and Jais, all run at temperature 0.5. Code is available on GitHub (iguanodon-ai/ArabizivsLLMs).

  • Average scores by dialect:

    • Egyptian: BLEU 9.65, chrF 34.64, COMET 0.67.
    • Lebanese: BLEU 7.52, chrF 26.59, COMET 0.65.
    • Algerian: BLEU 4.24, chrF 23.21, COMET 0.63.

    The authors attribute Algerian's low scores to heavy code-switching with French, Berber and Spanish, plus thin coverage in training data.

  • Model ranking: GPT-4o first, then Gemini, then Mistral Large and Gemma; Llama 3 and Jais were the weakest. Gemma did well despite its small size.

  • Few-shot examples helped, more so for English output than for MSA output. English prompts beat MSA prompts throughout. Segment length showed no clear link to scores.

Interesting ideas

  • I1. Every combination of target language, prompt language, number of shots, dialect-specific prompt and model was run as one full factorial grid, so each factor's effect can be read off separately rather than guessed.
  • I2. Each corpus slice reports its share of foreign and "mixed" words (a root from one language with affixes from another), a cheap per-slice difficulty measure that explained the gap between dialects.
  • I3. The LLM judge was not trusted on its own; its verdicts were first checked against expert human ratings and only then adopted.
  • I4. The authors watched for cases where surface-overlap scores rose while meaning-based scores stayed flat or fell, and treated this as improvement in form only.
  • I5. Data was collected in balanced amounts from contrasting sub-regions within each variety, so within-dialect variation is represented instead of a single "standard" voice.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation runs for LLM teacher tiers could sweep prompt variant, shot count and model in one grid, showing which choice actually moves silver-data quality.
  • I2 (tokens): The tokenizer/morphology report could count hybrid or foreign-affixed word forms per input text and flag such texts as likely low-confidence before lemmatisation.
  • I3 (learn): Before an LLM teacher tier is allowed to grade parses or frames, its judgements could be correlated with a small hand-checked sample, used as a gate.
  • I4 (back): The text → graph → text compressor could pair a surface metric with a meaning-level comparison of the re-parsed graph, so that word-overlap gains alone do not count as success.
  • I5 (tree): When building or converting treebanks, test sets could be stratified by register or source (chat, fables, school texts) so parser weaknesses on one variety are not hidden by averages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics Language models & pretraining

Cite

Cite as Perla Al Almaoui, Pierrette Bouillon and Simon Hengchen. 2025. Arabizi vs LLMs: Can the Genie Understand the Language of Aladdin?. Proceedings of Machine Translation Summit XX: Volume 2.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{almaoui2025arabizi,
  title = {{Arabizi vs LLMs: Can the Genie Understand the Language of Aladdin?}},
  author = {Perla {Al Almaoui}, Pierrette Bouillon and Simon Hengchen},
  booktitle = {Proceedings of Machine Translation Summit XX: Volume 2},
  year = {2025},
  url = {https://aclanthology.org/2025.mtsummit-2.4/}
}

References

35 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Perla Al Almaoui. 2024. ChatGPT comme outil de traduction automatique des discussions WhatsApp : évaluation de la traduction de différents dialectes en Arabizi. Master’s thesis, Université de Genève. find
  2. Wid H Allehaiby. 2013. Arabizi: An analysis of the romanization of the arabic script from a sociolinguistic perspective. Arab World English Journal, 4(3). Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku. find
  3. Nur Ayoubi. 2022. Shou, shinou, ey: Five major arabic dialects and what makes them unique. Middle East Eye. Accessed: 31 March 2025. find
  4. Ann Bies, Zhiyi Song, Mohamed Maamouri, Stephen Grimes, Haejoong Lee, Jonathan Wright, Stephanie Strassel, Nizar Habash, Ramy Eskander, and Owen Rambow. 2014. Transliteration of Arabizi into Arabic orthography: Developing a parallel annotated Arabizi-Arabic script SMS/chat corpus. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 93–103, Doha, Qatar. Association for Computational Linguistics. find
  5. Abdelkarim Chami. 2009. A historical background of the linguistic situation in algeria. find
  6. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. https://calculator.green-algorithms.org/ai ↗
  8. Gemini Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. ↗ our card
  9. Gemma Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. ↗
  10. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538. find
  11. Imane Guellil, Faiçal Azouaou, Mourad Abbas, and Sadat Fatiha. 2017. Arabizi transliteration of Algerian Arabic dialect into Modern Standard Arabic. In Social MT 2017/ First workshop on Social Media and User Generated Content Machine Translation, Prague, Czech Republic. find
  12. Salima Harrat, Karima Meftouh, and Kamel Smaili. 2019. Machine translation for arabic dialects (survey). Information Processing & Management, 56(2):262–273. Advance Arabic Natural Language Processing (ANLP) and its Applications. find
  13. Sui He. 2024. Prompting ChatGPT for translation: A comparative analysis of translation brief and persona prompts. In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 316–326, Sheffield, UK. European Association for Machine Translation (EAMT). find
  14. Amine Jules Iskandar. 2022. À l’origine du parler libanais (parts 1 and 2). Part 1 available at: https://terredecompassion.com/2022/03/01/a-lorigine-du-parler-libanais-1-2/, Part 2 available at: https://terredecompassion.com/2022/03/08/a-lorigine-du-parler-libanais-2-2/. Accessed: 2025-01-01. ↗
  15. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings on the Workshop on Statistical Machine Translation, pages 102– 121, New York City. Association for Computational Linguistics. our card
  16. Loïc Lannelongue, Jason Grealey, and Michael Inouye. 2021. Green algorithms: quantifying the carbon footprint of computation. Advanced science, 8(12):2100707. find
  17. Alexander Magidow. 2021. The old and the new: Considerations in arabic historical dialectology. Languages, 6(4):163. Accessed: 31 March 2025. find
  18. Imane Momayiz, Aissam Outchakoucht, Omar Choukrani, and Ali Nirheche. 2024. Terjamabench: A culturally specific dataset for evaluating translation models for moroccan darija. find
  19. OpenAI. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. ↗ our card
  20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗ our card
  21. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. ↗ our card
  22. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics. ↗ our card
  23. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. find
  24. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. our card
  25. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. our card
  26. Houda Saadane and Nizar Habash. 2015. A conventional orthography for Algerian Arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 69–79, Beijing, China. Association for Computational Linguistics. find
  27. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗ our card
  28. Ali Shazal, Aiza Usman, and Nizar Habash. 2020. A unified model for Arabizi detection and transliteration using sequence-to-sequence models. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 167–177, Barcelona, Spain (Online). Association for Computational Linguistics. find
  29. Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas. our card
  30. Lameen Souag. 2009. Siwa and its significance for Arabic dialectology. Zeitschrift für arabische Linguistik = Journal of Arabic linguistics = Journal de linguistique arabe, 51:51–75. find
  31. C. Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72–101. find
  32. Nikolai Vladimirovich Yushmanov. 1961. The structure of the arabic language. find
  33. Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine translation: A case study. Preprint, arXiv:2301.07069. ↗ our card
  34. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. ↗
  35. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/almaoui-2025-arabizi-vs-llms-can-genie/