Elamur Research⌕
☰
home / papers / 2026 / Ara-HOPE: Human-Centric Post-Editing Evaluation for Dialectal Arabic to Modern Standard Arabic Translation

Ara-HOPE: Human-Centric Post-Editing Evaluation for Dialectal Arabic to Modern Standard Arabic Translation

Summary

Translating Dialectal Arabic (DA) into Modern Standard Arabic (MSA) is closer to normalizing a variety than to translating between languages. Dialects differ from MSA in spelling, morphology, vocabulary, syntax and code-switching. Overlap metrics such as BLEU and AL-BLEU, and embedding metrics like BERTScore and COMET, handle these differences poorly. Detailed human schemes like MQM require heavily trained annotators. The authors adapt the HOPE post-editing metric into Ara-HOPE, a lighter human evaluation scheme for Levantine-to-MSA output. It has five error categories under three top-level classes, a three-step severity scale tied to post-editing effort, and a yes/no decision tree that minimally trained native speakers can follow. They used it to judge zero-shot translations from Jais, GPT-3.5 and NLLB-200. The framework exposes systematic differences between the systems. Across all three, dialect-specific terms and preservation of meaning account for most errors. This tells MT developers where to focus, and it gives low-resource varieties a practical evaluation recipe. The annotations are released publicly.

Key points

  • Taxonomy. The categories are Fluency (FLU), Meaning Transfer and Adaptation (ADP). Meaning Transfer splits into Proper Name (PRN), Dialect-Specific Term (TRM) and General Semantic Mistranslation (GSMIS).
  • Severity scale. Severity has three levels: 0 marks an error-free segment, and 2 marks a major error. The scale was shrunk deliberately to cut disagreement between annotators.
  • Design principles. Six principles shaped the scheme: identifiability, distinguishability, actionability, coverage, relevance and usability. They were refined through focus groups with native DA speakers.
  • Systems compared. The three systems were Arabic-centric Jais, general-purpose GPT-3.5 and the NLLB-200 baseline, all prompted zero-shot. TRM and GSMIS made up the bulk of errors for every system.
  • Limitations. The authors did not measure correlation with automatic metrics. They suggest LLM-as-a-judge as a way to reduce the annotation cost.

Interesting ideas

  • I1. An error category was added only after a gap appeared in real data: GSMIS was created once annotators kept seeing LLM meaning distortions that PRN and TRM could not cover. This keeps a taxonomy grounded in what actually happens. — For Mova (learn): Mova's negative-control evaluation could grow its error classes the same way, adding a catch-all bucket for answer failures and splitting it only once recurring patterns are visible.
  • I2. A flat error checklist was turned into a yes/no decision tree, which lowers annotator load and makes labels more consistent. Each verdict becomes a traceable path of answers. — For Mova (answer): Graders for school tests and fable QA could classify wrong answers by walking a binary tree (parse issue? wrong entity? wrong event order?), so every failure is diagnosed in an explainable way.
  • I3. Severity is tied to the effort needed to fix an output, not to an abstract quality score, and the scale has only three levels. This keeps scores comparable and actionable. — For Mova (back): Generated text and compressor round-trips could be scored by how much editing they need on a 0–2 scale per category, separating fluency problems from meaning loss.
  • I4. Fluency of the output is judged apart from fidelity to the source, so a fluent but wrong translation is still flagged. — For Mova (back): Mova's text→graph→text checks could score target-side grammaticality separately from graph-preservation errors, so that smooth wording cannot hide a dropped role.
  • I5. Dialect expressions are treated as units whose meaning comes from context, since word-by-word mapping fails on idioms and casual intent such as "what's wrong". — For Mova (frames): The prag crate could map idiomatic multiword expressions directly to speech-act or frame readings instead of composing them literally, much as Jais succeeded where NLLB-200 stayed literal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

2 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 1 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.vardial-1.13

@inproceedings{alabdullah2026ara,
  title = {{Ara-HOPE: Human-Centric Post-Editing Evaluation for Dialectal Arabic to Modern Standard Arabic Translation}},
  author = {Alabdullah, Abdullah and Han, Lifeng and Lin, Chenghua},
  booktitle = {Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects},
  year = {2026},
  doi = {10.18653/v1/2026.vardial-1.13},
  url = {https://doi.org/10.18653/v1/2026.vardial-1.13}
}

References

34 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Amr Keleg, AbdelRahim Elmadany, Chiyu Zhang, Injy Hamed, Walid Magdy, Houda Bouamor, and Nizar Habash. 2024. NADI 2024: The fifth nuanced Arabic dialect identification shared task. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 709–728, Bangkok, Thailand. Association for Computational Linguistics. find
  2. Abdullah Alabdullah, Lifeng Han, and Chenghua Lin. 2025. Advancing dialectal arabic to modern standard arabic machine translation. Preprint, arXiv:2507.20301. ↗ our card
  3. Bashar Alhafni, Sarah Al-Towaity, Ziyad Fawzy, Fatema Nassar, Fadhl Eryani, Houda Bouamor, and Nizar Habash. 2024. Exploiting dialect identification in automatic dialectal text normalization. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 42–54, Bangkok, Thailand. Association for Computational Linguistics. find
  4. Hanin Atwany, Nour Rabih, Ibrahim Mohammed, Abdul Waheed, and Bhiksha Raj. 2024. OSACT 2024 task 2: Arabic dialect to MSA translation. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024, pages 98–103, Torino, Italia. ELRA and ICCL. find our card
  5. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics. find our card
  6. Laith H. Baniata, Seyoung Park, and Seong-Bae Park. 2018. A neural machine translation model for arabic dialects that utilises multitask learning (mtl). find
  7. Computational Intelligence and Neuroscience, 2018. Publisher Copyright: © 2018 Laith H. Baniata et al. find
  8. Fadi Biadsy, Julia Hirschberg, and Nizar Habash. 2009. Spoken arabic dialect identification using phonotactic modeling. In Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages, Semitic ’09, page 53–61, USA. Association for Computational Linguistics. find
  9. Houda Bouamor, Hanan Alshikhabobakr, Behrang Mohit, and Kemal Oflazer. 2014. A human judgement corpus and a metric for arabic mt evaluation. In Conference on Empirical Methods in Natural Language Processing. find
  10. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The madar arabic dialect corpus and lexicon. In International Conference on Language Resources and Evaluation. find
  11. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc. find
  12. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement. Psychological Bulletin, 70(4):213–220. find
  13. Mona Diab, Nizar Habash, Owen Rambow, Mohamed Altantawy, and Yassine Benajiba. 2010. Colaba: Arabic dialect annotation and processing. In Lrec workshop on semitic language processing, pages 66– 74. find
  14. Júlia Falcão, Claudia Borg, Nora Aranberri, and Kurt Abela. 2024. COMET for low-resource machine translation evaluation: A case study of English-Maltese and Spanish-Basque. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3553–3565, Torino, Italia. ELRA and ICCL. for human evaluation of machine translation. In Proceedings of the Ninth Conference on Machine Translation, pages 1440–1453, Miami, Florida, USA. Association for Computational Linguistics. find
  15. Joseph L. Fleiss, Bruce Levin, and Myunghee Cho Paik. 2003. Statistical Methods for Rates and Proportions. Wiley. find
  16. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. find
  17. Serge Gladkoff and Lifeng Han. 2022. HOPE: A task-oriented and human-centric evaluation framework using professional post-editing towards more effective MT evaluation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 13–21, Marseille, France. European Language Resources Association. find
  18. Arle Lommel, Serge Gladkoff, Alan Melby, Sue Ellen Wright, Ingemar Strandvik, Katerina Gasova, Angelika Vaasa, Andy Benzo, Romina Marazzato Sparano, Monica Foresi, Johani Innis, Lifeng Han, and Goran Nenadic. 2024. The multi-range theory of translation quality measurement: MQM scoring models and statistical quality control. In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: Presentations), pages 75–94, Chicago, USA. Association for Machine Translation in the Americas. find
  19. Serge Gladkoff, Irina Sorokina, Lifeng Han, and Alexandra Alekseeva. 2022. Measuring uncertainty in translation quality evaluation (TQE). In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1454–1461, Marseille, France. European Language Resources Association. find
  20. Najet Hadj Mohamed, Malak Rassem, Lifeng Han, and Goran Nenadic. 2023. AlphaMWE-Arabic: Arabic edition of multilingual parallel corpora with multiword expression annotations. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 448– 457, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. find
  21. Injy Hamed, Caroline Sabty, Slim Abdennadher, Ngoc Thang Vu, Thamar Solorio, and Nizar Habash. 2025. A survey of code-switched Arabic NLP: Progress, challenges, and future directions. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4561–4585, Abu Dhabi, UAE. Association for Computational Linguistics. find
  22. Chao Han. 2020. Translation quality assessment: a critical methodological review. The Translator, 26(3):257–273. find
  23. Abdullah Khered, Youcef Benkhedda, and Riza Batista-Navarro. 2025. Dial2MSA-verified: A multi-dialect Arabic social media dataset for neural machine translation to Modern Standard Arabic. In Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4), pages 50–62, Abu Dhabi, UAE. Association for Computational Linguistics. find
  24. Katrin Kirchhoff, Dimitra Vergyri, Jeff Bilmes, Kevin Duh, and Andreas Stolcke. 2006. Morphology-based language modeling for conversational arabic speech recognition. Computer Speech & Language, 20(4):589–608. find
  25. Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, and Mariya Shmatova. 2024. Error span annotation: A balanced approach Arle Richard Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2013. Multidimensional quality metrics: a flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35, London, UK. Aslib. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. find our card
  26. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics. find
  27. Hamdy Mubarak. 2018. Dial2msa: A tweets corpus for converting dialectal arabic to modern standard arabic. OSACT, 3:49. find
  28. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics. ↗ our card
  29. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  30. Irene Rivera-Trigueros. 2022. Machine translation systems and quality assessment: a systematic review. Lang. Resour. Eval., 56(2):593–619. find
  31. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, and 13 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. Preprint, arXiv:2308.16149. ↗ our card
  32. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, and 20 others. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672. ↗
  33. Silvana Yakhni and Ali Chehab. 2025. Can LLMs translate cultural nuance in dialects? a case study on Lebanese Arabic. In Proceedings of the 1st Workshop on NLP for Languages Using Arabic Script, pages 114–135, Abu Dhabi, UAE. Association for Computational Linguistics. find
  34. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with BERT. arXiv preprint arXiv:1904.09675. *Figure 5: The Arabic version of the Ara-HOPE Annotation Decision Tree. A structured decision tree guiding annotators through error classification for evaluating DA-MSA translations using the Ara-HOPE framework.* *Figure 6: Structured decision tree guiding annotators through error classification for evaluating DA-MSA translations using the Ara-HOPE framework.* *Figure 7: The annotation guidelines provided to human annotators explain each error type with illustrative examples, assisting them in using the Ara-HOPE Annotation Decision Tree.* ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alabdullah-2026-ara-hope-human-centric-post/