Elamur Research⌕
☰
home / papers / 2014 / Automatic Transliteration of Romanized Dialectal Arabic

Automatic Transliteration of Romanized Dialectal Arabic

Paper · 2014 · Mohamed Al-Badrashiny, Ramy Eskander, Nizar Habash et al.

Summary

Egyptian Arabic (EGY) is often typed in Latin letters and digits, a practice called Arabizi. Existing EGY tools expect Arabic script, so they cannot read this text. The task is also hard to define, because the dialect has no standard spelling even in Arabic script. The authors fix the target as CODA, a conventional orthography designed for NLP, and frame the task as Arabizi → CODA conversion.

Their system, 3ARRIB, works in three steps:

  1. A character-level finite state transducer proposes every plausible Arabic-script spelling for each word.
  2. The CALIMA morphological analyzer removes candidates that are not valid EGY words.
  3. A language model picks the best surviving candidate in context.

CODAFY is used as a black-box preprocessing module. The baseline re-implements Darwish's (2013) design, which has no morphological filter. On a blind test set the full pipeline reaches 69.4% accuracy against 63.1% for the baseline, about a 17% error reduction. This matters because standardized output reduces sparsity for downstream EGY tools such as taggers, analyzers and MT.

Key points

  • Configurations on Dev (exact match):
    • FST-CALIMA-Tokenized-LM-5 scores 77.5%, against 74.1% for the Darwish-style FST-Untokenized-LM-5.
    • FST top choice alone gives 63.1%, and adding CALIMA filtering gives 66.1%.
    • Lookup of words seen in FST training data (INV-Selection) gives only 37.1%.
  • Test set: 69.4% vs 63.1% with exact match, and 73.9% vs 65.4% with Alif/Ya normalization.
  • LM order: 3-gram and 5-gram LMs performed equally, while the 7-gram LM was worst. Tokenized and untokenized LMs gave nearly identical results (77.5 vs 77.3).
  • MLE shortcut hurt: forcing the most frequent training mapping for seen words dropped Dev accuracy to 68.7%.
  • Error analysis:
    • Of the Dev errors, 29% are lost at the FST, 15% at CALIMA and 8% at the LM. The remaining 48% are cases where the correct candidate survived but was not chosen.
    • EGY-only words reach 84.6% accuracy, while pass-through items such as English, names, emoticons and digits reach only 46.2%.

Interesting ideas

  • I1. The system first generates many candidates with a cheap character model, then prunes them with a real morphological analyzer, then lets context choose; each stage stays small and inspectable.
  • I2. Before building the converter, the authors pinned down one canonical target spelling (CODA), which made both the task and its evaluation well-defined.
  • I3. Errors were traced to the stage where the gold answer first disappears, separating coverage loss at each component from bad final selection.
  • I4. A memorized most-frequent mapping for seen words performed worse than running the full contextual pipeline on them, so lookup shortcuts can quietly cost accuracy.
  • I5. Accuracy was split by word type (native dialect words vs pass-through items like emoticons, names and code-switched English), and the split revealed a weakness the overall score hid.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Unknown or misspelled English words could get candidate lemmas from a character model, filtered by the morphology lexicon, with the final pick made by sentence context.
  • I2 (tree): UD dialect conversion between treebanks could first declare one explicit canonical target scheme, so that conversion errors become measurable instead of being matters of taste.
  • I3 (learn): Evaluation reports could record, per pipeline stage, whether the correct analysis is still among the candidates, showing whether to fix generation or ranking.
  • I4 (learn): Frequency-lookup baselines could serve as standing negative controls, so that any memorization shortcut must prove it beats the contextual model.
  • I5 (answer): Test results on school tests and fable QA could be broken down by question type, so that a weak category is not masked by a strong average.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓George Washington UniversityWashington · United States51 works · 8,148 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Mohamed Al-Badrashiny, Ramy Eskander, Nizar Habash and Owen Rambow. 2014. Automatic Transliteration of Romanized Dialectal Arabic. Proceedings of the Eighteenth Conference on Computational Natural Language Learning.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-1604

@inproceedings{albadrashiny2014automatic,
  title = {{Automatic Transliteration of Romanized Dialectal Arabic}},
  author = {Mohamed Al-Badrashiny, Ramy Eskander, Nizar Habash and Owen Rambow},
  booktitle = {Proceedings of the Eighteenth Conference on Computational Natural Language Learning},
  year = {2014},
  doi = {10.3115/v1/w14-1604},
  url = {https://doi.org/10.3115/v1/w14-1604}
}

References

25 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G. Al-Gaphari and M. Al-Yadoumi. 2010. A method to convert Sana’ani accent to Modern Standard Arabic. International Journal of Information Science and Management, pages 39–49. find
  2. Yaser Al-Onaizan and Kevin Knight. 2002. Machine transliteration of names in arabic text. In Proceedings of the ACL-02 Workshop on Computational Approaches to Semitic Languages. find
  3. Tim Buckwalter. 2004. Buckwalter Arabic Morphological Analyzer Version 2.0. LDC catalog number LDC2004L02, ISBN 1-58563-324-0. ↗
  4. Achraf Chalabi and Hany Gerges. 2012. Romanized Arabic Transliteration. In Proceedings of the Second Workshop on Advances in Text Input Methods (WTIM 2012).
  5. Kareem Darwish. 2013. Arabizi Detection and Conversion to Arabic. CoRR. ↗
  6. Ahmed El Kholy and Nizar Habash. 2010. Techniques for Arabic Morphological Detokenization and Orthographic Denormalization. In Proceedings of the seventh International Conference on Language Resources and Evaluation (LREC), Valletta, Malta. find
  7. Ramy Eskander, Nizar Habash, Owen Rambow, and Nadi Tomeh. 2013. Processing Spontaneous Orthography. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Atlanta, GA. find
  8. Noura Farra, Nadi Tomeh, Alla Rozovskaya, and Nizar Habash. 2014. Generalized Character-Level Spelling Error Correction. In Proceedings of the Conference of the Association for Computational Linguistics (ACL), Baltimore, Maryland, USA. ↗
  9. Andrew Freeman, Sherri Condon, and Christopher Ackerman. 2006. Cross linguistic name matching in English and Arabic. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 471–478, New York City, USA.
  10. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. ↗
  11. Nizar Habash, Mona Diab, and Owen Rabmow. 2012a. Conventional Orthography for Dialectal Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Istanbul.
  12. Nizar Habash, Ramy Eskander, and Abdelati Hawwari. 2012b. A Morphological Analyzer for Egyptian Arabic. In Proceedings of the Twelfth Meeting of the Special Interest Group on Computational Morphology and Phonology, pages 1–9, Montréal, Canada.
  13. Nizar Habash, Ryan Roth, Owen Rambow, Ramy Eskander, and Nadi Tomeh. 2013. Morphological Analysis and Disambiguation for Dialectal Arabic. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Atlanta, GA.
  14. Nizar Habash. 2008. Four Techniques for Online Handling of Out-of-Vocabulary Words in Arabic-English Statistical Machine Translation. In Proceedings of ACL-08: HLT, Short Papers, pages 57– 60, Columbus, Ohio. find
  15. Nizar Habash. 2010. Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. ↗
  16. Kevin Knight and Jonathan Graehl. 1997. Machine transliteration. In Proceedings of the European chapter of the Association for Computational Linguistics.
  17. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the Association for Computational Linguistics, Prague, Czech Republic. find
  18. Franz Joseph Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. ↗
  19. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan M. Roth. 2014. MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland.
  20. Wael Salloum and Nizar Habash. 2011. Dialectal to Standard Arabic Paraphrasing to Improve Arabic-English Statistical Machine Translation. In Proceedings of the First Workshop on Algorithms and Resources for Modelling of Dialects and Language Varieties, pages 10–21, Edinburgh, Scotland. ↗
  21. Wael Salloum and Nizar Habash. 2013. Dialectal Arabic to English Machine Translation: Pivoting through Modern Standard Arabic. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Atlanta, GA.
  22. Hassan Sawaf. 2010. Arabic dialect handling in hybrid machine translation. In Proceedings of the Conference of the Association for Machine Translation in the Americas (AMTA), Denver, Colorado.
  23. Khaled Shaalan, Hitham Abo Bakr, and Ibrahim Ziedan. 2007. Transferring Egyptian Colloquial into Modern Standard Arabic. In International Conference on Recent Advances in Natural Language Processing (RANLP), Borovets, Bulgaria. find
  24. Andreas Stolcke. 2002. SRILM - an Extensible Language Modeling Toolkit. In Proceedings of the International Conference on Spoken Language Processing (ICSLP), volume 2, pages 901–904, Denver, CO. find
  25. Ines Zribi, Rahma Boujelbane, Abir Masmoudi, Mariem Ellouze, Lamia Belguith, and Nizar Habash. 2014. A Conventional Orthography for Tunisian Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/badrashiny-2014-automatic-transliteration-romanized-dialectal-arabic/