Elamur Research⌕
☰
home / papers / 2018 / Unified Guidelines and Resources for Arabic Dialect Orthography

Unified Guidelines and Resources for Arabic Dialect Orthography

Summary

Modern Standard Arabic has a fixed spelling standard, but spoken Arabic dialects have none. As a result, one dialectal word can appear online in dozens of forms, which fragments data and hurts NLP systems. Earlier Conventional Orthography for Dialectal Arabic (CODA) work covered one dialect at a time. Its rules were often vague or ad hoc, and it only considered dialect text as input to systems, ignoring readable output from ASR, MT or chatbots. The authors propose CODA*, a dialect-independent version with four parts:

  • general rules kept apart from dialect-specific specifications,
  • a phonological notation called CAPHI, modeled on Arpabet,
  • a multidialectal Seed Lexicon recording earlier spelling decisions,
  • public web pages exposing these resources.

They apply it to 28 city dialects across the Arab world, from Rabat to Muscat. Three large dialect-processing projects at three universities already use the framework. This makes it a practical shared basis for consistent annotation and for generating dialect text that people can read.

Key points

  • The motivating example is the Egyptian word /mabiPulha:S/ ('he does not say it'). It has 27 attested spellings on Google Search. Variation comes from separate choices, such as attaching or detaching the negation clitic, and writing the stem by sound versus by etymology.
  • Coverage is 28 city dialects in four macro-regions: Maghreb, Nile Basin, Fertile Crescent and Arabian Peninsula. Examples include Fes, Sfax, Aswan, Salt, Mosul, Taiz and Doha.
  • CODA* builds on earlier single-dialect versions: CODA-Egyptian, Tunisian, Palestinian, Algerian, Gulf, Moroccan and Yemeni.
  • The noise problem is large:
    • about 32% of words in online MSA comments contain spelling errors;
    • close to 24% of Egyptian Arabic words do not follow CODA;
    • in MT evaluation, 10% of output tokens judged correct but missing from the reference were orthographic variants of reference tokens.
  • Current users are MADAR (CMU Qatar & NYUAD), the Gulf Arabic Annotated Corpus (NYUAD) and the Columbia Arabic Dialect Annotation project.

Interesting ideas

  • I1. Work at the city level instead of the region level. Picking concrete cities avoids fuzzy labels like "Levantine", which blur a continuum. This is useful for any dialect continuum where regional groupings are contested.
  • I2. Design spelling conventions for output as well as input. Normalization rules that suit a parser may look odd to readers. If a system will generate text (ASR transcripts, MT, chatbots), the convention must stay familiar to native readers.
  • I3. Separate universal rules from per-variety specifications. Exceptions are kept out of the core rules and managed in their own layer. This makes it much easier to extend the scheme to a new variety without rewriting it.
  • I4. Keep a shared seed lexicon of past decisions. Annotators working on a new dialect can check how cognate words were spelled elsewhere. This promotes consistency across varieties and reduces repeated debates, which helps any multi-team annotation effort.
  • I5. Measure spelling chaos with web frequency counts. Counting search-engine hits for each variant of a single word cheaply shows how serious the inconsistency problem is. The same counts also show which variants are most common.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 52 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{habash2018unified,
  title = {{Unified Guidelines and Resources for Arabic Dialect Orthography}},
  author = {Habash, Nizar and Eryani, Fadhl and Khalifa, Salam and Rambow, Owen and Abdulrahim, Dana and Erdmann, Alexander and Faraj, Reem and Zaghouani, Wajdi and Bouamor, Houda and Zalmout, Nasser and Hassan, Sara and Al-Shargi, Faisal and Alkhereyf, Sakhar and Abdulkareem, Basma and Eskander, Ramy and Salameh, Mohammad and Saddiki, Hind},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  doi = {10.63317/2k835qnyjdtw},
  url = {https://doi.org/10.63317/2k835qnyjdtw}
}

References

32 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abu Ali, D. and Habash, N. (2016). Botta: An arabic dialect chatbot. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations, pages 208–212, Osaka, Japan, December. find
  2. Al-Badrashiny, M., Eskander, R., Habash, N., and Rambow, O. (2014). Automatic Transliteration of Romanized Dialectal Arabic. CoNLL-2014, page 30. find
  3. Al-Shargi, F., Kaplan, A., Eskander, R., Habash, N., and Rambow, O. (2016). A Morphologically Annotated Corpus and a Morphological Analyzer for Moroccan and Sanaani Yemeni Arabic. In Proceedings of the International Conference on Language Resources and Evaluation (LREC), Portorož, Slovenia. find
  4. Ali, A., Mubarak, H., and Vogel, S. (2014). Advances in dialectal arabic speech recognition: A study using twitter to. In Improve Egyptian ASR. International Workshop on Spoken Language Translation (IWSLT. Citeseer. find
  5. Alkuhlani, S. and Habash, N. (2011). A Corpus for Modeling Morpho-Syntactic Agreement in Arabic: Gender, Number and Rationality. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL’11), Portland, Oregon, USA. find our card
  6. Arkadiusz, P. (2006). Le nationalisme linguistique au liban autour de sa‘id ‘aql et l’idée de langue libanaise dans la revue “lebnaan” en nouvel alphabet. Arabica, 53(4):423–471. find
  7. ‘Asaakir, K. (1950). A Method for Writing Modern Arabic Dialects with Arabic Letters. (in Arabic). The Arab Academy Magazine, 8(181–192), January. find
  8. Badawi, E.-S. and Hinds, M. (1986). A Dictionary of Egyptian Arabic. Librairie du Liban. find
  9. Bies, A., Song, Z., Maamouri, M., Grimes, S., Lee, H., Wright, J., Strassel, S., Habash, N., Eskander, R., and Rambow, O. (2014). Transliteration of arabizi into arabic orthography: Developing a parallel annotated arabizi-arabic script sms/chat corpus. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 93–103. find
  10. Bouamor, H., Habash, N., Salameh, M., Zaghouani, W., Rambow, O., Abdulrahim, D., Obeid, O., Khalifa, S., Eryani, F., Erdmann, A., and Oflazer, K. (2018). The MADAR Arabic Dialect Corpus and Lexicon. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018), May. find
  11. Darwish, K. (2013). Arabizi Detection and Conversion to Arabic. CoRR. ↗
  12. Diab, M., Habash, N., Rambow, O., AlTantawy, M., and Benajiba, Y. (2010). Colaba: Arabic dialect annotation and processing. In Proceedings of the LREC Workshop for Language Resources (LRs) and Human Language Technologies (HLT) for Semitic Languages: Status, Updates, and Prospects. find
  13. Diab, M. T., Al-Badrashiny, M., Aminian, M., Attia, M., Elfardy, H., Habash, N., Hawwari, A., Salloum, W., Dasigi, P., and Eskander, R. (2014). Tharwa: A large scale dialectal arabic-standard arabic-english lexicon. In LREC, pages 3782–3789. find
  14. Erdmann, A., Habash, N., Taji, D., and Bouamor, H. (2017). Low Resourced Machine Translation via Morpho-syntactic Modeling: The Case of Dialectal Arabic. In MT Summit. ↗
  15. Eskander, R., Habash, N., Rambow, O., and Tomeh, N. (2013). Processing Spontaneous Orthography. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Atlanta, GA. find
  16. Habash, N., Soudi, A., and Buckwalter, T. (2007). On Arabic Transliteration. In A. van den Bosch et al., editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. ↗
  17. Habash, N., Diab, M., and Rabmow, O. (2012). Conventional Orthography for Dialectal Arabic. In Proceedings of LREC, Istanbul, Turkey. find our card
  18. Habash, N. Y. (2010). Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers. find
  19. Jarrar, M., Habash, N., Akra, D., and Zalmout, N. (2014). Building a Corpus for Palestinian Arabic: a Preliminary Study. ANLP 2014, page 18. find
  20. Jarrar, M., Habash, N., Alrimawi, F., Akra, D., and Zalmout, N. (2016). Curras: an annotated corpus for the Palestinian Arabic dialect. Language Resources and Evaluation, pages 1–31. ↗
  21. Khalifa, S., Habash, N., Abdulrahim, D., and Hassan, S. (2016). A Large Scale Corpus of Gulf Arabic. In Proceedings of the International Conference on Language Resources and Evaluation (LREC), Portorož, Slovenia. find
  22. Khalifa, S., Habash, N., Eryani, F., Obeid, O., Abdulrahim, D., and Kaabi, M. A. (2018). A morphologically annotated corpus of emirati arabic. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018), May. find
  23. Maamouri, M., Buckwalter, T., and Cieri, C. (2004). Dialectal Arabic Telephone Speech Corpus: Principles, Tool design, and Transcription Conventions. In NEM-LAR International Conference on Arabic Language Resources and Tools. find
  24. Maamouri, M., Bies, A., Kulick, S., Ciul, M., Habash, N., and Eskander, R. (2014). Developing an egyptian arabic treebank: Impact of dialectal morphology on annotation and tool development. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). European Language Resources Association (ELRA). find
  25. Meftouh, K., Harrat, S., Jamoussi, S., Abbas, M., and Smaili, K. (2015). Machine translation experiments on PADIC: A parallel Arabic dialect corpus. In The 29th Pacific Asia conference on language, information and computation. find
  26. Pasha, A., Al-Badrashiny, M., Kholy, A. E., Eskander, R., Diab, M., Habash, N., Pooleery, M., Rambow, O., and Roth, R. (2014). MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In In Proceedings of LREC, Reykjavik, Iceland. find our card
  27. Saadane, H. and Habash, N. (2015). A Conventional Orthography for Algerian Arabic. In ANLP Workshop 2015, page 69. find
  28. Shoup, J. E. (1980). Phonological aspects of speech recognition. Trends in speech recognition, pages 125–138. find
  29. Watson, J. C. (2007). The Phonology and Morphology of Arabic. Oxford University Press. find
  30. Zaghouani, W., Mohit, B., Habash, N., Obeid, O., Tomeh, N., Rozovskaya, A., Farra, N., Alkuhlani, S., and Oflazer, K. (2014). Large scale Arabic error annotation: Guidelines and framework. In International Conference on Language Resources and Evaluation (LREC 2014). find our card
  31. Zalmout, N., Erdmann, A., and Habash, N. (2018). Noise-robust morphological disambiguation for dialectal arabic. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2018). find our card
  32. Zribi, I., Boujelbane, R., Masmoudi, A., Ellouze Khmekhem, M., Hadrich Belguith, L., and Habash, N. (2014). A Conventional Orthography for Tunisian Arabic. In In Proceedings of LREC, Reykjavik, Iceland. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/habash-2018-unified-guidelines-resources-arabic-dialect/