Elamur Research
☰
home / papers / 2012 / A Morphological Analyzer for Egyptian Arabic

A Morphological Analyzer for Egyptian Arabic

What it is about

The paper presents CALIMA, a morphological analyzer for Egyptian Arabic (EGY), built by extending an older lexicon, the Egyptian Colloquial Arabic Lexicon (ECAL, about 66K entries). Four steps: a small set of hand-written finite-state rules inserts diacritics into the undiacritized entries; about 4,600 rules map the ECAL tags to the LDC's EGY tag set and to CODA, a conventional orthography for dialects; the analysis tables are converted to the SAMA format, with complex prefixes, stems and suffixes and their compatibility; then the tables are extended with extra clitics and with non-CODA spellings, which map back to CODA in the output. The result has about 100K stems over 36K lemmas and 48 million analysable word forms. On a blind, manually annotated set of 3,300 words, the correct answer is among the analyses for 84.1% of words, against 57.4% for ECAL and 63.7% for SAMA 3.1 (MSA). Merged with an extended SAMA, the coverage reaches 92.1%. In a review of 100 failures, 30% were gold-annotation errors, 22% non-CODA spellings, 20% incomplete paradigms, and 21% near-misses on case or mood.

Ideas

For Mova

I1 goes into languages/hbo/07-weak-verbs.md and the lexicon of tools/hbo. I2 goes into languages/hbo/01-script.md and the unpointed mode of tools/hbo. I3 is an evaluation step for tools/hbo and tools/macula compare, recorded in languages/hbo/13-mova-tools.md.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W12-2301.pdf

Summary

Arabic NLP tools are mostly built for Modern Standard Arabic (MSA) and handle dialects poorly. Egyptian Arabic (EGY) differs from MSA in sound system, morphology and vocabulary. It also has clitics MSA lacks, such as indirect-object pronouns and the circumfix negation mA+…+š, and it has no standard spelling. The authors built CALIMA (Columbia Arabic Language and dIalect Morphological Analyzer), a large, linguistically faithful out-of-context analyzer for EGY. They started from the Egyptian Colloquial Arabic Lexicon (ECAL), a list of inflected forms, and turned it into prefix/stem/suffix tables of the kind used by SAMA. They then extended these tables to generalize across paradigms and to accept more spelling variants. Output follows the Linguistic Data Consortium POS guidelines for EGY, and variant spellings are mapped to CODA, a conventional dialect orthography. On a manually annotated test set, CALIMA includes the correct analysis for about 84% of words. Combining it with an extended SAMA3.1 raises this to about 92%. This shows that a dialect-specific resource clearly beats MSA tools adapted to dialect input.

Key points

  • Source data: ECAL comes from the CALLHOME Egyptian Arabic corpus (140 telephone conversations). It has about 66K fully inflected forms, with no diacritics and no morpheme segmentation.
  • Correct analysis in the returned set:
    • ECAL: 57.4%
    • SAMA3.1: 63.7%
    • extended SAMA3.1: 68.8%
    • CALIMA-core: 73.9%
    • CALIMA: 84.1%
    • CALIMA + extended SAMA3.1: 92.1%, with only 1.0% of words getting no analysis
  • Effect of the extensions: they make CALIMA about ten times larger than CALIMA-core (48M vs 4M recognizable words) and add roughly 10 points absolute.
  • Error analysis (100 failures of the best combined system):
    • 30% were gold-tag errors.
    • 22% were non-CODA spellings the system did not cover.
    • 20% were incomplete paradigms.
    • 21% were SAMA3.1 analyses that were nearly right but carried MSA case/mood marks absent in EGY.
    • 5% were typos and 2% were speech effects such as letter lengthening.
  • Planned work: automatic paradigm completion, more orthographic variants, English/MSA glosses and a public release.

Interesting ideas

  • Compile a form list into a tabular analyzer. Converting an inflected-form lexicon into compatible prefix/stem/suffix tables gives broader coverage than the original list while keeping lookup simple. This suits any morphologically rich language where a word list exists but a rule-based grammar does not.
  • Accept many spellings, emit one standard. Recognizing many spelling variants but always outputting a single conventional orthography cuts sparsity for downstream tools without rejecting real-world text. This is relevant to any non-standardized or informally written variety.
  • Report a three-way evaluation. Scoring with three outcomes (correct analysis present, only wrong analyses, nothing returned) alongside lexicon size separates coverage gaps from overgeneration. A single accuracy number hides that distinction.
  • Merge dialect and standard analyzers. Combining a dialect analyzer with a standard-language analyzer recovers shared vocabulary cheaply and minimized empty outputs. However, standard-language features (here case/mood) leak in and need filtering.
  • Categorize residual errors, including gold errors. Classifying the remaining failures, and counting annotation mistakes in the gold data, shows which next step pays off most, here spelling extensions and paradigm completion.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{habash2012morphological,
  title = {{A Morphological Analyzer for Egyptian Arabic}},
  author = {Habash, Nizar and Eskander, Ramy and Hawwari, Abdelati},
  booktitle = {Proceedings of the Twelfth Meeting of the Special Interest Group on Computational Morphology and Phonology},
  year = {2012},
  url = {https://aclanthology.org/W12-2301/}
}

References

29 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ernest T. Abdel-Massih, Zaki N. Abdel-Malek, and El-Said M. Badawi. 1979. A Reference Grammar of Egyptian Arabic. Georgetown University Press. find
  2. Hitham Abo Bakr, Khaled Shaalan, and Ibrahim Ziedan. 2008. A Hybrid Approach for Converting Written Egyptian Colloquial Dialect into Diacritized Arabic. In The 6th International Conference on Informatics and Systems, INFOS2008. Cairo University. find
  3. Imad Al-Sughaiyer and Ibrahim Al-Kharashi. 2004. Arabic Morphological Analysis Techniques: A Comprehensive Survey. Journal of the American Society for Information Science and Technology, 55(3):189– 213. find
  4. Mohamed Altantawy, Nizar Habash, and Owen Rambow. 2011. Fast Yet Rich Morphological Analysis. In Proceedings of the 9th International Workshop on Finite-State Methods and Natural Language Processing (FSMNLP 2011), Blois, France. find
  5. Kenneth Beesley, Tim Buckwalter, and Stuart Newton. 1989. Two-Level Finite-State Analysis of Arabic Morphology. In Proceedings of the Seminar on Bilingual Computing in Arabic and English, page n.p. find
  6. Tim Buckwalter. 2004. Buckwalter arabic morphological analyzer version 2.0. LDC catalog number LDC2004L02, ISBN 1-58563-324-0. find
  7. Tim Buckwalter. 2007. Issues in Arabic Morphological Analysis. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. find
  8. Mark W. Cowell. 1964. A Reference Grammar of Syrian Arabic. Georgetown University Press. find
  9. Mona Diab, Kadri Hacioglu, and Daniel Jurafsky, 2007. Arabic Computational Morphology: Knowledge-based and Empirical Methods, chapter Automated Methods for Processing Arabic Text: From Tokenization to Base Phrase Chunking. Springer. find
  10. Wallace Erwin. 1963. A Short Reference Grammar of Iraqi Arabic. Georgetown University Press. find
  11. Hassan Gadalla, Hanaa Kilany, Howaida Arram, Ashraf Yacoub, Alaa El-Habashi, Amr Shalaby, Krisjanis Karins, Everett Rowson, Robert MacIntyre, Paul Kingsbury, David Graff, and Cynthia McLemore. 1997. CALLHOME Egyptian Arabic Transcripts. In Linguistic Data Consortium, Philadelphia. find
  12. Hassan Gadalla. 2000. Comparative Morphology of Standard and Egyptian Arabic. LINCOM EUROPA. find
  13. David Graff, Mohamed Maamouri, Basma Bouziri, Sondos Krouna, Seth Kulick, and Tim Buckwalter. 2009. Standard Arabic Morphological Analyzer (SAMA) Version 3.1. Linguistic Data Consortium LDC2009E73. find
  14. Nizar Habash and Owen Rambow. 2005. Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan. find our card
  15. Nizar Habash and Owen Rambow. 2006. MAGEAD: A Morphological Analyzer and Generator for the Arabic Dialects. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 681–688, Sydney, Australia. find our card
  16. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. find
  17. Nizar Habash, Mona Diab, and Owen Rabmow. 2012. Conventional Orthography for Dialectal Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Istanbul. find
  18. Nizar Habash. 2007. Arabic Morphological Representations for Machine Translation. In Antal van den Bosch and Abdelhadi Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Kluwer/Springer. find
  19. Nizar Habash. 2010. Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. find
  20. Clive Holes. 2004. Modern Arabic: Structures, Functions, and Varieties. Georgetown Classics in Arabic Language and Linguistics. Georgetown University Press. find
  21. H. Kilany, H. Gadalla, H. Arram, A. Yacoub, A. El-Habashi, and C. McLemore. 2002. Egyptian Colloquial Arabic Lexicon. LDC catalog number LDC99L22. find
  22. George Anton Kiraz. 2000. Multi-Tiered Nonlinear Morphology Using Multi-Tape Finite Automata: A Case Study on Syriac and Arabic. Computational Linguistics, 26(1):77–105. find
  23. Mohamed Maamouri, Ann Bies, Tim Buckwalter, Mona Diab, Nizar Habash, Owen Rambow, and Dalila Tabessi. 2006. Developing and Using a Pilot Dialectal Arabic Treebank. In Proceedings of the Language Resources and Evaluation Conference (LREC), Genoa, Italy. find
  24. Mohamed Maamouri, Ann Bies, Seth Kulick, Dalila Tabessi, and Sondos Krouna. 2012a. Egyptian Arabic Treebank Pilot. find
  25. Mohamed Maamouri, Sondos Krouna, Dalila Tabessi, Nadia Hamrouni, and Nizar Habash. 2012b. Egyptian Arabic Morphological Annotation Guidelines. find
  26. Emad Mohamed, Behrang Mohit, and Kemal Oflazer. 2012. Annotating and Learning Morphological Segmentation of Egyptian Colloquial Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Istanbul. find
  27. Jason Riesa and David Yarowsky. 2006. Minimally Supervised Morphological Segmentation with Applications to Machine Translation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA06), pages 185–192, Cambridge,MA. find
  28. Wael Salloum and Nizar Habash. 2011. Dialectal to Standard Arabic Paraphrasing to Improve Arabic-English Statistical Machine Translation. In Proceedings of the First Workshop on Algorithms and Resources for Modelling of Dialects and Language Varieties, pages 10–21, Edinburgh, Scotland. find
  29. Otakar Smrž. 2007. Functional Arabic Morphology. Formal System and Implementation. Ph.D. thesis, Charles University in Prague, Prague, Czech Republic. find