Elamur Research⌕
☰
home / papers / 2016 / Morphologically Annotated Corpora and Morphological Analyzers for Moroccan and Sanaani Yemeni Arabic

Morphologically Annotated Corpora and Morphological Analyzers for Moroccan and Sanaani Yemeni Arabic

Paper · 2016 · Faisal Al-Shargi, Aidan Kaplan, Ramy Eskander et al.

Summary

Arabic dialects are now widely written online, but NLP tools mostly target Modern Standard Arabic (MSA) and Classical Arabic. The authors build the first annotated resources for two dialects with almost no prior computational support: Moroccan Arabic (MOR) and the Sanaani variety of Yemeni Arabic (YEMS). For each dialect they collect a corpus from many genres, including web forums, Facebook humor, folktales, oral interview transcripts, textbooks, radio programs and poems. They annotate the corpora morphologically with the DIWAN tool, using a dialect-specific conventional orthography (CODA), and then derive morphological analyzers from the annotations. The paper also describes the distinctive phonology, morphology, syntax and lexicon of each dialect. On YEMS, a corpus-derived lookup or analyzer used alone leaves many tokens uncovered. Combining it with existing MSA and Egyptian analyzers covers nearly all tokens with the correct full analysis. These resources open both dialects to downstream tools such as taggers and sentiment mining, and the authors plan to release them publicly.

Key points

  • Corpus sizes: the MOR corpus has about 64K tokens (64,170) and the YEMS corpus about 32.5K. Both cover many genres; the largest MOR source is Facebook humor and the largest YEMS source is the Heidelberg Semitisches Tonarchiv interviews.

  • Orthography: internet text appears in spontaneous Arabic-script spelling, while romanized textbook sentences were converted into Arabic script following a CODA defined for each dialect.

  • MSA handling: tokens with purely MSA morphology are tagged as MSA so they can be kept out of the dialect analyzer. Words that look like MSA but inflect dialectally are counted as dialect.

  • YEMS results on TEST (full-analysis recall):

    Analyzer setupAllOOV
    Lookup YEMS alone67.6%43.3%
    ALMOR YEMS69.3%—
    SAMA MSA77.8%—
    Union of the YEMS lookup, the YEMS ALMOR analyzer, extended SAMA and the Egyptian ALMOR analyzer95.1%4.6%

    The combined system produces about 25 analyses per word.

  • Evaluation columns: recall is reported separately for POS, a reduced POS5 tagset, lemma, CODA form, stem and the full analysis, on both DEV and TEST.

Interesting ideas

  • I1. Instead of picking one analyzer, take the union of a small in-domain lookup, an analyzer extracted from that lookup, and broad analyzers for related varieties; coverage climbs sharply while each part stays simple.
  • I2. Report recall feature by feature (POS, coarse POS, lemma, normalized form, stem, full analysis) together with the OOV rate and the number of analyses per word, which shows where coverage is lost and how much ambiguity a fix adds.
  • I3. Explicitly mark tokens belonging to another language variety during annotation so they are excluded from lexicon extraction instead of contaminating it.
  • I4. Define a conventional spelling target for noisy, non-standard writing and annotate every token with its normalized form next to the raw one.
  • I5. Collect evaluation data deliberately across many genres, from oral transcripts and folktales to forum comments and humor, so the analyzer is not tuned to a single register.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The en lemmatizer could merge a treebank-derived lexicon, a rule-based analyzer and a broad lexicon as separate candidate sources, recording which source supplied each analysis to keep it explainable.
  • I2 (learn): Mova's morphology evaluation could track OOV rate and ambiguity alongside per-feature recall, so that a coverage gain that floods the tagger with candidates is caught.
  • I3 (tokens): Foreign words, code-switches or archaic forms in English training text could be flagged as such, so the induced lexicon stays clean while the parser still handles them.
  • I4 (tokens): Storing a normalized form for misspelled or informal English tokens (web text, dialogue in tales) would let later stages work on clean lemmas while keeping the original surface form for generation.
  • I5 (answer): Test sets for the fable and tale readers could mix narrative, dialogue and informal registers to expose genre-specific gaps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States241 works · 54,053 citations🎓Yale UniversityNew Haven · United States95 works · 12,765 citations🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations🎓Leipzig UniversityLeipzig · Germany26 works · 8,622 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksMorphology · 405 works

Cite

Cite as Faisal Al-Shargi, Aidan Kaplan, Ramy Eskander, Nizar Habash and Owen Rambow. 2016. Morphologically Annotated Corpora and Morphological Analyzers for Moroccan and Sanaani Yemeni Arabic. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alshargi2016morphologically,
  title = {{Morphologically Annotated Corpora and Morphological Analyzers for Moroccan and Sanaani Yemeni Arabic}},
  author = {Faisal Al-Shargi, Aidan Kaplan, Ramy Eskander, Nizar Habash and Owen Rambow},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/2hchewb29iga},
  url = {https://doi.org/10.63317/2hchewb29iga}
}

References

34 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdullah, A.-H. (1991). Hawliat yamaniah: alyemen fi alqrn altasi’ ashar almiladi. dar Al-Hikma, Sana’a, Yemen. find
  2. Al-Iryani, M. A. (1996). al-Mu’jam al-yemeni fi algah walturath. dar alfkr, Damascus, Syria. find
  3. al Salam Al-Amri, A. (2000). Texts in Sanani Arabic. O. Harrassowitz, Wiesbaden, Germany. find
  4. Al-Shargi, F. and Rambow, O. (2015). Diwan: A dialectal word annotation tool for Arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 49–58, Beijing, China, July. Association for Computational Linguistics.
  5. Behnstedt, P. (1985). Die nordjemenitischen Dialekte. L. Reichert, Wiesbaden, Germany. find
  6. Behnstedt, P. (2006). Die nordjemenitischen Dialekte. II/3. Glossar fa - Ya. L. Reichert, Wiesbaden, Germany. find
  7. Bouamor, H., Habash, N., and Oflazer, K. (2014). A multi-dialectal parallel corpus of arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). European Language Resources Association (ELRA). find
  8. Boujelbane, R., Ellouze Khemekhem, M., and Belguith, L. H. (2013). Mapping Rules for Building a Tunisian Dialect Lexicon and Generating Corpora. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 419–428, Nagoya, Japan. find
  9. Brustad, K. (2000). The Syntax of Spoken Arabic: A Comparative Study of Moroccan, Egyptian, Syrian, and Kuwaiti Dialects. Georgetown University Press. find
  10. Chekayri, A. (2011). An Introduction to Moroccan Arabic and Culture. Repertorio Español de Bibliografía Árabe e Islámica: 2010. Georgetown University Press. find
  11. Diab, M. T., Al-Badrashiny, M., Aminian, M., Attia, M., Elfardy, H., Habash, N., Hawwari, A., Salloum, W., Dasigi, P., and Eskander, R. (2014). Tharwa: A large scale dialectal arabic-standard arabic-english lexicon. In LREC, pages 3782–3789.
  12. Eskander, R., Habash, N., and Rambow, O. (2013). Automatic Extraction of Morphological Lexicons from Morphologically Annotated Corpora. In Proceedings of tenth Conference on Empirical Methods in Natural Language Processing. find
  13. Graff, D., Maamouri, M., Bouziri, B., Krouna, S., Kulick, S., and Buckwalter, T. (2009). Standard Arabic Morphological Analyzer (SAMA) Version 3.1. Linguistic Data Consortium LDC2009E73. find
  14. Habash, N., Soudi, A., and Buckwalter, T. (2007). On Arabic Transliteration. In A. van den Bosch et al., editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. ↗
  15. Habash, N., Diab, M., and Rambow, O. (2012a). Conventional orthography for dialectal Arabic. In Proceedings of the Eighth Language Resources and Evaluation Conference, pages 711–718, Istanbul, Turkey, May. European Language Resources Association.
  16. Habash, N., Eskander, R., and Hawwari, A. (2012b). A Morphological Analyzer for Egyptian Arabic. In Proceedings of the Twelfth Meeting of the Special Interest Group on Computational Morphology and Phonology, pages 1–9, Montréal, Canada.
  17. Habash, N., Roth, R., Rambow, O., Eskander, R., and Tomeh, N. (2013). Morphological Analysis and Disambiguation for Dialectal Arabic. In Proceedings of the 2013 Conference of NAACL-HLT, Atlanta, GA.
  18. Habash, N. (2007). Arabic Morphological Representations for Machine Translation. In Antal van den Bosch et al., editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Kluwer/Springer. ↗
  19. Harrell, R., Sobelman, H., and Fox, T. (2004). A Dictionary of Moroccan Arabic: Moroccan-English. George-Analyzer Token Recall System POS POS5 Lemma CODA Stem All OOV Analyses W ord SAMA M SA 86.5 96.1 81.5 94.1 89.4 75.1 14.9 10.7 ALMOR EGY 81.3 91.9 69.6 93.5 86.3 60.5 19.3 4.4 SAMA ext +ALMOR EGY 91.1 97.4 86.7 94.6 93.5 80.5 8.7 19.3 Lookup YEMS 72.5 83.6 71.2 93.2 77.9 68.6 41.9 0.9 ALMOR YEMS 77.9 88.5 75.3 94.0 85.4 69.3 27.6 2.6 Lookup YEMS +ALMOR YEMS +SAMA ext +ALMOR EGY 97.1 99.2 95.7 96.6 97.6 92.4 5.4 22.8 Table 7: YEMS morphological analysis recall on DEV. The columns are described in Section 6. Analyzer Token Recall System POS POS5 Lemma CODA Stem All OOV Analyses W ord SAMA M SA 89.8 96.8 83.7 95.5 89.6 77.8 12.7 11.7 ALMOR EGY 83.0 92.5 69.1 94.5 86.7 60.6 17.5 4.8 SAMA ext +ALMOR EGY 93.9 97.9 89.3 95.8 93.5 83.0 7.3 21.0 Lookup YEMS 71.4 82.6 69.3 93.3 75.7 67.6 43.3 1.0 ALMOR YEMS 77.8 87.6 75.1 94.6 84.4 69.3 27.5 3.4 Lookup YEMS +ALMOR YEMS +SAMA ext +ALMOR EGY 98.3 99.3 97.4 97.6 97.9 95.1 4.6 25.4 Table 8: YEMS morphological analysis recall on TEST. The columns are described in Section 6. town classics in Arabic language and linguistics. Georgetown University Press. find
  20. Harrell, R. (1962). A Short Reference Grammar of Moroccan Arabic: With Audio CD. Georgetown classics in Arabic language and linguistics. Georgetown University Press. find
  21. Jarrar, M., Habash, N., Akra, D., and Zalmout, N. (2014). Building a corpus for palestinian arabic: a preliminary study. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 18– 27, Doha, Qatar, October. Association for Computational Linguistics.
  22. Jastrow, O. (1984). Zur Phonologie und Phonetik des Sana’nischen. In Entwicklungsprozesse in der Arabischen Republik Jemen, page 289âĂŞ304. L.Reichert, Wiesbaden, Germany. find
  23. Khalifa, S., Habash, N., Abdulrahim, D., and Hassan, S. (2016). A Large Scale Corpus of Gulf Arabic. In Proceedings of the International Conference on Language Resources and Evaluation (LREC), Portorož, Slovenia.
  24. Kilany, H., Gadalla, H., Arram, H., Yacoub, A., El-Habashi, A., and McLemore, C. (2002). Egyptian Colloquial Arabic Lexicon. LDC catalog number LDC99L22. find
  25. Maamouri, M., Bies, A., Buckwalter, T., Diab, M., Habash, N., Rambow, O., and Tabessi, D. (2006). Developing and using a pilot dialectal arabic treebank. In LREC, Genoa, Italy.
  26. Maamouri, M., Bies, A., Kulick, S., Ciul, M., Habash, N., and Eskander, R. (2014). Developing an egyptian arabic treebank: Impact of dialectal morphology on annotation and tool development. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). European Language Resources Association (ELRA). find
  27. Naïm-Sanbar, S. (1994). Contribution à lâĂŹétude de lâĂŹaccent yéménite: Le parler des femmes de lâĂŹancienne génération. Zeitschrift für Arabische Linguistik, page 27.67âĂŞ89. find
  28. Pasha, A., Al-Badrashiny, M., Diab, M., El Kholy, A., Eskander, R., Habash, N., Pooleery, M., Rambow, O., and Roth, R. M. (2014). MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland.
  29. Salloum, W. and Habash, N. (2014). ADAM: Analyzer for Dialectal Arabic Morphology. Journal of King Saud University-Computer and Information Sciences, 26(4):372–378. ↗
  30. Smaïli, K., Abbas, M., Meftouh, K., and Harrat, S. (2014). Building resources for Algerian Arabic dialects. In 15th Annual Conference of the International Communication Association Interspeech. find
  31. The Peace Corps. (2011). Moroccan Arabic. Based on Moroccan Arabic by Abdelghani Lamnaouar (1994); Abderrahmane Boujnab, editor. find
  32. Tratz, S., Briesch, D., Laoudi, J., Voss, C., and Holland, V. M. (2014). Language and dialect identification in social media analysis. Proc. SPIE, 9122:91220K– 91220K–11. find
  33. Watson, J. C. (1993). A syntax of San’ani Arabic. O. Harrassowitz, Wiesbaden, Germany. find
  34. Zribi, I., Boujelbane, R., Masmoudi, A., Ellouze, M., Belguith, L., and Habash, N. (2014). A Conventional Orthography for Tunisian Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shargi-2016-morphologically-annotated-corpora-morphological-analyzers/