Elamur Research⌕
☰
home / papers / 2015 / Natural Language Processing for Dialectical Arabic: A Survey

Natural Language Processing for Dialectical Arabic: A Survey

Paper · 2015 · Abdulhadi Shoufan and Sumaya Alameri

Summary

This survey asks what computational work existed on spoken Arabic varieties, as opposed to Modern Standard Arabic (MSA), as of 2015, and where the gaps were. The authors collected as many papers as they could find. They sorted them into four areas: basic analyses (morphology, POS tagging, syntax, orthography), building resources (lexicons, wordnets, corpora, treebanks), semantic-level analysis and synthesis (for example machine translation), and dialect identification. Each paper was placed in a matrix (Table 1) that links research topics to the dialects covered, so readers can quickly find prior work for a given task and variety.

The overview shows that the field was growing but still small compared with the number of speakers. Egyptian and Levantine received the most attention. Corpus building, dialect identification and speech recognition dominated, while syntactic parsing was almost absent.

The authors call for:

  • a hierarchical way to specify dialect granularity,
  • public release of annotated resources,
  • more parsing research.

The survey works mainly as a reference map for low-resource varieties.

Key points

  • Papers are organised in a topic-by-dialect matrix. Egyptian is the most studied dialect, Levantine is unusually well covered, and Sudanese, Moroccan and Algerian are barely touched despite large populations. The authors note that Arabic dialects have over 390 million speakers.
  • Syntax is the weakest area. Examples include:
    • Chiang et al. (2006), a Levantine parser that works through MSA;
    • the pilot Levantine Arabic Treebank (Maamouri et al., 2006), about 26,000 words of annotated telephone conversation;
    • a later Egyptian treebank (Maamouri et al., 2014).
  • Morphology tools named in the survey:
    • MAGEAD, an analyzer and generator for Levantine with MSA-to-Levantine conversion;
    • an Egyptian analyzer;
    • an MSA tagger retargeted to Egyptian (Habash et al., 2013), covering tagging, diacritization, lemmatization and tokenization.
  • Spelling has no standard in the dialects. CODA was proposed as a conventional orthography, with a Tunisian variant, and several works handle Arabizi (Arabic written in Latin letters).
  • Resources mentioned include:
    • crowdsourced parallel corpora via Mechanical Turk: Levantine-English (1.1M words) and Egyptian-English (380k words);
    • a 52M-word online commentary dataset annotated for dialect level;
    • a 43K-word Palestinian corpus.

Interesting ideas

  • I1. Chiang et al. parse a dialect without any dialect treebank: they map each word to standard-language candidates, parse with the standard parser, then put the original words back on the leaves. This reuses a strong parser for a weak variety.
  • I2. A conventional orthography like CODA fixes one canonical spelling per word before analysis, so downstream tools see far fewer variant forms.
  • I3. Laying out work as a matrix of tasks against language varieties makes missing combinations visible at a glance.
  • I4. The authors argue that a dialect label like "Levantine" is too coarse and propose a hierarchy of varieties so that work can be pinned to the right level.
  • I5. Several dialect lexicons were bootstrapped by extending a standard-language lexicon or by linking each dialect entry to its standard synonym. This yields usable resources cheaply.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): informal or nonstandard English (chat, transcribed speech) could be normalized, parsed by gparse or the perceptron parser, and the UD tree relabeled with the original tokens, keeping the mapping as an explainable trace.
  • I2 (tokens): a documented canonical-spelling layer in crate en would map variant spellings to one form for lemma lookup, while storing the surface form for back-mapping.
  • I3 (learn): a coverage grid of pipeline stages against text genres (tests, word problems, fables, tales), filled with evaluation scores, would show where silver data or repair rules are most needed.
  • I4 (tree): treebanks and UD dialect conversions could carry a hierarchical variety tag (e.g., English > American > a specific register), so conversion rules and evaluation can be scoped precisely.
  • I5 (world): regional or slang words could be linked to their standard WordNet synsets, so the knowledge graph and verb classes apply to them without new ontology entries.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Khalifa University of Science and TechnologyAbu Dhabi · United Arab Emirates3 works · 196 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksMorphology · 405 works

Cite

Cite as Abdulhadi Shoufan and Sumaya Alameri. 2015. Natural Language Processing for Dialectical Arabic: A Survey. Proceedings of the Second Workshop on Arabic Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w15-3205

@inproceedings{shoufan2015natural,
  title = {{Natural Language Processing for Dialectical Arabic: A Survey}},
  author = {Abdulhadi Shoufan and Sumaya Alameri},
  booktitle = {Proceedings of the Second Workshop on Arabic Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/w15-3205},
  url = {https://doi.org/10.18653/v1/w15-3205}
}

References

91 references, 44 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Mageed, M., Diab, M., & Kübler, S. (2014, January). Samar: Subjectivity and sentiment analysis for arabic social media. Comput. Speech Lang., 28(1), 20–37. Retrieved from http://dx.doi.org/10 .1016/j.csl.2013.03.001 doi: 10 .1016/j.csl.2013.03.001 ↗
  2. Abuata, B., & Al-Omari, A. (2015). A rule-based stemmer for arabic gulf dialect. Journal of King Saud University-Computer and Information Sciences. ↗
  3. Akbacak, M., Vergyri, D., Stolcke, A., Scheffer, N., & Mandal, A. (2011). Effective arabic dialect classification using diverse phonotactic models. In Interspeech (Vol. 11, pp. 737–740). find
  4. Al-Gaphari, G., & Al-Yadoumi, M. (2012). A method to convert sanaani accent to modern standard arabic. International Journal of Information Science and Management (IJISM), 8(1), 39–49. find
  5. Alghamdi, M., Alhargan, F., Alkanhal, M., Alkhairy, A., Eldesouki, M., & Alenazi, A. (2008). Saudi accented arabic voice bank. Journal of King Saud University-Computer and Information Sciences, 20, 45–64. ↗
  6. Ali, A. (2014). Advances in dialectal arabic speech recognition: A study using twitter to improve egyptian asr. In Proceedings of the international workshop on spoken language translation (iwslt).
  7. Al-Mannai, K., Sajjad, H., Khader, A., Al Obaidli, F., Nakov, P., & Vogel, S. (2014). Unsupervised word segmentation improves dialectal arabic to english machine translation. ANLP 2014, 207. find
  8. Almeman, K., & Lee, M. (2012). Towards developing a multi-dialect morphological analyzer for arabic. In 4th international conference on arabic language processing, rabat, morocco. find
  9. Almeman, K., & Lee, M. (2013). Automatic building of arabic multi dialect text corpora by bootstrapping dialect words. In Communications, signal processing, and their applications (iccspa), 2013 1st international conference on (pp. 1–6). find
  10. Al-Sabbagh, R., & Girju, R. (2010). Mining the web for the induction of a dialectical arabic lexicon. In Lrec.
  11. Al-Sabbagh, R., & Girju, R. (2012a). A supervised pos tagger for written arabic social networking corpora. In Proceedings of konvens (pp. 39–52). find
  12. Al-Sabbagh, R., & Girju, R. (2012b). Yadac: Yet another dialectal arabic corpus. In Lrec (pp. 2882–2889).
  13. Bakr, H. A., Shaalan, K., & Ziedan, I. (2008). A hybrid approach for converting written egyptian colloquial dialect into diacritized arabic. In The 6th international conference on informatics and systems, infos2008. cairo university. Belgacem, M., Antoniadis, G., & Besacier, L. find
  14. (2010). Automatic identification of arabic dialects. In Lrec.
  15. Benajiba, Y., & Diab, M. (2010). A web application for dialectal arabic text annotation. In Proceedings of the lrec workshop for language resources (lrs) and human language technologies (hlt) for semitic languages: Status, updates, and prospects. find
  16. Biadsy, F., Hirschberg, J., & Habash, N. (2009). Spoken arabic dialect identification using phonotactic modeling. In Proceedings of the eacl 2009 workshop on computational approaches to semitic languages (pp. 53–61).
  17. Bies, A., Song, Z., Maamouri, M., Grimes, S., Lee, H., Wright, J., . . . Rambow, O. (2014). Transliteration of arabizi into arabic orthography: Developing a parallel annotated arabizi-arabic script sms/chat corpus. ANLP 2014, 93.
  18. Bouchlaghem, R., & Elkhlifi, A. (2014). Tunisian dialect wordnet creation and enrichment using web resources and other wordnets. ANLP 2014, 104. find
  19. Boujelbane, R., BenAyed, S., & Belguith, L. H. (2013). Building bilingual lexicon to create dialect tunisian corpora and adapt language model. ACL 2013, 88. find
  20. Boujelbane, R., Mallek, M., Ellouze, M., & Belguith, L. H. (2014). Fine-grained pos tagging of spoken tunisian dialect corpora. In Natural language processing and information systems (pp. 59–62). Springer. find
  21. Brustad, K. (2000). The syntax of spoken arabic: A comparative study of moroccan, egyptian, syrian, and kuwaiti dialects. Georgetown University Press. find
  22. Buckwalter, T. (2004). Buckwalter arabic morphological analyzer version 2.0. ldc catalog number ldc2004l02 (Tech. Rep.). ISBN 1-58563-3-0. find
  23. Cavalli-Sforza, V., Saddiki, H., Bouzoubaa, K., Abouenour, L., Maamouri, M., & Goshey, E. (2013). Bootstrapping a wordnet for an arabic dialect from other wordnets and dictionary resources. In Computer systems and applications (aiccsa), 2013 acs international conference on (pp. 1–8). find
  24. Chiang, D., Diab, M. T., Habash, N., Rambow, O., & Shareef, S. (2006). Parsing arabic dialects. In Eacl. ↗
  25. Condon, S., Parvaz, D., Aberdeen, J. S., Doran, C., Freeman, A., & Awad, M. (2010). Evaluation of machine translation errors in english and iraqi arabic. In Lrec.
  26. Cotterell, R., & Callison-Burch, C. (2014). A multi-dialect, multi-genre corpus of informal written arabic. In Proceedings of the language resources and evaluation conference (lrec).
  27. Darwish, K. (2013). Arabizi detection and conversion to arabic. arXiv preprint arXiv:1306.6755. ↗
  28. Darwish, K., & Gao, W. (2014). Simple effective microblog named entity recognition: Arabic as an example. Proc. of LREC, Reykjavik, Iceland.
  29. Darwish, K., & Magdy, W. (2014). Arabic information retrieval. Now Publishers. ↗
  30. Darwish, K., Sajjad, H., & Mubarak, H. (2014). Verifiably effective arabic dialect identification. EMNLP-2014.
  31. Dasigi, P., & Diab, M. T. (2011). Codact: Towards identifying orthographic variants in dialectal arabic. In Ijcnlp (pp. 318–326). ↗
  32. Diab, M., Habash, N., Rambow, O., Altantawy, M., & Benajiba, Y. (2010). Colaba: Arabic dialect annotation and processing. In Lrec workshop on semitic language processing (pp. 66–74). find
  33. Duh, K., & Kirchhoff, K. (2005). Pos tagging of dialectal arabic: a minimally supervised approach. In Proceedings of the acl workshop on computational approaches to semitic languages (pp. 55–62).
  34. Duh, K., & Kirchhoff, K. (2006). Lexicon acquisition for dialectal arabic using transductive learning. In Proceedings of the 2006 conference on empirical methods in natural language processing (pp. 399–407).
  35. Duwairi, R., Marji, R., Sha’ban, N., & Rushaidat, S. (2014). Sentiment analysis in arabic tweets. In Information and communication systems (icics), 2014 5th international conference on (pp. 1–6). find
  36. El-Beltagy, S. R., & Ali, A. (2013). Open issues in the sentiment analysis of arabic social media: A case study. In Innovations in information technology (iit), 2013 9th international conference on (pp. 215–220). find
  37. Elfardy, H., & Diab, M. (2012a). Aida: Automatic identification and glossing of dialectal arabic. In Proceedings of the 16th eamt conference (project papers) (pp. 83–83).
  38. Elfardy, H., & Diab, M. T. (2012b). Simplified guidelines for the creation of large scale dialectal arabic annotations. In Lrec (pp. 371– 378).
  39. Elfardy, H., & Diab, M. T. (2013). Sentence level dialect identification in arabic. In Acl (2) (pp. 456–461).
  40. El-Fishawy, N., Hamouda, A., Attiya, G. M., & Atef, M. (2014). Arabic summarization in twitter social network. Ain Shams Engineering Journal, 5(2), 411–420. ↗
  41. Graff, D., Buckwalter, T., Jin, H., & Maamouri, M. (2006). Lexicon development for varieties of spoken colloquial arabic. In Proceedings of the fifth international conference on language resources and evaluation (lrec) (pp. 999–1004).
  42. Graff, D., & Maamouri, M. (2012). Developing lmf-xml bilingual dictionaries for colloquial arabic dialects. In Lrec (pp. 269–274).
  43. Graja, M., Jaoua, M., & Hadrich Belguith, L. (2010). Lexical study of a spoken dialogue corpus in tunisian dialect. In The international arab conference on information technology (acit), benghazi–libya. find
  44. Habash, N., Diab, M. T., & Rambow, O. (2012). Conventional orthography for dialectal arabic. In Lrec (pp. 711–718).
  45. Habash, N., Eskander, R., & Hawwari, A. (2012). A morphological analyzer for egyptian arabic. In Proceedings of the twelfth meeting of the special interest group on computational morphology and phonology (pp. 1–9).
  46. Habash, N., & Rambow, O. (2006). Magead: a morphological analyzer and generator for the arabic dialects. In Proceedings of the 21st international conference on computational linguistics and the 44th annual meeting of the association for computational linguistics. ↗
  47. Habash, N., & Rambow, O. (2007). Morphophonemic and orthographic rules in a multi-dialectal morphological analyzer and generator for arabic verbs. In International symposium on computer and arabic language (iscal), riyadh, saudi arabia. find
  48. Habash, N., Rambow, O., Diab, M., & Kanjawi-Faraj, R. (2008a). Guidelines for annotation of arabic dialectness. In Proceedings of the lrec workshop on hlt & nlp within the arabic world (pp. 49–53). find
  49. Habash, N., Rambow, O., Diab, M., & Kanjawi-Faraj, R. (2008b). Guidelines for annotation of arabic dialectness. In Proceedings of the lrec workshop on hlt & nlp within the arabic world (pp. 49–53). find
  50. Habash, N., Roth, R., Rambow, O., Eskander, R., & Tomeh, N. (2013). Morphological analysis and disambiguation for dialectal arabic. In Proceedings of naacl-hlt (pp. 426–432).
  51. Harrat, S., Meftouh, K., Abbas, M., Jamoussi, S., Saad, M., & Smaili, K. (2015). Cross-dialectal arabic processing. In Computational linguistics and intelligent text processing (pp. 620–632). Springer. find
  52. Harrat, S., Meftouh, K., Abbas, M., & Smaili, K. (2014). Building resources for algerian arabic dialects. Corpus (sentences), 4000(6415), 2415. find
  53. Hawwari, A., Attia, M., & Diab, M. (2014). A framework for the classification and annotation of multiword expressions in dialectal arabic. ANLP 2014, 48.
  54. Hedar, A. R., & Doss, M. (2013). Mining social networks arabic slang comments. IEEE Symposium on Computational Intelligence and Data Mining (CIDM). find
  55. Ibrahim, H. S., Abdou, S. M., & Gheith, M. (2015). Sentiment analysis for modern standard arabic and colloquial. arXiv preprint arXiv:1505.03105. ↗
  56. Iskra, D. J., Siemund, R., Borno, J., Moreno, A., Emam, O., Choukri, K., . . . others (2004). Orientel-telephony databases across northern africa and the middle east. In Lrec. find
  57. Jarrar, M., Habash, N., Akra, D., & Zalmout, N. (2014). Building a corpus for palestinian arabic: a preliminary study. ANLP 2014, 18.
  58. Jeblee, S., Feely, W., Bouamor, H., Lavie, A., Habash, N., & Oflazer, K. (2014). Domain and dialect adaptation for machine translation into egyptian arabic. ANLP 2014, 196.
  59. Jehl, L., Hieber, F., & Riezler, S. (2012). Twitter translation using translation-based cross-lingual retrieval. In Proceedings of the seventh workshop on statistical machine translation (pp. 410–421). Stroudsburg, PA, USA: Association for Computational Linguistics. Retrieved from http://dl.acm.org/citation .cfm?id=2393015.2393074 ↗
  60. Kirchhoff, K., & Vergyri, D. (2005). Cross-dialectal data sharing for acoustic modeling in arabic speech recognition. Speech Communication, 46(1), 37–51. ↗
  61. Lei, Y., & Hansen, J. H. (2009). Factor analysis-based information integration for arabic dialect identification. In Acoustics, speech and signal processing, 2009. icassp 2009. ieee international conference on (pp. 4337– 4340). find
  62. Maamouri, M., Bies, A., Buckwalter, T., Diab, M., Habash, N., Rambow, O., & Tabessi, D. (2006). Developing and using a pilot dialectal arabic treebank. In Proceedings of the fifth international conference on language resources and evaluation, lrec06.
  63. Maamouri, M., Bies, A., Kulick, S., Ciul, M., Habash, N., & Eskander, R. (2014). Developing an egyptian arabic treebank: Impact of dialectal morphology on annotation and tool development. Proc. of LREC, Reykjavik, Iceland. find
  64. Masmoudi, A., Habash, N., Ellouze, M., Estève, Y., & Belguith, L. H. (2015). Arabic transliteration of romanized tunisian dialect text: A preliminary investigation. In Computational linguistics and intelligent text processing (pp. 608–619). Springer. find
  65. Mohamed, E., Mohit, B., & Oflazer, K. (2012). Transforming standard arabic to colloquial arabic. In Proceedings of the 50th annual meeting of the association for computational linguistics: Short papers-volume 2 (pp. 176–180).
  66. Mourad, A., & Darwish, K. (2013). Subjectivity and sentiment analysis of modern standard arabic and arabic microblogs. In Proceedings of the 4th workshop on computational approaches to subjectivity, sentiment and social media analysis (pp. 55–64).
  67. Mubarak, H., & Darwish, K. (2014). Using twitter to collect a multi-dialectal corpus of arabic. ANLP 2014, 1.
  68. Pasha, A., Al-Badrashiny, M., Altantawy, M., Habash, N., Pooleery, M., Rambow, O., . . . Diab, M. (2013). Dira: Dialectal arabic information retrieval assistant. In The companion volume of the proceedings of international joint conference on natural language processing (ijcnlp) (pp. 13–16).
  69. Rytting, C. A., Zajic, D. M., Rodrigues, P., Wayland, S. C., Hettick, C., Buckwalter, T., & Blake, C. C. (2011). Spelling correction for dialectal arabic dictionary lookup. ACM Transactions on Asian Language Information Processing (TALIP), 10(1), 3. ↗
  70. Sadat, F., Kazemi, F., & Farzindar, A. (2014). Automatic identification of arabic language varieties and dialects in social media. SocialNLP 2014, 22.
  71. Sadat, F., Mallek, F., Sellami, R., Boudabous, M. M., & Farzindar, A. (2014). Collaboratively constructed linguistic resources for language vari-ants and their exploitation in nlp applications–the case of tunisian arabic and the social media. In Workshop on lexical and grammatical resources for language processing (p. 102). find
  72. Salloum, W., & Habash, N. (2011). Dialectal to standard arabic paraphrasing to improve arabic-english statistical machine translation. In Proceedings of the first workshop on algorithms and resources for modelling of dialects and language varieties (pp. 10– 21). ↗
  73. Salloum, W., & Habash, N. (2012). Elissa: A dialectal to standard arabic machine translation system. In Coling (demos) (pp. 385– 392).
  74. Salloum, W., & Habash, N. (2014). Adam: Analyzer for dialectal arabic morphology. Journal of King Saud University-Computer and Information Sciences, 26(4), 372–378. ↗
  75. Sansò, A. (2004). Med-typ: A typological database for mediterranean languages. In Lrec. find
  76. Sawaf, H. (2010). Arabic dialect handling in hybrid machine translation. In Proceedings of the conference of the association for machine translation in the americas (amta), denver, colorado.
  77. Soltau, H., Mangu, L., & Biadsy, F. (2011). From modern standard arabic to levantine asr: Leveraging gale for dialects. In Automatic speech recognition and understanding (asru), 2011 ieee workshop on (pp. 266– 271). find
  78. Tachicart, R., & Bouzoubaa, K. (2014). A hybrid approach to translate moroccan arabic dialect. In Intelligent systems: Theories and applications (sita-14), 2014 9th international conference on (pp. 1–5). find
  79. Tratz, S., Briesch, D., Laoudi, J., & Voss, C. (2013). Tweet conversation annotation tool with a focus on an arabic dialect, moroccan darija. LAW VII & ID, 135.
  80. Zaghouani, W. (2014). Critical survey of the freely available arabic corpora. In Proceedings of the workshop on free/open-source arabic corpora and corpora processing tools workshop programme (p. 1). find
  81. Zaghouani, W., Habash, N., & Mohit, B. (2014). The qatar arabic language bank guidelines. find
  82. Zaidan, O., & Callison-Burch, C. (2011). The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content. In Acl (short papers) (pp. 37–41).
  83. Zaidan, O., & Callison-Burch, C. (2012). Arabic dialect identification. Computational Linguistics (submitted). ↗
  84. Zaidan, O., & Callison-Burch, C. (2014). Arabic dialect identification. Computational Linguistics, 40(1), 171–202. ↗
  85. Zbib, R., Malchiodi, E., Devlin, J., Stallard, D., Matsoukas, S., Schwartz, R., . . . Callison-Burch, C. (2012). Machine translation of arabic dialects. In Proceedings of the 2012 conference of the north american chapter of the association for computational linguistics: Human language technologies (pp. 49–59).
  86. Zhang, Q., Boril, H., & Hansen, J. H. (2013). Supervector pre-processing for prsvm-based chinese and arabic dialect identification. find
  87. Zirikly, A., & Diab, M. (2014). Named entity recognition for dialectal arabic. ANLP 2014, 78. find
  88. Zirikly, A., & Diab, M. (2015). Named entity recognition for arabic social media. In Proceedings of naacl-hlt (pp. 176–185).
  89. Zribi, I., Boujelbane, R., Masmoudi, A., Ellouze, M., Belguith, L., & Habash, N. (2014). A conventional orthography for tunisian arabic. In Proceedings of the language resources and evaluation conference (lrec), reykjavik, iceland.
  90. Zribi, I., Graja, M., Khmekhem, M. E., Jaoua, M., & Belguith, L. H. (2013). Orthographic transcription for spoken tunisian arabic. In Computational linguistics and intelligent text processing (pp. 153–163). Springer. find
  91. Zribi, I., Khemakhem, M. E., & Belguith, L. H. (2013). Morphological analysis of tunisian dialect. In International joint conference on natural language processing (pp. 992– 996). ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shoufan-2015-natural-language-processing-dialectical-arabic/