Elamur Research⌕
☰
home / papers / 2016 / A Large Scale Corpus of Gulf Arabic

A Large Scale Corpus of Gulf Arabic

Paper · 2016 · Salam Khalifa, Nizar Habash, Dana Abdulrahim et al.

Summary

Arabic NLP tools mostly target Modern Standard Arabic (MSA). Among the dialects, Egyptian has gained a fair set of resources, but Gulf Arabic (GA) has been left with very little. GA has no standard spelling and differs from MSA in sound and word structure. The authors address this gap with the Gumar Corpus. It holds roughly 110 million words drawn from about 1,200 long conversational novels, which forum volunteers had compiled into MS Word files. Each document is labelled with its sub-dialect, covering the six Gulf Cooperation Council countries. The authors also report a pilot study on morphological annotation of GA. For it, they extend the CODA conventional-orthography guidelines to Gulf varieties and use MADAMIRA-EGY output as the starting point that annotators correct. The corpus can be browsed through a public web interface. Its main value is size and dialect coverage: it gives a basis for later GA analyzers, taggers, spell-checkers and MT, and it is built from written text rather than transcribed speech.

Key points

  • The Gumar Corpus contains 112,410,688 words, 9,335,224 sentences and 1,236 documents, taken from Gulf forum novels.
  • Earlier GA resources were small and focused on Emirati speech: EAC (about 2M transcribed words, around 15K annotated) and EMALAC (78K words of child language).
  • "Gulf Arabic" is used broadly here, meaning the indigenous varieties of Bahrain, Kuwait, Oman, UAE, Qatar and Saudi Arabia. Sub-dialect labels are assigned per document.
  • CODA, previously extended to Egyptian, Palestinian, Tunisian, Algerian and Maghrebi Arabic, is now extended to GA.
  • Morphological pre-annotation comes from MADAMIRA-EGY, following the Palestinian Curras approach of Jarrar et al.

Interesting ideas

  • I1. Pre-annotate an under-resourced variety with a tagger built for a close sister variety, then have people fix its output; this is cheaper than annotating from scratch.
  • I2. Define a conventional spelling layer (as CODA does) that maps the many ad-hoc spellings of a word to one canonical form while keeping the original text.
  • I3. Attach variety labels at the document level rather than per token, which keeps metadata cheap but still allows the data to be split by variety.
  • I4. Use long, community-curated fiction written in everyday language as a large source of authentic informal text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Use the en tagger and parsers to pre-annotate informal or dialectal English, then route only the disagreements to LLM-teacher or human correction tiers.
  • I2 (tokens): Add an explainable normalization step that links variant spellings like "gonna" or "u" to a canonical lemma, with the raw surface form stored alongside.
  • I3 (learn): Tag silver training texts by register or dialect so evaluation can report per-variety accuracy and detect domain-shift failures.
  • I4 (answer): Long conversational narratives could provide test material for QA on stories, beyond short fables.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States201 works · 100,180 citations🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations🎓University of BahrainIsa Town · Bahrain4 works · 271 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Salam Khalifa, Nizar Habash, Dana Abdulrahim and Sara Hassan. 2016. A Large Scale Corpus of Gulf Arabic. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{khalifa2016large,
  title = {{A Large Scale Corpus of Gulf Arabic}},
  author = {Salam Khalifa, Nizar Habash, Dana Abdulrahim and Sara Hassan},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/4h6puu8bogss},
  url = {https://doi.org/10.63317/4h6puu8bogss}
}

References

46 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abo Bakr, H., Shaalan, K., and Ziedan, I. (2008). A Hybrid Approach for Converting Written Egyptian Colloquial Dialect into Diacritized Arabic. In The 6th International Conference on Informatics and Systems, IN-FOS2008. Cairo University. find
  2. Abuata, B. and Al-Omari, A. (2015). A rule-based stemmer for Arabic Gulf Dialect. Journal of King Saud University - Computer and Information Sciences, 27(2):104 – 112. ↗
  3. Al-Badrashiny, M., Eskander, R., Habash, N., and Rambow, O. (2014). Automatic Transliteration of Romanized Dialectal Arabic. CoNLL-2014, page 30.
  4. Al-Sabbagh, R. and Girju, R. (2012). A supervised POS tagger for written Arabic social networking corpora. In Jeremy Jancsary, editor, Proceedings of KONVENS 2012, pages 39–52. ÖGAI, September. Main track: oral presentations. find
  5. Al-Shargi, F., Kaplan, A., Eskander, R., Habash, N., and Rambow, O. (2016). A Morphologically Annotated Corpus and a Morphological Analyzer for Moroccan and Sanaani Yemeni Arabic. In Proceedings of the International Conference on Language Resources and Evaluation (LREC), Portorož, Slovenia. find
  6. Al-Sughaiyer, I. A. and Al-Kharashi, I. A. (2004). Arabic morphological analysis techniques: A comprehensive survey. Journal of the American Society for Information Science and Technology, 55(3):189–213. ↗
  7. Bouamor, H., Habash, N., and Oflazer, K. (2014). A multi-dialectal parallel corpus of arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). European Language Resources Association (ELRA). find
  8. Brustad, K. (2000). The Syntax of Spoken Arabic: A Comparative Study of Moroccan, Egyptian, Syrian, and Kuwaiti Dialects. Georgetown University Press. find
  9. Buckwalter, T. (2004). Buckwalter Arabic Morphological Analyzer Version 2.0. LDC catalog number LDC2004L02, ISBN 1-58563-324-0. find
  10. Darwish, K. (2013). Arabizi Detection and Conversion to Arabic. CoRR. ↗
  11. Diab, M., Habash, N., Rambow, O., AlTantawy, M., and Benajiba, Y. (2010). Colaba: Arabic dialect annotation and processing. In Proceedings of the LREC Workshop for Language Resources (LRs) and Human Language Technologies (HLT) for Semitic Languages: Status, Updates, and Prospects. find
  12. Diab, M. T., Al-Badrashiny, M., Aminian, M., Attia, M., Elfardy, H., Habash, N., Hawwari, A., Salloum, W., Dasigi, P., and Eskander, R. (2014). Tharwa: A large scale dialectal arabic-standard arabic-english lexicon. In LREC, pages 3782–3789.
  13. Eskander, R., Habash, N., and Rambow, O. (2013). Automatic Extraction of Morphological Lexicons from Morphologically Annotated Corpora. In Proceedings of tenth Conference on Empirical Methods in Natural Language Processing. find
  14. Gadalla, H., Kilany, H., Arram, H., Yacoub, A., El-Habashi, A., Shalaby, A., Karins, K., Rowson, E., Mac-Intyre, R., Kingsbury, P., Graff, D., and McLemore, C. (1997). CALLHOME Egyptian Arabic Transcripts. In Linguistic Data Consortium, Philadelphia. find
  15. Graff, D., Maamouri, M., Bouziri, B., Krouna, S., Kulick, S., and Buckwalter, T. (2009). Standard Arabic Morphological Analyzer (SAMA) Version 3.1. Linguistic Data Consortium LDC2009E73. find
  16. Habash, N. and Rambow, O. (2006). MAGEAD: A Morphological Analyzer and Generator for the Arabic Dialects. In Proceedings of the 44th Meeting of the Association for Computational Linguistics (ACL’06), Sydney, Australia. ↗
  17. Habash, N. and Roth, R. (2009). CATiB: The Columbia Arabic Treebank. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 221–224, Suntec, Singapore.
  18. Habash, N., Soudi, A., and Buckwalter, T. (2007). On Arabic Transliteration. In A. van den Bosch et al., editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. ↗
  19. Habash, N., Eskander, R., and Hawwari, A. (2012a). A Morphological Analyzer for Egyptian Arabic. In NAACL-HLT 2012 Workshop on Computational Morphology and Phonology (SIGMORPHON2012), pages 1–9.
  20. Habash, N., Diab, M. T., and Rambow, O. (2012b). Conventional Orthography for Dialectal Arabic. In LREC, pages 711–718.
  21. Habash, N., Roth, R., Rambow, O., Eskander, R., and Tomeh, N. (2013). Morphological Analysis and Disambiguation for Dialectal Arabic. In Proceedings of the 2013 Conference of NAACL-HLT, Atlanta, GA.
  22. Halefom, G., Leung, T., and Ntelitheos, D. (2013). A corpus of Emirati Arabic. Technical Report NRF Grant (31 H001), United Arab Emirates University. find
  23. Holes, C. (1990). Gulf Arabic. Psychology Press. find
  24. Jarrar, M., Habash, N., Akra, D., and Zalmout, N. (2014). Building a Corpus for Palestinian Arabic: a Preliminary Study. ANLP 2014, page 18.
  25. Kilany, H., Gadalla, H., Arram, H., Yacoub, A., El-Habashi, A., and McLemore, C. (2002). Egyptian Colloquial Arabic Lexicon. LDC catalog number LDC99L22. find
  26. Maamouri, M. and Cieri, C. (2002). Resources for Arabic Natural Language Processing. In International Symposium on Processing Arabic, volume 1. find
  27. Maamouri, M., Bies, A., Buckwalter, T., and Mekki, W. (2004). The Penn Arabic Treebank: Building a Large-Scale Annotated Arabic Corpus. In NEMLAR Conference on Arabic Language Resources and Tools, pages 102–109, Cairo, Egypt. find
  28. Maamouri, M., Bies, A., Buckwalter, T., Diab, M., Habash, N., Rambow, O., and Tabessi, D. (2006). Developing and Using a Pilot Dialectal Arabic Treebank. In Proceedings of the Fifth International Conference on Language Resources and Evaluation, LRECâĂŹ06.
  29. Maamouri, M., Bies, A., Kulick, S., Tabessi, D., and Krouna, S. (2012a). Egyptian Arabic Treebank DF Parts 1-8 V2.0 - LDC catalog numbers LDC2012E93, LDC2012E98, LDC2012E89, LDC2012E99, LDC2012E107, LDC2012E125, LDC2013E12, LDC2013E21. find
  30. Maamouri, M., Krouna, S., Tabessi, D., Hamrouni, N., and Habash, N. (2012b). Egyptian Arabic Morphological Annotation Guidelines. find
  31. Maamouri, M., Bies, A., Kulick, S., Ciul, M., Habash, N., and Eskander, R. (2014). Developing an Egyptian Arabic Treebank: Impact of Dialectal Morphology on Annotation and Tool Development. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). European Language Resources Association (ELRA). find
  32. MacWhinney, B. (2000). The childes project: Tools for analyzing talk: Volume i: Transcription format and programs, volume ii: The database. Computational Linguistics, 26(4):657–657. find
  33. Masmoudi, A., Ellouze Khmekhem, M., Esteve, Y., Hadrich Belguith, L., and Habash, N. (2014). A corpus and phonetic dictionary for tunisian arabic speech recognition. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). European Language Resources Association (ELRA).
  34. Pasha, A., Al-Badrashiny, M., Kholy, A. E., Eskander, R., Diab, M., Habash, N., Pooleery, M., Rambow, O., and Roth, R. (2014). MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In In Proceedings of LREC, Reykjavik, Iceland.
  35. Qafisheh, H. A. (1977). A Short Reference Grammar of Gulf Arabic. find
  36. Saadane, H. and Habash, N. (2015). A Conventional Orthography for Algerian Arabic. In ANLP Workshop 2015, page 69.
  37. Salloum, W. and Habash, N. (2011). Dialectal to Standard Arabic Paraphrasing to Improve Arabic-English Statistical Machine Translation. In Proceedings of the First Workshop on Algorithms and Resources for Modelling of Dialects and Language Varieties, pages 10–21, Edinburgh, Scotland. ↗
  38. Sarnākh. (2014). Sukayk 1: riwāyah emārātiyya mah.aliyyah sākhirah. Midad Publishing & Distribution. find
  39. Shoufan, A. and Al-Ameri, S. (2015). Natural language processing for dialectical arabic: A survey. In ANLP Workshop 2015, page 36.
  40. Smaïli, K., Abbas, M., Meftouh, K., and Harrat, S. (2014). Building resources for Algerian Arabic dialects. In 15th Annual Conference of the International Communication Association Interspeech. find
  41. Smrž, O. and Hajič, J. (2006). The Other Arabic Treebank: Prague Dependencies and Functions. In Ali Farghaly, editor, Arabic Computational Linguistics: Current Implementations. CSLI Publications. find
  42. Turki, H., Adel, E., Daouda, T., and Regragui, N. (2016). A Conventional Orthography for Maghrebi Arabic. In Proceedings of the International Conference on Language Resources and Evaluation (LREC), Portorož, Slovenia. find
  43. Zaghouani, W., Mohit, B., Habash, N., Obeid, O., Tomeh, N., Rozovskaya, A., Farra, N., Alkuhlani, S., and Oflazer, K. (2014). Large scale Arabic error annotation: Guidelines and framework. In International Conference on Language Resources and Evaluation (LREC 2014).
  44. Zaghouani, W. (2014). Critical survey of the freely available arabic corpora. In Proceedings of the Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools, LREC, pages 1–8. find
  45. Zaidan, O. F. and Callison-Burch, C. (2011). The Arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 37–41. Association for Computational Linguistics.
  46. Zribi, I., Boujelbane, R., Masmoudi, A., Ellouze, M., Belguith, L., and Habash, N. (2014). A Conventional Orthography for Tunisian Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khalifa-2016-large-scale-corpus-gulf-arabic/