Elamur Research⌕
☰
home / papers / 2022 / Gulf Arabic Diacritization: Guidelines, Initial Dataset, and Results

Gulf Arabic Diacritization: Guidelines, Initial Dataset, and Results

Paper · 2022 · Nouf Alabbasi, Mohamed Al-Badrashiny, Maryam Aldahmani et al.

Summary

Arabic writing usually leaves out short vowels, so applications such as text-to-speech and language learning need a step that restores these diacritic marks. Most diacritization resources cover Modern Standard Arabic and Classical Arabic. Dialects have little consistent annotated data, and pronunciation can differ even between nearby towns. This paper builds a Gulf Arabic diacritization resource tied to the Dubai pronunciation. The authors selected 19,850 words from the CODA-normalized, morphologically tagged Emirati portion of the Gumar corpus of internet novels. In a workshop, two senior computational linguists and 15 native Emirati speakers wrote explicit diacritization rules and then annotated the text in stages: automatic pre-filling from a list of frequent words, correction by groups, and expert review. Two baselines were trained on the data. A word-level HMM reached 6.7% WER, and a character seq2seq model with windowed voting reached 8.6%. The guidelines and dataset offer a reproducible starting point for scaling up dialectal diacritization.

Key points

  • Dataset: 2,953 sentences / 19,850 words from the Gumar corpus (Emirati subset, CODAfied), split into 2,700 sentences / 18,174 words for training and 253 sentences / 1,676 words for testing.
  • Annotation pipeline:
    • The 1,300 most frequent words were diacritized out of context and applied automatically.
    • Four groups of native speakers, each handling about 5,000 words, then corrected the text and resolved disagreements together.
    • A senior Gulf-dialect linguist performed two review rounds.
  • Guidelines fix many conventions:
    • Word-final sukun is assumed and dropped, and default diacritics are omitted.
    • Ambiguity is broken by a fixed priority: kasra before fatha before dammah, and plural before dual.
    • Only the existing MSA diacritic marks are used.
  • HMM baseline: a KenLM word trigram model with A* lattice search, 6.7% WER. About 73.3% of its errors come from OOV words.
  • Seq2seq baseline: a re-implementation of the Mubarak et al. (2019) setup in OpenNMT, 8.6% WER.
    • The model stacks two LSTM layers that run in one direction only, each with a state size of 512, and applies a dropout rate of 0.3.
    • Input is limited to 5-word windows, with voting across windows.
    • Hallucinations account for 34.6% of its errors.

Interesting ideas

  • I1. Speed up annotation by labeling the most frequent word types once, without context, and projecting those labels onto the corpus so humans only fix exceptions; frequent types cover much of the text, so effort shrinks a lot.
  • I2. Write ambiguity tie-breakers into the guidelines as an explicit priority order, justified by frequency or set inclusion (plural is preferred over dual because dual is a subset of plural).
  • I3. Settle unclear cases with an operational probe: add a prefix to the word and see how the leading letter behaves (the "w test"), instead of relying on annotator intuition.
  • I4. Reduce seq2seq hallucination by feeding only short local windows and voting over the overlapping predictions for each token.
  • I5. Classify errors by cause (OOV versus hallucination) to judge whether the bottleneck is coverage, model choice, or data size.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data for lemmas, senses or verb classes could start from a type-level lexicon pass over frequent words, with witness parsers or LLM teachers reviewing only the contextual deviations.
  • I2 (tokens): The tagger and morphology could carry documented default rankings for ambiguous analyses (e.g., number or tense ambiguity), so every fallback choice is explainable rather than arbitrary.
  • I3 (tree): Treebank conversion and annotation guidelines could encode substitution or insertion tests as executable checks for disputed attachments or categories, making disagreements resolvable by procedure.
  • I4 (back): When regenerating text from the meaning graph or translating, local chunked generation with cross-window voting could catch invented or garbled words before they reach the output.
  • I5 (learn): Evaluation reports could break failures down by cause (unknown words, spurious structure, etc.) to decide whether to add data, lexicon entries, or repair rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations🎓King's College LondonLondon · United Kingdom42 works · 1,617 citations🎓Khalifa University of Science and TechnologyAbu Dhabi · United Arab Emirates3 works · 196 citations🎓University of SharjahSharjah · United Arab Emirates9 works · 202 citations🎓United Arab Emirates UniversityAl Ain · United Arab Emirates4 works · 89 citations🎓American University of SharjahSharjah · United Arab Emirates4 works · 81 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Nouf Alabbasi, Mohamed Al-Badrashiny, Maryam Aldahmani, Ahmed AlDhanhani, Abdullah Saleh Alhashmi, Fawaghy Ahmed Alhashmi, Khalid Al Hashemi, Rama Emad Alkhobbi, Shamma T Al Maazmi, Mohammed Ali Alyafeai, Mariam M Alzaabi, Mohamed Saqer Alzaabi, Fatma Khalid Badri, Kareem Darwish, Ehab Mansour Diab, Muhammad Morsy Elmallah, Amira Ayman Elnashar, Ashraf Hatim Elneima, MHD Tameem Kabbani, Nour Rabih, Ahmad Saad and Ammar Mamoun Sousou. 2022. Gulf Arabic Diacritization: Guidelines, Initial Dataset, and Results. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.33

@inproceedings{alabbasi2022gulf,
  title = {{Gulf Arabic Diacritization: Guidelines, Initial Dataset, and Results}},
  author = {Nouf Alabbasi, Mohamed Al-Badrashiny, Maryam Aldahmani, Ahmed AlDhanhani, Abdullah Saleh Alhashmi, Fawaghy Ahmed Alhashmi, Khalid {Al Hashemi}, Rama Emad Alkhobbi, Shamma T {Al Maazmi}, Mohammed Ali Alyafeai, Mariam M Alzaabi, Mohamed Saqer Alzaabi, Fatma Khalid Badri, Kareem Darwish, Ehab Mansour Diab, Muhammad Morsy Elmallah, Amira Ayman Elnashar, Ashraf Hatim Elneima, MHD Tameem Kabbani, Nour Rabih, Ahmad Saad and Ammar Mamoun Sousou},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.33},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.33}
}

References

19 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gheith A Abandah, Alex Graves, Balkees Al-Shagoor, Alaa Arabiyat, Fuad Jamour, and Majid Al-Taee. 2015. Automatic diacritization of arabic text using recurrent neural networks. International Journal on Document Analysis and Recognition (IJDAR), 18(2):183–197. ↗
  2. Ahmed Abdelali, Mohammed Attia, Younes Samih, Kareem Darwish, and Hamdy Mubarak. 2018. Diacritization of maghrebi arabic sub-dialects. arXiv preprint arXiv:1810.06619. ↗
  3. Yonatan Belinkov and James Glass. 2015. Arabic diacritization with recurrent neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2281– 2285, Lisbon, Portugal.
  4. Kareem Darwish, Ahmed Abdelali, Hamdy Mubarak, Younes Samih, and Mohammed Attia. 2018. Diacritization of moroccan and tunisian arabic dialects: A crf approach. OSACT, 3:62. find
  5. Kareem Darwish, Hamdy Mubarak, and Ahmed Abdelali. 2017. Arabic diacritization: Stats, rules, and hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17.
  6. Ali Fadel, Ibraheem Tuffaha, Mahmoud Al-Ayyoub, et al. 2019. Arabic text diacritization using deep neural networks. In 2019 2nd international conference on computer applications & information security (IC-CAIS), pages 1–7. IEEE.
  7. Ya’akov Gal. 2002. An hmm approach to vowel restoration in arabic and hebrew. In Proceedings of the ACL-02 workshop on Computational approaches to Semitic languages, pages 1–7. Association for Computational Linguistics.
  8. Nizar Habash, Mona Diab, and Owen Rambow. 2012. Conventional orthography for dialectal Arabic. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA).
  9. Mustafa Jarrar, Nizar Habash, Faeq Alrimawi, Diyam Akra, and Nasser Zalmout. 2017. Curras: an annotated corpus for the palestinian arabic dialect. Language Resources and Evaluation, 51(3):745–775. ↗
  10. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67–72, Vancouver, Canada. Association for Computational Linguistics.
  11. Mohammed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The penn arabic treebank: building a large-scale annotated arabic corpus. In NEMLAR Conference on Arabic Language Resources and Tools, pages 102—-109. find
  12. Hamdy Mubarak, Ahmed Abdelali, Kareem Darwish, Mohamed Eldesouki, Younes Samih, and Hassan Sajjad. 2019. A system for diacritizing four varieties of arabic. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 217–222.
  13. Rani Nelken and Stuart M Shieber. 2005. Arabic diacritization using weighted finite-state transducers. In Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages, pages 79–86. Association for Computational Linguistics.
  14. Mohsen Rashwan, Ahmad Al Sallab, M. Raafat, and Ahmed Rafea. 2015. Deep learning framework with confused sub-set resolution architecture for automatic arabic diacritization. In IEEE Transactions on Audio, Speech, and Language Processing, pages 505–516. ↗
  15. Nasser Zalmout and Nizar Habash. 2020. Joint diacritization, lemmatization, normalization, and fine-grained morphological tagging. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8297–8307.
  16. Taha Zerrouki and Amar Balla. 2017. Tashkeela: Novel corpus of arabic vocalized texts, data for auto-diacritization systems. Data in brief, 11:147–151. find
  17. Imed Zitouni, Jeffrey S Sorensen, and Ruhi Sarikaya. 2006. Maximum entropy based restoration of arabic diacritics. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 577–584. Association for Computational Linguistics.
  18. Salam Khalifa, Nizar Habash, Fadhl Eryani, Ossama Obeid, Dana Abdulrahim, and Meera Al Kaabi. 2018. A morphologically annotated corpus of emirati arabic. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018).
  19. Salam Khalifa, Sara Hassan, and Nizar Habash. 2017. A morphological analyzer for gulf arabic verbs. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 35–45.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alabbasi-2022-gulf-arabic-diacritization-guidelines-initial/