Elamur Research⌕
☰
home / papers / 2017 / SHAKKIL: An Automatic Diacritization System for Modern Standard Arabic Texts

SHAKKIL: An Automatic Diacritization System for Modern Standard Arabic Texts

Paper · 2017 · Amany Fashwan and Sameh Alansary

Summary

Arabic is normally written without short-vowel marks, so one written form can stand for several different words. SHAKKIL tries to restore these diacritics automatically in Modern Standard Arabic. It splits the task in two. The first part is word-internal (morphological) vowels. The second part is the case ending, which depends on the word's syntactic role.

For the internal vowels, a word passes through up to four layers:

  • an unambiguous-form lookup,
  • hand-extracted contextual rules that discard impossible analyses,
  • bigram language models over word forms, lemmas, stems, tags and suffixes,
  • a pattern-based fallback for out-of-vocabulary words.

Case endings come from ordered rules that look at a few neighbouring words. These rules use definiteness, stem pattern and verb transitivity, which are computed beforehand.

Rules and statistics were derived from the MASAR annotated corpus. On the standard ATB3 benchmark, the total WER is 14.78%. That is somewhat behind the best published systems (11.68–13.60%), but close to Metwally et al. The paper shows how explicit linguistic rules and statistical models can be combined.

Key points

  • Data: MASAR gold-standard MSA corpus (news, web articles, books), split into roughly 450k training, 55k development and 55k test words. Each word carries lemma, stem, affixes, tag, gender, number, definiteness, root, pattern, case ending and vocalization.
  • Rules: morphological rules fall into 11 categories (for example particles, pronouns, adverbs). Case-ending rules fall into 4 categories (verb mood, noun, adverb and adjective phrases). The case rules inspect a ±3-word window.
  • Language models: bigram models were built with Kneser-Ney, Witten-Bell and Katz back-off smoothing, and these were compared to choose the best one.
  • Which layer decided each test word:
    • 51.63% by the unambiguous layer,
    • 5.56% by rules alone,
    • 8.26% by rules plus statistics,
    • 32.99% by statistics alone,
    • 1.56% by the OOV layer (WER 11.2%, DER 6.7% on those words).
  • ATB3 benchmark (An-Nahar, 91 articles, about 52k words):
    • with case endings: WER 14.78%, DER 4.11%,
    • ignoring the last vowel: WER 4.81%, DER 1.93%.
    • On the MASAR test set: morphological WER 4.56%, syntactic WER 9.36%.

Interesting ideas

  • I1. A cascade that settles unambiguous words first, then rules, then statistics, then a fallback, and reports what share of tokens each stage resolved, makes it easy to see where decisions come from.
  • I2. Context rules are used to rule out analyses, not to pick one, and may leave several candidates for the statistical model to choose between; this keeps rules safe and lets them cooperate with a model.
  • I3. Features such as definiteness and verb transitivity are computed before the syntactic decision and fed into it as explicit conditions, instead of being learned implicitly.
  • I4. Error rates are reported both with and without the syntax-dependent final vowel, which separates lexical mistakes from structural ones.
  • I5. The error analysis tracks how a wrong proper-noun-versus-common-word decision also corrupts the case ending of the next word, i.e. it measures error propagation, not just isolated mistakes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): the en tagger/lemmatizer could tag every token with the stage that fixed its analysis, giving a free explanation trace and per-stage error accounting.
  • I2 (tokens): morphology rules could prune impossible readings (e.g. no verb right after a determiner) before the perceptron tagger scores what remains.
  • I3 (frames): VerbNet-derived transitivity and definiteness flags could be precomputed per token so that role assignment rules state their conditions openly and stay explainable.
  • I4 (learn): evaluation could report tagging/lemma errors separately from attachment errors on the same sentences, to show which layer a regression belongs to.
  • I5 (tree): when a POS error occurs, count how many downstream dependency arcs it breaks, to prioritise fixing the tagger errors that do the most damage.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Alexandria UniversityAlexandria · Egypt39 works · 287 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Amany Fashwan and Sameh Alansary. 2017. SHAKKIL: An Automatic Diacritization System for Modern Standard Arabic Texts. Proceedings of the Third Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-1311

@inproceedings{fashwan2017shakkil,
  title = {{SHAKKIL: An Automatic Diacritization System for Modern Standard Arabic Texts}},
  author = {Amany Fashwan and Sameh Alansary},
  booktitle = {Proceedings of the Third Arabic Natural Language Processing Workshop},
  year = {2017},
  doi = {10.18653/v1/w17-1311},
  url = {https://doi.org/10.18653/v1/w17-1311}
}

References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gheith A. Abandah, Alex Graves, Balkees Al-Shagoor, Alaa Arabiyat, Fuad Jamour, and Majid Al-Taee. 2015. Automatic diacritization of arabic text using recurrent neural networks. International Journal on Document Analysis and Recognition (IJDAR), 18(2):183–197. ↗
  2. Nizar Habash and Owen Rambow. 2007a. Arabic diacritization through full morphological tagging. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pages 53–56. Association for Computational Linguistics.
  3. Mohamed Attia Mohamed Elaraby Ahmed. 2000. A large-scale computational processor of the arabic morphology, and applications. Master’s thesis, Faculty of Engineering, Cairo University Giza, Egypt. find
  4. Nizar Habash, Ryan Gabbard, Owen Rambow, Seth Kulick, and Mitchell P. Marcus. 2007b. Determining case in arabic: Learning complex linguistic behavior requires complex linguistic features. In EMNLP-CoNLL, pages 1084–1092.
  5. Sameh Alansary. 2015. Basma: Bibalex standard arabic morphological analyzer. In 15th International Conference on Language Engineering. The Egyptian Society of Language Engineering (ESOLE). ↗
  6. Sameh Alansary. 2016a. Alserag: An automatic diacritization system for arabic. In 16th International Conference on Language Engineering. The Egyptian Society of Language Engineering (ESOLE). ↗
  7. Nizar Habash, Owen Rambow, and Ryan Roth. 2009. Mada+ tokan: A toolkit for arabic tokenization, diacritization, morphological disambiguation, pos tagging, stemming and lemmatization. In Proceedings of the 2nd international conference on Arabic language resources and tools (MEDAR), Cairo, Egypt, volume 41, page 62. find
  8. Sameh Alansary. 2016b. Masar: A morphologically annotated gold standard arabic resource. In 16th International Conference on Language Engineering. The Egyptian Society of Language Engineering (ESOLE). find
  9. Christopher D. Manning and Hinrich Schütze. 1999. Foundations of statistical natural language processing, volume 999. MIT Press. find
  10. Tim Buckwalter. 2004. Buckwalter arabic morphological analyzer version 2.0. linguistic data consortium, university of pennsylvania, 2002. ldc cat alog no.: Ldc2004l02. Technical report, ISBN 1-58563-324-0. find
  11. Aya S. Metwally, Mohsen A. Rashwan, and Amir F. Atiya. 2016. A multi-layered approach for arabic text diacritization. In Cloud Computing and Big Data Analysis (ICCCBDA), 2016 IEEE International Conference on, pages 389–393. IEEE. find
  12. Amine Chennoufi and Azzeddine Mazroui. 2016a. Impact of morphological analysis and a large training corpus on the performances of arabic diacritization. International Journal of Speech Technology, 19(2):269–280. ↗
  13. Mohsen A. Rashwan, Mohammad Al-Badrashiny, Mohamed Attia, and Sherif Abdou. 2009. A hybrid system for automatic arabic diacritization. In The 2nd International Conference on Arabic Language Resources and Tools. find
  14. Amine Chennoufi and Azzeddine Mazroui. 2016b. Morphological, syntactic and diacritics rules for automatic diacritization of arabic sentences. Journal of King Saud University-Computer and Information Sciences. ↗
  15. Mohsen A. Rashwan, Al-Badrashiny Mohamad, Mohamed Attia, Sherif Abdou, and Ahmed Rafea. 2011. A stochastic arabic diacritizer based on a hybrid of factorized and unfactorized textual features. IEEE Transactions on Audio, Speech, and Language Processing, 19(1):166–175. ↗
  16. Mona Diab, Mahmoud Ghoneim, and Nizar Habash. 2007. Arabic diacritization in the context of statistical machine translation. In Proceedings of MT-Summit.
  17. Mohsen A. Rashwan, Ahmad A. Al Sallab, Hazem M. Raafat, and Ahmed Rafea. 2014. Automatic arabic diacritics restoration based on deep nets. ANLP 2014, page 65.
  18. Amany Fashwan and Sameh Alansary. 2016. A rule based method for adding case ending diacritics for modern standard arabic texts. In 16th International Conference on Language Engineering. The Egyptian Society of Language Engineering (ESOLE). find
  19. Mohsen A. Rashwan, Ahmad A. Al Sallab, Hazem M. Raafat, and Ahmed Rafea. 2015. Deep learning framework with confused sub-set resolution architecture for automatic arabic diacritization. IEEE Transactions on Audio, Speech, and Language Processing, 23(3):505–516. ↗
  20. Amany Fashwan. 2016. Automatic diacritization of modern standard arabic texts: A corpus based approach. Master’s thesis, Faculty of Arts, University of Alexanderia, Egypt. ↗
  21. Ryan Roth, Owen Rambow, Nizar Habash, Mona Diab, and Cynthia Rudin. 2008. Arabic morphological tagging, diacritization, and lemmatization using lexeme models and feature ranking. In Proceedings of Nizar Habash and Owen Rambow. 2005. Arabic tokenization, part-of-speech tagging and morphological the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, pages 117–120. Association for Computational Linguistics.
  22. Khaled Shaalan, Hitham M. Abo Bakr, and Ibrahim Ziedan. 2008. A statistical method for adding case ending diacritics for arabic text. In Proceedings of Language Engineering Conference, pages 225–234. find
  23. Khaled Shaalan, Hitham M. Abo Bakr, and Ibrahim Ziedan. 2009. A hybrid approach for building arabic diacritizer. In Proceedings of the EACL 2009 workshop on computational approaches to semitic languages, pages 27–35. Association for Computational Linguistics.
  24. Anas Shahrour, Salam Khalifa, and Nizar Habash. 2015. Improving arabic diacritization through syntactic analysis. In EMNLP, pages 1309–1315.
  25. Imed Zitouni, Jeffrey S. Sorensen, and Ruhi Sarikaya. 2006. Maximum entropy based restoration of arabic diacritics. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 577–584. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fashwan-2017-shakkil-automatic-diacritization-system-modern/