Elamur Research⌕
☰
home / papers / 2019 / A System for Diacritizing Four Varieties of Arabic

A System for Diacritizing Four Varieties of Arabic

Paper · 2019 · Hamdy Mubarak, Ahmed Abdelali, Kareem Darwish et al.

Summary

Arabic is usually written without short vowels (diacritics). Readers must infer them to pronounce words, so automatic diacritic restoration matters for text-to-speech and language learning. This demo paper presents a deployed system that adds diacritics back for four varieties: Classical Arabic (CA), Modern Standard Arabic (MSA), and two dialects, Moroccan (DA-MA) and Tunisian (DA-TN).

The approach treats diacritization as character-level translation. An OpenNMT seq2seq model maps undiacritized character sequences to diacritized ones and needs no hand-built features, segmenters or POS taggers. To keep word order fixed, the model works on overlapping 7-word windows, and votes across windows pick each word's final form. Around the model sit:

  • Farasa-based cleaning,
  • a fastText character-n-gram classifier that guesses the variety,
  • a Flask/Gunicorn/NGINX web service with a REST API.

The authors also built a large CA corpus and release a CA test set. Word error rates drop substantially below every compared system for all four varieties, showing that one feature-free architecture transfers across standard and dialectal Arabic.

Key points

  • WER: MSA 4.5% (best prior, Microsoft ATKS, 12.3%); CA 3.7% (Farasa 12.8%); DA-MA 1.4% (Bi-LSTM 2.7%); DA-TN 2.5% (Bi-LSTM 3.6%). This is a 25–60% error reduction over earlier neural systems.
  • Data:
    • MSA: 4.5m-word training corpus, tested on WikiNews (18.3k words).
    • CA: a publisher's 65m-word corpus, with a released test set of 5,000 sentences (~400k words).
    • DA: diacritized New Testament translations (166k Moroccan and 157k Tunisian words), split 70/10/20.
  • Model: OpenNMT-tf with two RNN layers of 500 units and 300-dimensional embeddings. Inputs are 7-word sliding windows, combined by majority voting; ties go to the window where the word sits in the centre.
  • Variety identification: fastText on character 3–6-grams. Accuracy rises with input length, from 93.8% at 5 words to 99.1% at 20 words.
  • Deployment: a 4-core, 16 GB Azure VM with no GPU handles about 672 requests per second, averaging 149 ms per request.

Interesting ideas

  • I1. Overlapping fixed-size windows let one token get several predictions, which are then voted on, with the most central context breaking ties; this bounds input length while still giving each token full context.
  • I2. Manual error analysis found that in 7% of CA "errors" both the gold and the system output were valid alternatives, so raw WER overstates the error rate.
  • I3. The seq2seq model sometimes produced output unrelated to the input, and the authors propose forcing output letters to match input letters exactly.
  • I4. A cheap character-level classifier first detects the language variety and routes the text to a dedicated model, with its accuracy reported against input length.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Very long sentences could be parsed as overlapping chunks, voting on each token's head and relation and trusting the chunk where the token is most central.
  • I2 (learn): Evaluation could add an explicit "both acceptable" category when sampling disagreements, so models are not penalised for legitimate variant analyses.
  • I3 (back): Generation and the text → graph → text compressor could include a hard check that required content tokens survive the round trip, flagging hallucinated material.
  • I4 (tokens): A similar router could detect register or genre (e.g. informal web text vs. edited prose) before tagging, and log confidence by input length to know when routing is unreliable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Hamdy Mubarak, Ahmed Abdelali, Kareem Darwish, Mohamed Eldesouki, Younes Samih and Hassan Sajjad. 2019. A System for Diacritizing Four Varieties of Arabic. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-3037

@inproceedings{mubarak2019system,
  title = {{A System for Diacritizing Four Varieties of Arabic}},
  author = {Hamdy Mubarak, Ahmed Abdelali, Kareem Darwish, Mohamed Eldesouki, Younes Samih and Hassan Sajjad},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations},
  year = {2019},
  doi = {10.18653/v1/d19-3037},
  url = {https://doi.org/10.18653/v1/d19-3037}
}

References

24 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nizar Habash, Ramy Eskander, and Abdelati Hawwari. 2012. A morphological analyzer for egyptian arabic. In SIGMORPHON 2012, pages 1–9. ACL.
  2. Gheith A. Abandah, Alex Graves, Balkees Al-Shagoor, Alaa Arabiyat, Fuad Jamour, and Majid Al-Taee. 2015. Automatic diacritization of arabic text using recurrent neural networks. International Journal on Document Analysis and Recognition (IJDAR), 18(2):183–197. ↗
  3. Nizar Habash and Owen Rambow. 2007. Arabic diacritization through full morphological tagging. In HLT-NAACL, pages 53–56.
  4. Mustafa Jarrar, Nizar Habash, Faeq Alrimawi, Diyam Akra, and Nasser Zalmout. 2017. Curras: an annotated corpus for the palestinian arabic dialect. Language Resources and Evaluation, 51(3):745–775. ↗
  5. Ahmed Abdelali, Mohammed Attia, Younes Samih, Kareem Darwish, and Hamdy Mubarak. 2018. Diacritization of maghrebi arabic sub-dialects. arXiv preprint arXiv:1810.06619. ↗
  6. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651. ↗
  7. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of NAACL-HLT 2016 (Demonstrations), pages 11–16. Association for Computational Linguistics.
  8. Salam Khalifa, Sara Hassan, and Nizar Habash. 2017. A morphological analyzer for gulf arabic verbs. WANLP 2017, page 35.
  9. Ibrahim A. Asadi. 2017. Reading arabic with the diacritics for short vowels: vowelised but not necessarily easy to read. Writing Systems Research, 9(2):137–147. ↗
  10. G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. 2017. OpenNMT: Open-Source Toolkit for Neural Machine Translation. ArXiv e-prints.
  11. Oleksii Kuchaiev, Boris Ginsburg, Igor Gitman, Vitaly Lavrukhin, Carl Case, and Paulius Micikevicius. 2018. Openseq2seq: extensible toolkit for distributed and mixed precision training of sequence-to-sequence models. arXiv preprint arXiv:1805.10387. ↗
  12. Yonatan Belinkov and James Glass. 2015. Arabic diacritization with recurrent neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2281– 2285, Lisbon, Portugal.
  13. Denny Britz, Anna Goldie, Thang Luong, and Quoc Le. 2017. Massive Exploration of Neural Machine Translation Architectures. ArXiv e-prints.
  14. Hamdy Mubarak, Ahmed Abdelali, Hassan Sajjad, Younes Samih, and Kareem Darwish. 2019. Highly effective arabic diacritization using sequence to sequence modeling. In Proceedings of NAACL-HLT 2019.
  15. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. ↗
  16. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan M Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC-2014, Reykjavik, Iceland.
  17. Kareem Darwish, Ahmed Abdelali, Hamdy Mubarak, Younes Samih, and Mohammed Attia. 2018. Diacritization of moroccan and tunisian arabic dialects: A CRF approach. In OSACT 3: The 3rd Workshop on Open-Source Arabic Corpora and Processing Tools, page 62. find
  18. Mohsen Rashwan, Ahmad Al Sallab, M. Raafat, and Ahmed Rafea. 2015. Deep learning framework with confused sub-set resolution architecture for automatic arabic diacritization. In IEEE Transactions on Audio, Speech, and Language Processing, pages 505–516. ↗
  19. Kareem Darwish, Hamdy Mubarak, and Ahmed Abdelali. 2017. Arabic diacritization: Stats, rules, and hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17.
  20. Ahmed Said, Mohamed El-Sharqwi, Achraf Chalabi, and Eslam Kamal. 2013. A hybrid approach for arabic diacritization. In Natural Language Processing and Information Systems, pages 53–64, Berlin, Heidelberg. find
  21. Moustafa Elshafei, Husni Al-Muhtaseb, and Mansour Alghamdi. 2006. Statistical methods for automatic diacritization of arabic text. In The Saudi 18th National Computer Conference. Riyadh, volume 18, pages 301–306. find
  22. Youssef Sherif. 2018. Arabic tacotron text to speech. find
  23. Dimitra Vergyri and Katrin Kirchhoff. 2004. Automatic diacritization of arabic for acoustic modeling in speech recognition. In Proceedings of the workshop on computational approaches to Arabic script-based languages, COLING’04, pages 66–73, Geneva, Switzerland. ↗
  24. Ya’akov Gal. 2002. An HMM approach to vowel restoration in arabic and hebrew. In Proceedings of the ACL-02 workshop on Computational approaches to Semitic languages, pages 1–7. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mubarak-2019-system-diacritizing-four-varieties-arabic/