Elamur Research⌕
☰
home / papers / 2020 / Deep Diacritization: Efficient Hierarchical Recurrence for Improved Arabic Diacritization

Deep Diacritization: Efficient Hierarchical Recurrence for Improved Arabic Diacritization

Paper · 2020 · Badr AlKhamissi, Muhammad ElNokrashy and Mohamed Gabr

Summary

Written Arabic usually leaves out short vowels, consonant doubling and similar marks, so readers infer them from context. Restoring these diacritics automatically helps speech synthesis, language learning and other downstream tasks. The authors treat diacritization as character sequence labelling and design a hierarchical network. A word-level Bi-LSTM, fed with pretrained fastText vectors, builds sentence context. A character-level Bi-LSTM then runs separately inside each word. A cross-level attention module lets every character look at the other words in the sentence.

The base model is D2. An extended version, D3, adds a forward LSTM decoder that can take partially diacritized text as hints. Each character gets one class from a fixed list of valid diacritic combinations. Further gains come from sentence dropout and majority voting.

On the cleaned Tashkeela benchmark, the best system reaches 5.34% WER, cutting the previous best error by about 30.56% relative. Because character recurrence is confined to single words, training and inference are also faster. The design has no Arabic-specific parts.

Key points

  • The data is the cleaned Tashkeela corpus prepared by Fadel et al. (2019a): about 2,449k training tokens, 119k dev tokens and 125k test tokens.
  • Splitting recurrence into word and character levels reduces the serial cost from roughly O(T_s·T_w) to O(T_s + T_w), given parallel processing across words.
  • The output space has 15 classes: three short vowels, each in four variants (plain, with tanwīn, with shaddah, or with both), plus sukūn, bare shaddah and "none".
  • Main settings:
    • word encoder: 2-layer Bi-LSTM, 256 units;
    • character encoder: 2-layer Bi-LSTM, 512 units;
    • 32-dim character embeddings and dropout of 0.2 on the character encoder input;
    • Adam at a learning rate of 0.002, batches of 128 segments, learning rate halved when validation loss stalls.
  • Error analysis findings:
    • Case endings are harder than core-word diacritics.
    • The commonest confusions are dammah↔kasrah and dammah↔fatḥah.
    • Some errors come from inconsistent gold annotation, such as several valid spellings of "<lY".

Interesting ideas

  • I1. Run a small character-level model inside each word, conditioned on a contextual vector for that word from a sentence-level encoder; this keeps sequences short and parallel while still using sentence context.
  • I2. Instead of predicting several marks independently, enumerate only the legal combinations as single classes, so malformed outputs cannot occur.
  • I3. Let the decoder receive optional known labels in place of its own previous predictions, so partial hints in the input can steer the rest of the output.
  • I4. Check whether each word's attention peaks on its grammatical governor; this is a cheap probe of whether a neural model has learned syntax, and it shows a failure mode where the model attends to an earlier word with the same role.
  • I5. Use error analysis to separate model mistakes from gold-standard inconsistencies, such as several correct forms of the same word, before trusting benchmark numbers.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): A morphology/lemma tagger in crate en could analyse each word's characters this way, using sentence context to pick suffix readings, without paying for long character sequences.
  • I2 (tokens): UD feature bundles (Number+Person+Tense etc.) could be predicted as an enumerated set of attested valid bundles, giving consistent and explainable tags.
  • I3 (tree): gparse or the perceptron parser could accept arcs that witness parsers agree on as fixed hints, then fill in the remaining structure around them.
  • I4 (learn): attention or score maps from gparse could be compared against UD heads as an evaluation and negative-control signal, flagging cases where the model groups words by role rather than by dependency.
  • I5 (learn): silver and gold treebank evaluation could first detect identical contexts annotated differently, then clean them or score them leniently so repair rules are not trained on noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Badr AlKhamissi, Muhammad ElNokrashy and Mohamed Gabr. 2020. Deep Diacritization: Efficient Hierarchical Recurrence for Improved Arabic Diacritization. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alkhamissi2020deep,
  title = {{Deep Diacritization: Efficient Hierarchical Recurrence for Improved Arabic Diacritization}},
  author = {Badr AlKhamissi, Muhammad ElNokrashy and Mohamed Gabr},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.4/}
}

References

27 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hamza Abbad and Shengwu Xiong. 2020. Multi-components system for automatic arabic diacritization. In Joemon M. Jose, Emine Yilmaz, João Magalhães, Pablo Castells, Nicola Ferro, Mário J. Silva, and Flávio Martins, editors, Advances in Information Retrieval, pages 341–355, Cham. Springer International Publishing. find
  2. Aqil Azmi. 2013. A survey of automatic arabic diacritization techniques. Natural Language Engineering, 21, 10. Zerrouki Barqawi. ↗
  3. Shakkala. 2017. Shakkala, Arabic text vocalization. https://github.com/Barqawiz/ ↗
  4. Yonatan Belinkov and James Glass. 2015. Arabic diacritization with recurrent neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2281–2285, Lisbon, Portugal, September. Association for Computational Linguistics.
  5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗
  6. Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555. ↗
  7. Kareem Darwish, Hamdy Mubarak, and Ahmed Abdelali. 2017. Arabic diacritization: Stats, rules, and hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17, Valencia, Spain, April. Association for Computational Linguistics.
  8. Muhammad N. ElNokrashy. 2020. Extensible RNN cells for PyTorch. https://github.com/munael/ pt-rnn. ↗
  9. Moustafa Elshafei, Husni Al-Muhtaseb, and Mansour Alghamdi. 2006. Statistical methods for automatic diacritization of arabic text. The Saudi 18th National Computer Conference. Riyadh, 18:301–306, 01. find
  10. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019a. Arabic text diacritization using deep neural networks. In 2019 2nd International Conference on Computer Applications Information Security (ICCAIS), pages 1–7, May.
  11. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019b. Neural Arabic text diacritization: State of the art results and a novel approach for machine translation. In Proceedings of the 6th Workshop on Asian Translation, pages 215–225, Hong Kong, China, November. Association for Computational Linguistics. ↗
  12. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 1019–1027. Curran Associates, Inc.
  13. Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. 2005. Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition. In Włodzisław Duch, Janusz Kacprzyk, Erkki Oja, and Sławomir Zadrożny, editors, Artificial Neural Networks: Formal Models and Their Applications – ICANN 2005, pages 799–804, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  14. Nizar Habash, Owen Rambow, and Ryan Roth. 2009. MADA+TOKAN: A toolkit for Arabic tokenization, diacritization, morphological disambiguation, POS tagging, stemming and lemmatization. Proceedings of the 2nd International Conference on Arabic Language Resources and Tools (MEDAR), 01. find
  15. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015. ↗
  16. Hamdy Mubarak, Ahmed Abdelali, Hassan Sajjad, Younes Samih, and Kareem Darwish. 2019. Highly effective Arabic diacritization using sequence to sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2390–2395, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  17. Rani Nelken and Stuart M. Shieber. 2005. Arabic diacritization using weighted finite-state transducers. In Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages, pages 79–86, Ann Arbor, Michigan, June. Association for Computational Linguistics.
  18. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1094–1101, Reykjavik, Iceland, May. European Language Resources Association (ELRA).
  19. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc. ↗
  20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August. Association for Computational Linguistics.
  21. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929– 1958. find
  22. Teresa Tinsley and Kathryn Board. 2013. Languages for the Future. British Council. find
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762. ↗
  24. Nasser Zalmout and Nizar Habash. 2020. Joint diacritization, lemmatization, normalization, and fine-grained morphological tagging. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8297–8307, Online, July. Association for Computational Linguistics.
  25. Taha Zerrouki and Amar Balla. 2017. Tashkeela: Novel corpus of arabic vocalized texts, data for auto-diacritization systems. Data in Brief, 11:147 – 151. find
  26. Imed Zitouni and Ruhi Sarikaya. 2009. Arabic diacritic restoration approach based on maximum entropy models. Computer Speech & Language, 23:257–276, 07. ↗
  27. Imed Zitouni, Jeffrey S. Sorensen, and Ruhi Sarikaya. 2006. Maximum entropy based restoration of Arabic diacritics. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 577–584, Sydney, Australia, July. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alkhamissi-2020-deep-diacritization-efficient-hierarchical-recurrence/