Elamur Research⌕
☰
home / papers / 2019 / Arabic Text Diacritization Using Deep Neural Networks

Arabic Text Diacritization Using Deep Neural Networks

Summary

Arabic is usually written without most of its short-vowel marks (diacritics), yet these marks decide pronunciation and meaning. The same consonant skeleton can be a verb ("went") or a noun ("gold") depending on context. The authors argue that progress on restoring these marks has been slowed by scarce open resources and inconsistent evaluation. They make three contributions. First, they review existing diacritization systems, both neural and rule-based or statistical. Second, they release a free, cleaned benchmark built from the Tashkeela Corpus. Third, they redefine the two standard measures, the per-mark error rate (DER) and the per-word error rate (WER), so that digits and punctuation are excluded from scoring, which makes the results stricter. They then run six publicly reachable tools on the benchmark. The open-source neural system Shakkala, a B-LSTM over character embeddings, wins by a wide margin. Its DER falls to about 3%, while the best non-neural tool, Mishkal, stays near 14%. The work matters because it gives Arabic diacritization a shared, reproducible baseline to beat.

Key points

  • The benchmark is extracted from the Tashkeela Corpus and holds about 55K lines and roughly 2.3M words, released for anyone to use.
  • Six systems were compared: Farasa, Harakat, MADAMIRA (Aramorph), Mishkal, Tashkeela-Model and Shakkala. Ali-Soft was left out because its output had too many defects:
    • it skipped final letters;
    • it duplicated letters and words;
    • it inserted spurious Dagger Alif and Wasla marks.
  • Shakkala's error rates:
    • DER is 3.73% with case endings and 2.88% without them, counting "no diacritic" positions.
    • Its best WER is 6.37%.
  • Mishkal is the best non-neural tool, with DER 13.78% and WER 21.92%. Tashkeela-Model performs worst, with WER above 90%.
  • Each metric is reported in four variants: with or without the case ending, and including or excluding letters that carry no diacritic. The authors also contrast their definitions with those of Zitouni and Sarikaya.

Interesting ideas

  • I1. Report one metric under several explicit definitions (with/without word-final case marks, counting or skipping "empty" positions), since rankings and gaps can shift with the definition. — For Mova (learn): Evaluation reports could show each score in a few labelled variants, such as with and without hard constructions. This would reveal whether a gain is real or just an artefact of how the metric is defined.
  • I2. Leave tokens that carry no real decision, such as digits and punctuation, out of the error rate so trivial items cannot inflate accuracy. — For Mova (tree): Parser evaluation could include a strict LAS that ignores punctuation attachments, giving a more honest view of how well real syntactic decisions are made.
  • I3. Shakkala was retrained several times, each time dropping training data that hurt the model, which is a simple form of data cleaning driven by model behaviour. — For Mova (learn): Silver data agreed by witness parsers could be filtered the same way, removing sentences whose inclusion lowers held-out scores before the final training run.
  • I4. Before comparing third-party tools, the authors catalogued each tool's systematic output quirks, normalised the fixable ones and excluded tools whose output could not be aligned. — For Mova (learn): When external parsers serve as witnesses, a per-witness list of known artefacts plus a normalisation pass would stop their quirks from passing as disagreement or as agreement.
  • I5. One spelling can belong to different parts of speech, so context-reading character-level BiLSTMs are used to choose the reading from surrounding text. — For Mova (tokens): English homographs like "lead" or "wind" could be resolved by a small character-plus-context tagger whose choice is logged, keeping the decision explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

28 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 15 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1905.01965 · arXiv 1905.01965

@misc{fadel2019arabic,
  title = {{Arabic Text Diacritization Using Deep Neural Networks}},
  author = {Fadel, Ali and Tuffaha, Ibraheem and Al-Jawarneh, Bara' and Al-Ayyoub, Mahmoud},
  year = {2019},
  doi = {10.48550/arxiv.1905.01965},
  eprint = {1905.01965},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1905.01965}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. N. Y. Habash, “Introduction to arabic natural language processing,” Synthesis Lectures on Human Language Technologies, vol. 3, no. 1, pp. 1–187, 2010. find
  2. A. Farghaly and K. Shaalan, “Arabic natural language processing: Challenges and solutions,” ACM Transactions on Asian Language Information Processing (TALIP), vol. 8, no. 4, p. 14, 2009. find
  3. M. Al-Ayyoub, A. A. Khamaiseh, Y. Jararweh, and M. N. Al-Kabi, “A comprehensive survey of arabic sentiment analysis,” Information Processing & Management, vol. 56, no. 2, pp. 320–342, 2019. find
  4. G. A. Abandah, A. Graves, B. Al-Shagoor, A. Arabiyat, F. Jamour, and M. Al-Taee, “Automatic diacritization of arabic text using recurrent neural networks,” International Journal on Document Analysis and Recognition (IJDAR), vol. 18, no. 2, pp. 183–197, 2015. find
  5. Y. Belinkov and J. Glass, “Arabic diacritization with recurrent neural networks,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015, pp. 2281–2285. find our card
  6. Y. Jararweh, M. Al-Ayyoub, and E. Benkhelifa, “Advanced arabic natural language processing (anlp) and its applications: Introduction to the special issue,” Information Processing & Management, vol. 56, no. 2, pp. 259–261, 2019. find
  7. M. Al-Ayyoub, A. Nuseir, K. Alsmearat, Y. Jararweh, and B. Gupta, “Deep learning for arabic nlp: A survey,” Journal of computational science, vol. 26, pp. 522–531, 2018. find
  8. I. Zitouni and R. Sarikaya, “Arabic diacritic restoration approach based on maximum entropy models,” Computer Speech & Language, vol. 23, no. 3, pp. 257–276, 2009. find
  9. A. Pasha, M. Al-Badrashiny, M. T. Diab, A. El Kholy, R. Eskander, N. Habash, M. Pooleery, O. Rambow, and R. Roth, “Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic.” in LREC, vol. 14, 2014, pp. 1094–1101. find our card
  10. A. Shahrour, S. Khalifa, and N. Habash, “Improving arabic diacritization through syntactic analysis,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015, pp. 1309– 1315. TABLE V C OMPARISON OF OUR DEFINITION OF DER AND THE DEFINITION OF Z ITOUNI AND S ARIKAYA [8] With case ending Without case ending Including ’no diacritic’ 4.55% 6.25% 3.85% 5.56% DER Ours Zitouni and Sarikaya [8] TABLE VI C OMPARISON OF OUR DEFINITION OF WER AND THE DEFINITION OF Z ITOUNI AND S ARIKAYA [8] With case ending Without case ending Including ’no diacritic’ 16.67% 16.67% 12.50% 12.50% WER Ours Zitouni and Sarikaya [8] With case ending Without case ending Including ’no diacritic’ 21.43% 23.93% 18.37% 17.03% 34.38% 29.94% 16.09% 13.78% 49.96% 52.96% 3.73% 2.88% DER Farasa Harakat MADAMIRA (Aramorph) Mishkal Tashkeela-Model Shakkala With case ending Without case ending Including ’no diacritic’ 58.88% 53.13% 41.83% 32.03% 76.58% 59.07% 39.78% 26.42% 96.80% 94.16% 11.19% 6.53% WER Farasa Harakat MADAMIRA (Aramorph) Mishkal Tashkeela-Model Shakkala find our card
  11. R. Alnefaie and A. M. Azmi, “Automatic minimal diacritization of arabic texts,” Procedia Computer Science, vol. 117, pp. 169–174, 2017. find
  12. M. Bebah, C. Amine, M. Azzeddine, and L. Abdelhak, “Hybrid approaches for automatic vowelization of arabic texts,” arXiv preprint arXiv:1410.2646, 2014. ↗
  13. A. Chennoufi and A. Mazroui, “Morphological, syntactic and diacritics rules for automatic diacritization of arabic sentences,” Journal of King Saud University-Computer and Information Sciences, vol. 29, no. 2, pp. 156–163, 2017. find
  14. K. Darwish, H. Mubarak, and A. Abdelali, “Arabic diacritization: Stats, rules, and hacks,” in Proceedings of the Third Arabic Natural Language Processing Workshop, 2017, pp. 9–17. With case ending Without case ending Excluding ’no diacritic’ 6.25% 9.09% 6.25% 9.09% With case ending Without case ending Excluding ’no diacritic’ 16.67% 16.67% 12.50% 12.50% TABLE VII DER RESULTS With case ending Without case ending Excluding ’no diacritic’ 24.90% 27.55% 20.64% 18.55% 40.03% 33.87% 17.59% 14.22% 58.50% 60.92% 4.36% 3.33% TABLE VIII WER RESULTS With case ending Without case ending Excluding ’no diacritic’ 57.28% 51.84% 38.33% 28.29% 75.39% 57.22% 35.63% 21.92% 96.03% 92.45% 10.89% 6.37% find our card
  15. A. Fashwan and S. Alansary, “Shakkil: an automatic diacritization system for modern standard arabic texts,” in Proceedings of the Third Arabic Natural Language Processing Workshop, 2017, pp. 84–93. find
  16. A. M. Azmi and R. S. Almajed, “A survey of automatic arabic diacritization techniques,” Natural Language Engineering, vol. 21, no. 3, pp. 477–495, 2015. find
  17. M. Elshafei, H. Al-Muhtaseb, and M. Alghamdi, “Statistical methods for automatic diacritization of arabic text,” in The Saudi 18th National Computer Conference. Riyadh, vol. 18, 2006, pp. 301–306. find
  18. A. Barqawi and T. Zerrouki, “Shakkala, arabic text vocalization,” 2017. [Online]. Available: https://github.com/Barqawiz/Shakkala ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fadel-2019-arabic-text-diacritization-using-deep/