Elamur Research⌕
☰
home / papers / 2019 / Automatic Diacritization as Prerequisite Towards the Automatic Generation of Arabic Lexical Recognition Tests

Automatic Diacritization as Prerequisite Towards the Automatic Generation of Arabic Lexical Recognition Tests

Paper · 2019 · Osama Hamed

Summary

This paper treats diacritic restoration as the first building block toward automatically generating Arabic lexical recognition tests (LRTs). LRTs are yes/no vocabulary tests that mix real words with nonwords. They need reliable frequency counts of diacritized forms, but diacritized Arabic text is scarce: it is mostly religious or locked in the commercial PATB. The author therefore checks how well off-the-shelf tools can add diacritics to raw text. Two freely available systems, Farasa and Madamira, are compared with a CRF sequence-labeling baseline built in DKPro TC. The comparison uses three free corpora (Quran, RDI, Tashkeela) and ten-fold cross-validation. Both tools leave out many diacritics that the gold data contains. To keep the comparison fair and match how LRT items are displayed, scoring uses a "relaxed" mode. It skips default diacritics and Sukun, and the metric variants DER-1 and WER-1 also drop case endings. Farasa beats Madamira and the baseline in every setting. This makes Farasa the practical choice for diacritizing a large crawled corpus from which test words can be counted.

Key points

  • Corpus sizes used: Quran 78K words, RDI subset 297K, Tashkeela subset 4,926K. All three are free and religious/Classical in genre, while PATB is commercial newswire.
  • The metrics are DER and WER, plus DER-1 and WER-1, which ignore each word's last letter and so exclude case endings. LRT stimuli show lexical but not syntactic diacritics.
  • Gold text carries about 0.83–0.84 diacritics per letter. Madamira and Farasa assign only about 0.55–0.61, so strict scoring would unfairly penalize them.
  • The baseline is a CRF that labels each consonant with its diacritics. Its features are character n-grams over the neighbouring characters, reaching out to the 6th character.
  • Farasa outperforms Madamira and the CRF baseline under all conditions. Madamira beats the baseline on RDI and Tashkeela but does worse on the Quran.

Interesting ideas

  • I1. Fit the evaluation to the downstream use: score only the diacritics the application will actually show (no case endings, no default marks) instead of full gold fidelity.
  • I2. Before comparing tools, measure how densely each one annotates (diacritics per letter versus gold). This reveals that a scoring method is biased against systems that annotate less.
  • I3. Normalize away writing-style conventions (here, whether Sukun marks unvowelled letters) in both gold and output before computing error rates.
  • I4. Cast restoration of a hidden layer as per-character sequence labeling, accepting that one form may have several valid label sequences.
  • I5. Generate test items (words plus nonwords) automatically from corpus statistics to allow repeated testing.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation of tagger or parser output could ignore distinctions the later stages never use, so that error numbers reflect what actually matters downstream.
  • I2 (learn): When comparing witness parsers or LLM teacher tiers, first count how often each emits optional labels or features, then normalize the scoring so sparse but correct output is not punished.
  • I3 (tree): During UD dialect conversion between treebanks, canonicalize purely conventional annotation choices before measuring agreement, so style differences do not count as parse errors.
  • I4 (tokens): Morphology and lemma disambiguation could be framed as labeling with explicitly allowed alternative analyses, keeping ambiguity visible rather than forcing a single guess.
  • I5 (answer): School-course tests could get fresh distractor items generated from corpus frequencies, so a reader's vocabulary is probed repeatedly without memorized answers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksEducation · 212 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Osama Hamed. 2019. Automatic Diacritization as Prerequisite Towards the Automatic Generation of Arabic Lexical Recognition Tests. Proceedings of the 3rd International Conference on Natural Language and Speech Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hamed2019automatic,
  title = {{Automatic Diacritization as Prerequisite Towards the Automatic Generation of Arabic Lexical Recognition Tests}},
  author = {Osama Hamed},
  booktitle = {Proceedings of the 3rd International Conference on Natural Language and Speech Processing},
  year = {2019},
  doi = {10.17185/duepublico/72018},
  url = {https://doi.org/10.17185/duepublico/72018}
}

References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gheith Abandah, Alex Graves, Balkees Al-Shagoor, Alaa Arabiyat, Fuad Jamour, and Majid Al-Taee. 2015. Automatic diacritization of arabic text using recurrent neural networks. International Journal on Document Analysis and Recognition (IJDAR), 18(2):183–197. ↗
  2. Afnan Aqel, Sahar Alwadei, and Mohammad Dahab. 2015. Building an Arabic Words Generator. International Journal of Computer Applications, 112(14). find
  3. Aqil Azmi and Reham Almajed. 2015. A survey of automatic arabic diacritization techniques. Natural Language Engineering, 21(03):477–495. find
  4. Harun Baharudin, Zawawi Ismail, Adelina Asmawi, and Normala Baharuddin. 2014. TAV of Arabic language measurement. Mediterranean Journal of Social Sciences, 5(20):2402. ↗
  5. Mohamed Bebah, Chennoufi Amine, Mazroui Azzeddine, and Lakhouaja Abdelhak. 2014. Hybrid approaches for automatic vowelization of arabic texts. arXiv preprint arXiv:1410.2646. ↗
  6. Tim Buckwalter and Dilworth Parkinson. 2014. A frequency dictionary of Arabic: Core vocabulary for learners. Routledge. ↗
  7. Lynne Cameron. 2002. Measuring vocabulary size in English as an additional language. Language Teaching Research, 6(2):145–173. ↗
  8. Andy Cole, David Graff, and Kevin Walker. 2001. Arabic newswire part 1 corpus (1-58563-190-6). Linguistic Data Consortium (LDC). find
  9. Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A New Fast and Accurate Arabic Word Segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. European Language Resources Association (ELRA).
  10. Kareem Darwish, Hamdy Mubarak, and Ahmed Abdelali. 2017. Arabic diacritization: Stats, rules, and hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17.
  11. Johannes Daxenberger, Oliver Ferschke, Iryna Gurevych, Torsten Zesch, et al. 2014. DKPro TC: A Java-based Framework for Supervised Learning Experiments on Textual Data. In ACL (System Demonstrations), pages 61–66. Corpus Approach All Diacritics Ignore Last DER WER DER-1 WER-1 Quran Seq. Labeling Madamira Farasa 15.1 14.5 7.8 22.0 26.4 14.0 7.6 10.2 5.0 13.5 15.6 6.8 RDI Seq. Labeling Madamira Farasa 16.7 12.5 8.3 28.0 20.4 13.8 13.6 8.6 5.0 12.0 10.2 5.1 Tashkeela Seq. Labeling Madamira Farasa 24.0 12.4 8.3 35.4 20.3 13.9 15.0 8.5 5.0 22.0 10.1 5.1 *Table 5: Error rates in relaxed evaluation mode.* Mona Diab, Mahmoud Ghoneim, and Nizar Habash. 2007. Arabic diacritization in the context of statistical machine translation. In Proceedings of MT-Summit.
  12. Nizar Habash. 2010. Introduction to Arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. ↗
  13. Nizar Habash, Owen Rambow, and Ryan Roth. 2009. MADA + TOKAN: A toolkit for Arabic tokenization, diacritization, morphological disambiguation, POS tagging, stemming and lemmatization. In Proceedings of the 2nd international conference on Arabic language resources and tools (MEDAR), Cairo, Egypt, pages 102–109. Osama Hamed and Torsten Zesch. Generating Non-words for Vocabulary Proficiency Testing. find
  14. Osama Hamed and Torsten Zesch. 2017. The Role of Diacritics in Designing Lexical Recognition Tests for Arabic. In 3rd International Conference on Arabic Computational Linguistics (ACLing 2017), Dubai, UAE. Elsevier. find
  15. Osama Hamed and Torsten Zesch. 2018. The role of diacritics in increasing the difficulty of Arabic lexical recognition tests. In Proceedings of the 7th workshop on NLP for Computer Assisted Language Learning, Stockholm, Sweden. LiU Electronic Press.
  16. Yasser Hifny. 2012. Smoothing techniques for arabic diacritics restoration. In 12th Conference on Language Engineering, pages 6–12. find
  17. John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. find
  18. Kristin Lemhöfer and Mirjam Broersma. 2012. Introducing LexTALE: A quick and valid lexical test for advanced learners of English. Behavior Research Methods, 44(2):325–343. ↗
  19. Aya S Metwally, Mohsen A Rashwan, and Amir F Atiya. 2016. A multi-layered approach for arabic text diacritization. In Cloud Computing and Big Data Analysis (ICCCBDA), 2016 IEEE International Conference on, pages 389–393. IEEE. find
  20. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC, pages 1094–1101.
  21. Mohsen Rashwan, Mohamed Al-Badrashiny, Mohamed Attia, Sherif Abdou, and Ahmed Rafea. 2011. A stochastic arabic diacritizer based on a hybrid of factorized and unfactorized textual features. Audio, Speech, and Language Processing, IEEE Transactions on, 19(1):166–175. ↗
  22. John Read. 2004. Plumbing the depths: How should the construct of vocabulary knowledge be defined. Vocabulary in a second language, pages 209–227. find
  23. Robert Ricks. 2015. The Development of Frequency-Based Assessments of Vocabulary Breadth and Depth for L2 Arabic. find
  24. Norbert Schmitt. 2014. Size and depth of vocabulary knowledge: What the research shows. Language Learning, 64(4):913–951. ↗
  25. Tzu-Hua Wang. 2007. What strategies are effective for formative assessment in an e-learning environment? Journal of Computer Assisted Learning, 23(3):171– 186. ↗
  26. Taha Zerrouki and Amar Balla. 2017. Tashkeela: Novel corpus of Arabic vocalized texts, data for auto-diacritization systems. Data in Brief, 11:147– 151. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hamed-2019-automatic-diacritization-as-prerequisite-automatic/