Elamur Research⌕
☰
home / papers / 2014 / Smoothing methods for a morpho-statistical approach of automatic diacritization Arabic texts (Méthodes de lissage d'une approche morpho-statistique pour la voyellation automatique des textes arabes) [in French]

Smoothing methods for a morpho-statistical approach of automatic diacritization Arabic texts (Méthodes de lissage d'une approche morpho-statistique pour la voyellation automatique des textes arabes) [in French]

Paper · 2014 · Amine Chennoufi and Azzeddine Mazroui

Summary

Most Arabic text is written without short vowels and other diacritics. This leaves the large majority of words ambiguous, and it hurts parsing, translation, treebank building and speech technology. The authors present AlKhalil Diacritizer, a three-stage hybrid system. In the first stage, the open-source analyzer AlKhalil Morpho Sys proposes every possible vowelized form of each word. A dictionary of frequent words, mined from about 250 million words in eight online corpora, supplements the analyzer. In the second stage, a word-level first-order HMM picks one candidate per word using Viterbi decoding. Its observations are unvowelized words and its hidden states are vowelized words. Its parameters are maximum-likelihood estimates from roughly 57 million vowelized words of Arabic books, and smoothing deals with word transitions never seen in training. In the third stage, a character-level HMM handles words the analyzer cannot process. The system is compared with published diacritizers. It gets the lowest full-word error and diacritic error rates, but it is not the best once case endings are ignored. Errors on the final, syntax-dependent character make up about half of its word errors.

Key points

  • Pipeline:

    • AlKhalil Morpho Sys plus a frequent-word dictionary of 16,369 vowelized entries with their alternative vowelizations.
    • A word-level HMM with smoothing and Viterbi decoding.
    • A character-level HMM fallback for unanalyzed words.

  • Training corpus: about 57 million vowelized words from classical and contemporary Arabic books (the Tashkeela resource on SourceForge is cited).

  • The emission matrix is effectively binary. An unvowelized word either matches a vowelized form once its diacritics are stripped, or it does not.

  • Full-word scores against the strongest compared system (Rashwan et al., 2011):

    MetricAlKhalil DiacritizerRashwan et al.
    WER110.06%12.50%
    DER13.72%3.80%

  • On the variants that ignore the final character, Rashwan et al. score better:

    MetricAlKhalil DiacritizerRashwan et al.
    WER25.40%3.10%
    DER22.09%1.20%

  • The authors blame part of the remaining error on spelling mistakes in the corpora, which degrade the morphological analyzer. Planned work: clean and extend the corpora, complete partially vowelized dictionary entries, and use the analyzer's syntactic information for word-final diacritics.

Interesting ideas

  • I1. A precomputed lexicon of the most frequent words sits in front of the rule-based analyzer, which both speeds analysis and lets hand-checked forms override analyzer output for the commonest tokens.
  • I2. The analyzer generates the candidate set and the statistical model only chooses among those candidates, so the 0/1 emission matrix turns the HMM into a pure contextual selector over linguistically licensed options.
  • I3. A finer-grained character-level model serves as a fallback only for words the analyzer cannot handle, rather than replacing the analyzer everywhere.
  • I4. Reporting error rates both with and without the final character separates morphological errors from syntax-driven case-ending errors, which shows where the remaining errors come from.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): A frequent-word table with verified lemma/morphology entries could short-circuit the en analyzer on high-frequency forms and pin down their preferred readings explainably.
  • I2 (tokens): Lemma and morphology disambiguation could keep the rule analyzer as the candidate generator and use a sequence model only to rank, so every choice traces back to a valid analysis.
  • I3 (tokens): Unknown or misspelled English words could go to a character-level guesser for POS and lemma, flagged as low-confidence fallback output.
  • I4 (learn): Evaluations could add paired metrics that drop one error class, such as attachment versus label or inflection versus lemma, to locate which component is responsible.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksFrench · 83 works

Cite

Cite as Amine Chennoufi and Azzeddine Mazroui. 2014. Smoothing methods for a morpho-statistical approach of automatic diacritization Arabic texts (Méthodes de lissage d'une approche morpho-statistique pour la voyellation automatique des textes arabes) [in French]. Proceedings of TALN 2014 (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chennoufi2014smoothing,
  title = {{Smoothing methods for a morpho-statistical approach of automatic diacritization Arabic texts (Méthodes de lissage d'une approche morpho-statistique pour la voyellation automatique des textes arabes) [in French]}},
  author = {Amine Chennoufi and Azzeddine Mazroui},
  booktitle = {Proceedings of TALN 2014 (Volume 2: Short Papers)},
  year = {2014},
  url = {https://aclanthology.org/F14-2010/}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alghamdi M., Muzaffar Z., Alhakami H. (2010), "Automatic Restoration of Arabic Diacritics: A Simple, Purely Statistical Approach," The Arabian Journal for Science and Engineering, vol. 35, 2010. find
  2. Buckwalter T. (2004). Arabic Morphological Analyzer version 2.0. LDC2004L02. find
  3. Chen S. F., Goodman J. (1998). An empirical study of smoothing techniques for language modeling, Harvard Univ.,Computer Science Group, Cambridge, MA, Tech. Rep. TR-10-98, August 1998. find
  4. Chennoufi A., Mazroui A. (2014). Vers un Voyaliseur Automatique des Textes Classiques de la Langue Arabe. 1ère Journée Doctorale Nationale sur L'ingénierie de la Langue Arabe, (JDILA'14), 8 Février 2014, Rabat, Maroc. find
  5. Debili F., Achour H. (1998). Voyellation automatique de l'arabe. In Proceedings of the workshop on Computation approaches to Semitic languages, COLING-ACL '98. Montréal. find
  6. El-Sadany T., Hashish M. (1988). Semi-automatic vowelization of arabic verbs. In 10th NC Conference, Saudi Arabia. find
  7. Gale W., Church K.(1994). What is wrong with adding one? In Corpus-Based Research into Language, N. Oostdijk and P. D. Haan, Eds. Amsterdam, The Netherlands: Rodopi, 1994, pp. 189–198. find
  8. Gal Y. (2002). An HMM Approach to Vowel Restoration in Arabic and Hebrew. In ACL-02 Workshop on Computational Approaches to Semitic Languages.
  9. Habash N., Rambow O.(2007), “Arabic diacritization through full morphological tagging,” in Proceedings of NAACL/HLT 2007. Companion Volume, Short Papers, Rochester, New York, USA. April. 2007.pages53-56
  10. Hifny Y.(2013). Restoration of arabic diacritics using dynamic programming. In The 8th International Conference on Computer Engineering & Systems (ICCES’2013), 26-28 Nov. 2013, Cairo, Egypt, pages 3-8 ISBN:978-1-4799-0078-7. find
  11. Maamouri M., Bies A., Kulick S. (2006). Diacritization: A Challenge to Arabic Treebank Annotation and Parsing. In Proceedings of the British Computer Society Arabic NLP/MT Conference. find
  12. Manning C. , Schutze H. (1999). Foundations of Statistical Natural Language Processing, MIT Press. Cambridge, MA. find
  13. Messaoudi A., Lamel L., Gauvain J. (2004). The LIMSI RT04 BN Arabic system. In Proc. Darpa RT04, Palisades NY. find
  14. Neuhoff D.L. (1975). The Viterbi Algorithm as an Aid in Text Recognition. In IEEE Transaction on Information Theory. Pages 222-226. find
  15. Ney H., Essen U., Kneser R. (1994). On structuring probabilistic dependences in stochastic language modeling. Computer, Speech, and Language, vol. 8, pp. 1{38, 1994. Ould Bebah M. O. A, Mazroui. find
  16. A, Meziane A., Lakhouaja A. (2011). Alkhali Morpho Sys. In International Computing Conference in Arabic. Riadh, Arabie Saoudite. find
  17. Ould Bebah M. O. A., Mazroui A ., Lakhouaja A., Chennoufi A.(2013). Approche morpho-statistique pour la voyellation automatique des texts arabes. In 8th International Conference on intelligent system: theories and applications (SITA’13), May 08-09, 2013, EMI, Rabat , Morocco. find
  18. Vergyri D., Kirchoff K. (2004). Automatic diacritization of Arabic for Acoustic Modeling in Speech Recognition. In Proceedings of the Workshop on Computational Approaches to Arabic Script-based Languages. Geneva. Pages 66-73. ↗
  19. Rashwan M., Al-Badrashiny M. , Attia M., Abdou S.M., Rafea A. (2011), "A Stochastic Arabic Diacritizer Based on a Hybrid of Factorized and Unfactorized Textual Features," Audio, Speech, and Language Processing, pages. 166-175. ↗
  20. Schlippe T., Nguyen T. , Vogel S. (2008). Diacritization as a Machine Translation Problem and as a Sequence Labeling Problem. In 8th AMTA conference, Hawaii. Pages 21-25. find
  21. Zitouni I., Sarikaya R. (2009), "Arabic Diacritic Restoration Approach Based on Maximum Entropy Models," Computer Speech & Language, vol. 23, no. 3, pp. 257-276, 2009. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chennoufi-2014-smoothing-methods-morpho-statistical-approach/