Elamur Research⌕
☰
home / papers / 2014 / Hybrid approaches for automatic vowelization of Arabic texts

Hybrid approaches for automatic vowelization of Arabic texts

Paper · 2014 · Mohamed Bebah, Chennoufi Amine, Mazroui Azzeddine et al.

Summary

Most Arabic text is written without diacritics, the short vowels and related marks. This leaves the large majority of words ambiguous and hampers parsing, text-to-speech and learners. The authors build a hybrid diacritization (vowelization) system with two stages. First, the open-source analyzer AlKhalil Morpho Sys lists every vowelized reading a word could have when taken alone. They add a lexical database of the most frequent Arabic words, drawn from eight online corpora totalling over 250 million words. This database fixes the diacritics of common words and limits the slowdown caused by too many candidate analyses. Second, two hidden Markov models choose among the candidates in context. In one, the hidden states are whole vowelized words. In the other, they are only the diacritic pattern stripped of its letters. Viterbi decoding picks the best path. The system is trained on a representative corpus and tested with and without the added lexicon. The work matters because commercial diacritizers are closed, while this one builds on an open analyzer and can be extended and embedded elsewhere.

Key points

  • Missing diacritics make over 90% of words in Arabic text ambiguous (figure cited from earlier work by Debili and Achour).
  • Stage 1 uses AlKhalil Morpho Sys (open source, on SourceForge) to produce all out-of-context vowelizations of each word.
  • A frequent-word lexicon built from eight corpora (over 250M words) is merged into the analyzer. It improves accuracy for common words and speeds up analysis.
  • Stage 2 runs two HMMs over the same observations (unvowelized words). Their hidden states are either full vowelized words or bare diacritic sequences, and Viterbi decodes the optimal path.
  • The paper situates itself against rule-based, statistical and hybrid systems: ArabDiac/RDI, KACST quad-grams, Nelken & Shieber FSTs, Zitouni et al. MaxEnt, and Buckwalter-based selection.

Interesting ideas

  • I1. An analyzer proposes a closed candidate set and a sequence model only chooses within it, so every output is a licensed analysis rather than a free guess.
  • I2. A frequency lexicon patches the analyzer for common words, giving fixed answers and cutting combinatorial blow-up where the rule system overgenerates.
  • I3. Two views of the same hidden variable are modelled: the full word, and an abstract pattern without lexical content. The abstract view generalizes to unseen words, while the full view keeps lexical precision.
  • I4. Ablation with and without the added lexicon measures that resource's own contribution instead of only the final system.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The tagger/lemmatizer in crate en could restrict choices to morphology-generated candidates and then disambiguate, keeping each decision traceable to a lexicon entry.
  • I2 (tokens): A corpus-derived table of frequent wordforms with their preferred lemma/tag could short-circuit expensive analysis and fix systematic analyzer errors on high-frequency items.
  • I3 (learn): Training could pair a lexicalized model with a delexicalized one (e.g. tag/shape sequences) so rare words still get sensible predictions, and their disagreement can flag cases for review.
  • I4 (learn): Evaluation should routinely toggle each compiled resource (lexicons, repair rules) on and off, as a control showing whether it actually helps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Arab Center for Research and Policy StudiesDoha · Qatar9 works · 69 citations🎓Mohamed I UniversityOujda · Morocco4 works · 433 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksEducation · 212 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Mohamed Bebah, Chennoufi Amine, Mazroui Azzeddine and Lakhouaja Abdelhak. 2014. Hybrid approaches for automatic vowelization of Arabic texts. arXiv:1410.2646.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.5121/ijnlc.2014.3404 · arXiv 1410.2646

@misc{bebah2014hybrid,
  title = {{Hybrid approaches for automatic vowelization of Arabic texts}},
  author = {Mohamed Bebah, Chennoufi Amine, Mazroui Azzeddine and Lakhouaja Abdelhak},
  year = {2014},
  doi = {10.5121/ijnlc.2014.3404},
  eprint = {1410.2646},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.5121/ijnlc.2014.3404}
}

References

24 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Debili, Fathi & Hadhemi Achour (1998) Voyellation automatique de l’arabe. In Proceedings of the workshop on Computation approaches to Semitic languages, COLING-ACL ’98, pages 42–49. find
  2. Maamouri, Mohamed, Ann Bies, and Seth Kulick. (2006) Diacritization: a challenge to Arabic treebank annotation and parsing. In Proceedings of the British Computer Society Arabic NLP/MT Conference. find
  3. Zitouni, Imed, Jefrey S. Sorensen, and Ruhi Sarikaya. (2006) Maximum entropy based restoration of arabic diacritics. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics. Workshop on Computational approaches to Semitic Languages, Sydney, Australia. July 2006, pages 577– 584.
  4. Vergyri, Dimitra & Katrin Kirchhoff. (2004) Automatic diacritization of arabic for acoustic modeling in speech recognition. In Proceedings of the Workshop on Computational Approaches to Arabic Script-based Languages. COLING, Geneva, pages 66–73. ↗
  5. Messaoudi, Abdel, Lori Lamel, and Jean-Luc Gauvain. (2004) The limsi rt04 b arabic system. In Proceedings DARPA RT04, Palisades NY. find
  6. Elshafei, Moustafa, Husni Al-Muhtaseb, and Mansour Alghamdi. (2006) Machine generation of arabic diacritical marks. In The 2006 World Congress in Computer Science Computer Engineering, and Applied Computing. Las Vegas, USA., pages 128–133. find
  7. Emam, Ossama and Volker Fischer. (2005) Hierarchical approach for the statistical vowelization of arabic text. Technical report, IBM Corporation Intellectual Property Law, Austin, TX, US. find
  8. Schlippe, Tim, ThuyLinh Guyen, and ThuyLinh Vogel. (2008) Diacritization as a machinetranslation problem and as a sequence labeling problem. In 8th AMTA conference, Hawai., pages 21–25. find
  9. Gal, Yaakov. (2002) An hmm approach to vowel restoration in arabic and hebrew. In Proceedings of the Workshop on Computational Approaches to Semitic Languages- Philadelphia- Association for Computational Linguistics, pages 27–33.
  10. Nelken, Rani and Stuart M. Shieber. (2005) Arabic diacritization using weighted finite-state transducers. In Proceedings of the ACL 2005 Workshop On Computational Approaches To Semitic Languages, Ann Arbor, Michigan, USA,, pages 79–86.
  11. Habash, Nizar and Owen Rambow. (2007) Arabic diacritization through full morphological tagging. In Proceeding NAACL-Short ’07 Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics - Companion Volume - Short Papers Rochester - New York- USA, pages 53–56.
  12. Bebah, Mohamed Ould Abdallahi Ould, Abdelouafi Meziane, Azzeddine Mazroui, and Abdelhak Lakhouaja. (2012) Approche morpho-statistique pour la voyellation des texts arabes. Journal of Computer Science and Engineering, 5(1). find
  13. Bebah, Mohamed Ould Abdallahi Ould, Abdelouafi Meziane, Azzeddine Mazroui, and Abdelhak Lakhouaja. (2011) Alkhalil morpho sys. In 7th International Computing Conference in Arabic, May 31- June 2, 2011, Riyadh, Saudi Arabia. find
  14. El-Sadany, T and M Hashish. (1988) Semi-automatic vowelization of arabic verbs. In 10th NC Conference, Jeddah, Saudi Arabia. find
  15. Manning, Chris and Hinrich Schutze. (1999) Foundations of statistical natural language processing. Massachusetts Institute of Technology Press - Library of Congress Cataloging in publication Information. find
  16. Deltour, Amelie. (2003) Methodes statistiques pour la voyellisation des texts arabes. Master’s thesis, ENSIMAG-Karlsruhe University. find
  17. Mansour, Alghamdi, Muhammad Khursheed, Mustafa Elshafei, Fayz Alhargan, Muhammed Alkanhal, Abu Aus Alshamsan, Saad Alqahtani, Syed Zeeshan Muzaffar, Yasser Altowim, Adnan Yusuf, and Husni Almuhtasib. 2006. Automatic arabic text diacritizer-final report ci 25 02. Technical report, KING ABDUL AZIZ CITY FOR SCIENCE AND TECHNOLOGY KACST. find
  18. Rashwan, Mohsen, Mohammad Al-Badrashiny, Mohamed Attia, and Sherif M. Abdou. 2009. A hybrid system for automatic arabic diacritization. In Natural Language Processing and Knowledge Engineering. NLP-KE 2009 Cairo, Egypt,, pages 1–8. find
  19. Buckwalter, Tim. 2004. Arabic morphological analyzer version 2.0 - ldc2004l02. In Linguistic Data Consortium, University of Pennsylvania, 2002. LDC Cat alog No.: LDC2004L02, ISBN 1-58563-324-0. find
  20. Abbas, Mourad and Kamel Smaili. 2005. Comparison of topic identification methods for Arabic language. In the International conference RANLP05 Recent Advances in Natural Language Processing, Borovets Bulgary, pages 21–23. find
  21. Rafalovitch, Alexandre and Robert Dale. 2009. United nations general assembly resolutions: a six-language parallel corpus. In Proceedings of the MT Summit XII, Ottawa, Canada,, pages 292–299. find
  22. Atiyya, Muhammad, Khalid Choukri, and Mustafa Yaseen. 2005. Specifications of the Arabic written corpus produced within the nemlar project. Technical report, NEMLAR, Center for Sprogteknologi. find
  23. Neuhoff, D.L. 1975. The viterbi algorithm as an aid in text recognition. IEEE Transaction on Information Theory, pages 222–226. find
  24. Hifni, Yasser. 2012. Smoothing techniques for arabic diacritics restoration. In Proceedings of the Twelfth Conference on Language Engineering (ESOLEC’12). Authors Mohamed Ould Abdallahi Ould Bebah Researcher at Doha Institute for Graduate Studies since 2013. "Doctorat" in Computer Sciences, Mohamed I University, Oujda, Morocco, 2013. "DESA" in "Numerical Analysis, Computer science and Signal Processing" from Mohamed I University, 2005. Member of Arabic NLP unit, LaRI Laboratory, Mohamed I University since 2005. Member of Language Studies unit at the Center of Social and Human Studies and Researches (CERHSO) in Oujda since 2005. Member of Arabic Language Engineering Society in Morocco (ALESM) since 2012. Amine CHENNOUFI Master in Computer Sciences from Mohamed I University, Oujda, Morocco 2010. Engineering degree in Meteorology from the National School of Meteorology ENM in Toulouse in France since 1994. Since January 2011, He prepares his PhD thesis in Arabic Natural Language Processing within the Computer Research Laboratory (LaRI). His research interests are especially in Automatic vowelization of Arabic language. Professionally he is the responsible of Meteorological Centre of Oujda Airport. Azzeddine Mazroui "Doctorat d’Etat" in Numerical Analysis, University Mohammed I Morocco, 2000. PHD in Probability and Statistics, Pierre & Marie Curie University France, 1993. Professor of mathematics and Computer Sciences in University Mohammed I. Member of Computer Research Laboratory (LaRI). Director of the ANLP unit in the LaRI laboratory. Abdelhak Lakhouaja "Doctorat d’Etat" in Computer Sciences, University Mohammed I Morocco, 2000. Professor of Computer Sciences in University Mohammed I. Member of Computer Research Laboratory (LaRI). Cofounder of the ANLP unit in the LaRI laboratory. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bebah-2014-hybrid-approaches-automatic-vowelization-arabic/