Elamur Research⌕
☰
home / papers / 2009 / A Hybrid Approach for Building Arabic Diacritizer

A Hybrid Approach for Building Arabic Diacritizer

Paper · 2009 · Khaled Shaalan, Hitham M. Abo Bakr and Ibrahim Ziedan

Summary

Arabic is normally written without short vowels and other diacritic marks, so many written forms map to several readings. This complicates speech synthesis, recognition and text analysis. The authors build a diacritizer that splits the task in two. Word-internal marks are treated as a morphological problem. The final-letter case ending is treated as a syntactic one.

For internal marks, three components vote:

  • an unambiguous-entry lookup in an Arabic lexicon;
  • a database of frequent diacritized two-word expressions;
  • an SVM tokenizer/POS tagger whose tags pick among candidates from a modified Buckwalter analyzer.

Case endings are then predicted by a separate SVM. It uses POS, base-phrase chunk and positional features, trained on tags extracted from the Penn Arabic Treebank (ATB). After feature and rule refinements, the full system reaches 11.795% WER and about 3.245% DER. That compares favourably with the figures reported for MADA-D. The result suggests that a fairly simple hybrid design can rival heavier statistical systems.

Key points

  • The final pipeline LR+DB+SVM+CE+SHR scores 11.795% WER and roughly 3.245% DER. The previously reported MADA-D numbers are 14.9% WER and 4.8% DER.
  • Case-ending accuracy rose from 95.35% (CE1, window −3/+3) to 96.57% (CE2). The gain came from adding last-character and last-two-character features.
  • In ablations (Table 9), WER falls from 35.81% for LR+DB to 17.31% once the CE model is added. With the "accepts any" (AA) fallback it reaches 16.66%.
  • ArabicSVMTools gave a better internal-only result (12.79% WER / 9.94% DER) than the authors' own tagger (16.28% / 11.36%). However, their tagger runs about 4× faster because it uses fewer features.
  • Buckwalter's analyzer was modified by deleting case-ending diacritics from its suffix table. This shrinks overgeneration: one word that had 7 analyses differing only by case ending now has a single internal candidate.

Interesting ideas

  • I1. Separate the ambiguity that belongs to the word from the ambiguity that belongs to its sentence role, and strip the latter out of the morphological analyzer so it stops multiplying candidates.
  • I2. Combine several solvers by agreement first, and fall back to a fixed trust order (exact lookup > collocation memory > statistical model) only when they disagree.
  • I3. Treat a lexicon hit with exactly one candidate as final and skip the other components entirely.
  • I4. Cheap orthographic cues such as the last one or two characters noticeably improved both tagging and case prediction. They also removed grammatically impossible outputs.
  • I5. Report error rates with and without the hardest sub-problem (case endings) to show where the errors actually come from.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): the en morphology could withhold context-dependent readings (e.g. syntactic function of an -ing form) and leave them to the parser, keeping the lemma/feature lattice small and its choices easier to explain.
  • I2 (learn): witness parsers producing silver data could use the same agree-then-rank rule, with a recorded reason for every chosen label.
  • I3 (tokens): unambiguous lexicon entries could short-circuit the tagger, saving compute and giving a trivially explainable decision path.
  • I4 (tokens): suffix features in the en tagger, and a check rejecting agreement-impossible combinations, could cut absurd morphological tags at little cost.
  • I5 (learn): evaluations could be split by component, e.g. scoring with and without attachment-heavy relations, so regressions are traced to the stage that caused them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓British University in DubaiDubai · United Arab Emirates11 works · 589 citations🎓Zagazig UniversityZagazig · Egypt4 works · 51 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Khaled Shaalan, Hitham M. Abo Bakr and Ibrahim Ziedan. 2009. A Hybrid Approach for Building Arabic Diacritizer. Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{shaalan2009hybrid,
  title = {{A Hybrid Approach for Building Arabic Diacritizer}},
  author = {Khaled Shaalan, Hitham M. Abo Bakr and Ibrahim Ziedan},
  booktitle = {Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages},
  year = {2009},
  doi = {10.3115/1621774.1621780},
  url = {https://doi.org/10.3115/1621774.1621780}
}

References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abo Bakr H. M. , Shaalan K., Ziedan I., 2008, "A Statistical Method for Adding Case Ending Diacritics for Arabic Text", The Eighth Conference on Language Engineering, ESOLEC’2008, Page 225-234, Cairo, Egypt,Deceber 17-18 2008. find
  2. Ananthakrishnan, Narayanan S., and Bangalore S., (2005), “Automatic diacritization of arabic transcripts for asr”. In Proceedings of ICON-05, Kanpur, India. find
  3. Buckwalter T., (2002). Buckwalter Arabic morphological analyzer version 1.0. Technical report, Linguistic Data Consortium, LDC2002L49 and ISBN 1-58563- 257-0. find
  4. Cristianini N. and Taylor J.S., (2000), “An Introduction to Support Vector Machines and Other Kernel-based Learning Methods”, The Press Syndicate of the University of Cambridge, Cambridge, United Kingdom. ↗
  5. Diab M., Hacioglu K., and Jurafsky D., (2004), "Automatic Tagging of Arabic Text: From Raw Text to Base Phrase Chunks," In Proc. of HLT/NAACL 2004, Boston.
  6. Diab M., Hacioglu K., and Jurafsky D.,(2007), “Arabic Computational Morphology Knowledge-based and Empirical Methods” - Chapter 7 “Automatic Processing of Modern Standard Arabic Text”,ISBN: 978-1-4020-6046-5, SpringerLink. find
  7. El-Imam Y., (2003). Phonetization of Arabic: rules and algorithms. Computer Speech and Language, 18:339– 373. ↗
  8. El-Sadany T. and Hashish M., (1988). Semi-automatic vowelization of Arabic verbs. In 10th NC Conference, Jeddah, Saudi Arabia. find
  9. Elshafei M., Al-Muhtaseb H., and Alghamdi M., (2006), “Statistical Methods for Automatic Diacritization of Arabic Text”. The Saudi 18th National Computer Conference. Riyadh. 18: 301-306. find
  10. Emam O. and Fisher V. (2004)., A hierarchical approach for the statistical vowelization of Arabic text. Technical report, IBM patent filed, DE9-2004-0006, US patent application US2005/0192809 A1. find
  11. Gal Y., (2002). An HMM approach to vowel restoration in Arabic and Hebrew. In ACL-02 Workshop on Computational Approaches to Semitic Languages.
  12. Habash N. and Rambow O., (2007), “Arabic Diacritization through Full Morphological Tagging”, In Proceedings of the North American chapter of the Association for Computational Linguistics (NAACL), Rochester, New York.
  13. Hearst M. A., (1998), "Support Vector Machines," IEEE Intelligent Systems, vol. 13, no. 4, pp. 18-28, Jul/Aug, 1998. ↗
  14. Kirchhoff K. and Vergyri D., (2005). Cross-dialectal data sharing for acoustic modeling in Arabic speech recognition. Speech Communication, 46(1):37–51, May. ↗
  15. Kudo T. and Matsumoto Y., (2003), " Fast methods for kernel-based text analysis," In Proceedings of the 41st Annual Meeting on Association For Computational Linguistics - Volume 1 (Sapporo, Japan, July 07 - 12, 2003). Annual Meeting of the ACL. Association for Computational Linguistics, Morristown.
  16. Maamouri, M., Bies, A. & Buckwalter, T. (2004). The Penn Arabic treebank: Building a largescale annotated Arabic corpus. In NEMLAR Conference on Arabic Language Resources and Tools, Cairo, Egypt. find
  17. Maamouri M., Bies A., Kulick S., (2008), "Enhanced Annotation and Parsing of the Arabic Treebank"; INFOS 2008, Cairo, Egypt, March 27-29, 2008. find
  18. Sang E. and Buchholz S., (2000), ” Introduction to the CoNLL-2000 Shared Task: Chunking”, Proceeding of CoNLL-2000 and LLL-2000,Page 127-132, Lisbon,Portugal.
  19. Zitouni I., Sorensen J. S., and Sarikaya R., (2006), “Maximum entropy based restoration of Arabic diacritics”. In Proceedings of ACL’06.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shaalan-2009-hybrid-approach-building-arabic-diacritizer/