📄 A Layered Language Model based Hybrid Approach to Automatic Full Diacritization of Arabic

Summary

Arabic is normally written without most short-vowel and related marks, which leaves many words ambiguous for both readers and downstream tools such as POS tagging, text-to-speech and MT. The authors restore full diacritization (lexical plus word-final syntactic marks) for MSA. Their method chains three diacritized language models: one over surface words, one over prefix/stem/suffix morphemes, and one over characters with their diacritics. A lattice search, Viterbi or A*, picks the best candidate in context. The word model runs first. Its output is then checked against the SAMA morphological analyzer, and the morpheme model resolves the remaining choices. The character model handles only words that are still unknown. They train and test on the LDC Arabic Treebank Part 3 split used by earlier work, but they also carve out a separate development set. On the held-out TEST set the system reaches 10.87% WER for full diacritization and 3.0% WER when case endings are ignored. It leads on the lexical level, which shows that a carefully layered n-gram approach can match or beat neural systems of its time.

Key points

• The data is LDC2004T11 (Arabic Treebank Part 3 v1.0, Annahar News Text). It has 600 documents and 340,281 words. TEST is the chronologically last 15%, about 52K words in 90 documents. The original training part was split into TRAIN (~259K) and DEV (~29K).
• n-gram orders are kept aligned across layers. One word corresponds to 3 morpheme grams and 4 character grams, so a word bigram maps to a 6-gram MLM and an 8-gram CLM.
• The best DEV configuration used Viterbi with 2-gram-equivalent models. It reached 6.11% WER (FULL) and 2.61% WER (MORPH).
• On TEST the system scores 10.87% WER for FULL and 3.0% WER for lexical-only diacritization. It beats all baselines at the MORPH level. At the FULL level, Abandah et al. (2015) do better (9.07% WER) because their system handles case endings better.
• The system is compared against five baselines: Belinkov and Glass (2015), Abandah et al. (2015), Rashwan et al. (2011), Habash and Rambow (2007) and Zitouni et al. (2006).

Interesting ideas

• I1. The system backs off by granularity: whole-word evidence is trusted first, morpheme evidence comes next, and a character model is used only for words still unresolved, so each decision traces to one layer.
• I2. A rule-based morphological analyzer serves as a filter on statistical output: only analyses consistent with the LM's choice are kept, while unknown words keep every candidate.
• I3. Unknown words do not split the sentence into separate pieces; they stay in the lattice with an unknown-word probability, so the neighbouring words still give context across them.
• I4. The authors noticed that earlier work tuned on its own test set, so they cut a dev set out of training and touched TEST only once, with the final configuration.
• I5. Scores are reported twice, with and without word-final syntactic marks, which separates lexical errors from grammar-driven ones.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): lemma and morphology lookup for unseen English words could follow the same order: lexicon, then affix analysis, then a character model. Each output would record which layer produced it.
• I2 (tokens): the morphology analyzer could prune its readings against the tagger's choice. Untagged or unknown tokens would keep their full candidate set for later stages to decide.
• I3 (tree): the parser and tagger could treat OOV tokens as placeholders inside one search instead of breaking the input. This keeps attachment decisions around rare words informed by the whole sentence.
• I4 (learn): every repair rule or teacher-tier threshold should be tuned on a dev slice that is separate from the reported evaluation set, so scores are not inflated.
• I5 (learn): parser and reader evaluations could likewise be split by error source, such as lemma/POS errors versus attachment or role errors. This shows which stage actually needs work.