📄 Diacritization for Real-World Arabic Texts

Summary

Arabic is usually written without short vowels, so one written form can correspond to words with different meanings and parts of speech. Restoring these diacritics helps downstream tasks such as machine translation. The authors treat diacritization as a character-level classification task. For each letter, the system decides which short vowel follows it: none, a vowel, or a shadda combination. The features are a window of neighboring characters plus up to five surrounding words, and the classifier is the memory-based learner TiMBL. Trained on the Penn Arabic Treebank (ATB), the system is competitive with Zitouni et al. even though it uses no linguistic preprocessing. Performance drops on real web text, mainly because sites such as AFP write the hamza inconsistently or omit it, while the treebank spells it carefully. Adding a separate TiMBL-based hamza restoration step before diacritization lowers WER from 9.20% to 7.38% on treebank data. On four human-rated AFP news stories, it lowers WER from 7.96% to 5.93%. The main lesson is that orthographic gaps between clean training corpora and real text must be bridged explicitly.

Key points

• Task definition: each character gets a class (short vowel, shadda plus vowel, or no vowel). Case endings, mood endings, nunation and hamza are left out of the diacritization target.
• Features and model: the features are the focus character, 5 characters on each side, and 5 whole words on each side. The model is TiMBL (k-nearest-neighbor, feature weighting by gain ratio / information gain).
• Data:
  • ATB parts 1 and 2 (about 170k AFP words plus about 160k Ummah words), evaluated with 10-fold CV.
  • ATB part 3 (An Nahar, about 288k train / 52k devtest), used for comparison with Zitouni et al.
  • An AFP-only 90/10 split, used for the hamza experiments.
• Treebank results with restored hamza: CER 2.54% and WER 7.38% with punctuation; CER 2.83% and WER 13.66% without punctuation. These are close to the setting where both training and test data have perfect hamza.
• Real-world test: four AFP stories from 2 Dec 2008 (1,332 words), checked by two native-speaker raters with an arbitrator. Hamza restoration cut WER by about 2 points, an error reduction of 25.5%.

Interesting ideas

• I1. When clean training data and messy input differ in one orthographic feature, train a dedicated restorer for that feature on the same corpus and run it first, instead of degrading the main model.
• I2. Turning a word-level restoration problem into per-character decisions avoids needing a dictionary entry for every word. Feeding whole neighboring words as symbolic features still gives the classifier lexical context.
• I3. The authors separate in-vitro testing (same treebank) from in-vivo testing (real web pages), and they simulate the mismatch by stripping hamza from test data while keeping it in training.
• I4. Memory-based learning keeps irregular examples instead of smoothing them away, and its feature weights show which context matters most. This fits a system that must justify its decisions.
• I5. The human raters found errors that could only be fixed with discourse knowledge: a noun phrase needed dual marking because the prisoner count is revealed only at the end of the story.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): A small restorer for apostrophes, casing or hyphenation in noisy English could run before the tagger, so the tagger keeps seeing treebank-like input.
• I2 (tokens): This could help recover the spelling or morphology of unknown or misspelled words, with per-character choices that can be inspected.
• I3 (learn): Deliberately corrupted copies of held-out treebank data (no punctuation, lowercased) could serve as controls that show how much each normalization step really helps.
• I4 (tokens): A nearest-neighbor lemma or morphology fallback could cite the stored training instance behind each choice as its explanation.
• I5 (world): Entity counts tracked in the world model could be fed back to check or revise number agreement made earlier in the text.