📄 Diacritic Recognition Performance in Arabic ASR

Summary

Arabic writing usually leaves out diacritics, the marks for short vowels, consonant doubling and case endings. Most speech corpora are therefore transcribed without them, and modern ASR systems seldom output them. Earlier work concluded that diacritized training transcripts make ASR worse, but it only reported overall error rates. It never checked how well the diacritics themselves were recognized compared with adding them afterwards using a text-only diacritizer.

The authors fine-tune Whisper (medium) and Wav2Vec XLS-R on ClArTTS, a roughly 10-hour single-speaker classical Arabic corpus with full manual diacritics. They compare three transcript conditions: UD (undiacritized, with diacritics restored after recognition by Shakkelha or D2), MD (manually diacritized) and AD (automatically diacritized). Diacritic quality is measured separately from base recognition through two metrics, coverage and precision.

Diacritics predicted by the ASR model beat text-based post-processing, most clearly when training used manual diacritics. Automatic diacritics gave mixed or equal results, and the cost in base WER/CER was small. This suggests the audio signal carries vowel evidence that text-only restoration cannot use.

Key points

• Data: ClArTTS has about 10 h of training audio (9500 segments) and a 1 h test set (205 segments). The Arabic Speech Corpus test set (100 utterances) serves as out-of-domain data for diacritic evaluation only.
• Without fine-tuning, Whisper-medium is nearly diacritic-free:
  • on ClArTTS: WER 16.7%, CER 4.8%, diacritic coverage 1.9%;
  • on ASC: WER 53.4% and coverage 1.0%.
• Metrics:
  • WER/CER are reported both with and without diacritics.
  • Coverage is the number of diacritic marks per alphabetic character.
  • Precision is computed on matching words and is reported with and without case endings.
• The text diacritizers (Shakkelha, D2) are trained on Tashkeela. Retraining them on the ClArTTS transcripts made them worse, and their accuracy dropped noticeably even when given gold speech transcripts as input.
• MD fine-tuning gave the best diacritization, including on the out-of-domain ASC set. The underlying WER/CER rose only slightly.

Interesting ideas

• I1. Split a combined error into separate parts: base recognition error ignoring the extra layer, how often the layer is emitted (coverage), and how correct it is when emitted (precision). — For Mova (tokens): Score morphological features and lemmas with separate coverage and precision on correctly tokenized words, so tokenizer mistakes do not hide tagger quality.
• I2. When the prediction or the reference lacks a mark, the case counts only toward coverage, never as a precision error, so omissions are not penalized twice. — For Mova (learn): Evaluation of witness parsers could separate abstentions from wrong answers in the same way, making silver-data agreement scores easier to interpret.
• I3. Predicting an annotation inside the model that sees the richer input works better than restoring it later with a pipeline step that sees only the reduced output. — For Mova (tree): Predict UD features or enhanced relations jointly in gparse rather than with a post-hoc rule pass, while keeping each prediction traceable to explain it.
• I4. Training on manual, automatic and absent labels as three controlled conditions shows directly what silver labels cost compared with gold ones. — For Mova (learn): Run the same three-way comparison (gold treebank, silver agreed parses, no extra layer) before trusting LLM-teacher or witness-produced data.
• I5. Feeding a text tool gold input from a new domain shows how much of its error is domain shift rather than upstream noise. — For Mova (answer): Give the fable and word-problem readers gold UD trees to separate parser errors from reasoning errors.