📄 Automatic Diacritization of Arabic for Acoustic Modeling in Speech Recognition

Summary

Most Arabic speech transcripts are written in plain script, which leaves out short vowels, consonant doubling (shadda) and nunation (tanween). Acoustic models trained on such transcripts cannot learn the unwritten sounds well. Vergyri and Kirchhoff study how to restore these diacritics automatically so that cheap script-only data can be used to train recognizers. Unlike commercial text-only diacritizers, their methods use the audio itself. They compare three procedures, each adding fewer constraints than the last:

• acoustics plus morphological analyses plus contextual (tagger) scores;
• acoustics plus morphology only;
• acoustics alone, which allows any diacritic at any position.

Each procedure is scored against manual diacritization. Each is also tested by adding about 40 hours of automatically diacritized MSA broadcast news (FBIS) to Egyptian CallHome training data. More linguistic knowledge gives lower diacritization error. However, recognition word error rates hardly differ between the three methods. This suggests that even fairly noisy automatic diacritization (around 23% error) can still help acoustic modeling.

Key points

• Data: the FBIS MSA radio corpus has about 40 h and 240K words, with 54K script forms averaging 2.5 diacritizations each. The LDC CallHome ECA corpus has about 20 h of training data (160K words) and 6 h of test data, with both script and romanized transcriptions.
• In earlier work, a commercial MSA diacritizer had a 9–28% diacritization error rate on MSA but 48% on Egyptian text. Training on script forms instead of romanized forms raised WER by almost 10% relative.
• The simplified DECIPHER-based baseline scored 56.1% WER on dev96 and 42.7% on eval03. ROVER combination with the CH+FBIS systems lowered these to about 55.2–55.7% and 41.6–41.8%.
• About 10% of the FBIS data could not be aligned and was dropped. The pooled models used 300 genones, weighted CallHome utterances twice as heavily as FBIS, and were then MAP-adapted to CallHome.
• In the morphology-constrained setups, the best diacritization path was re-chosen at every EM iteration. For the acoustics-only networks, which were too large to run efficiently, the diacritized references were realigned only once.

Interesting ideas

• I1. Let a second modality pick among candidate analyses: morphology proposes a small set of possible forms, and the acoustic signal chooses among them. Text-only ambiguity is resolved by evidence from outside the text.
• I2. Re-decide noisy labels inside the training loop rather than fixing them once: each EM iteration realigns the data with the improved model.
• I3. Evaluate on two levels, intrinsic label accuracy and downstream task impact, and accept that they may disagree. Here, large differences in diacritization error made no measurable difference to WER.
• I4. Pool plentiful mismatched data with a small clean in-domain set, weight the clean set higher, and finish with adaptation on the clean set alone. Final adaptation helps absorb noise in the extra data.
• I5. Combine a model trained on extra data with the baseline instead of replacing the baseline. In this study the extra-data system alone barely improved, but ROVER combination did.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): When lemma or tag ambiguity cannot be settled from the word alone, the analyser could keep a lattice of candidates and let later evidence select one. That evidence could be tree agreement or frame fit.
• I2 (learn): Silver labels from witness parsers could be re-scored by the parser being trained after each epoch, instead of being frozen at creation time.
• I3 (learn): Silver-data variants should be judged both by their agreement rate and by end-task scores such as LAS or QA accuracy. This avoids over-investing in label quality that downstream steps do not reward.
• I4 (tree): The gparse parser could be trained on large converted or silver treebanks with lower weights, then fine-tuned on gold UD English data to soften conversion errors.
• I5 (learn): A parser trained with silver data could serve as an extra voting witness beside the gold-trained parser, adding diversity even when it does not win on its own.