📄 Hafs2Vec: A System for the IqraEval Arabic and Qur'anic Phoneme-level Pronunciation Assessment

Summary

Hafs2Vec is an entry to the Iqra'Eval 2025 shared task, which asks systems to find mispronounced phonemes in Modern Standard Arabic and Qur'anic speech. The author starts from a mismatch between two kinds of data. Learner recordings contain realistic mistakes but vary in acoustic quality. Professional recitations are clean but contain no errors.

To get the benefits of both, the official ~79 h Iqra'Eval training set was pooled with ~94 h of verses from 28 professional reciters. Standard MSA phonemizers ignore Tajweed, so the reciter audio was labelled by a custom phonemizer that applies Tajweed articulation rules and maps its output onto the official phoneme set. A fine-tuned wav2vec2-xls-r-1b with greedy decoding reached 46.50% F1 and 79.20% recall on QuranMB. That is close to the winning system (47.26% F1) and far above the baseline (28.64%).

The work shows that clean reference audio with rule-generated labels can strengthen error detection. The resulting system favours recall, which suits learner feedback.

Key points

• Data: reciter verses come from EveryAyah and the Qur'anic Universal Library, kept only if 10 s or shorter (~54k utterances). The Iqra'Eval set (~74k utterances) is CommonVoice Arabic plus Qur'anic material. The phonemizer handles Idgham, Iqlab, Ikhfaa and Qalqala.
• Training: 15 epochs, effective batch size 352, AdamW, learning rate 3e-5, warm-up ratio 0.1. The 1B xls-r model beat its 300M variant by ~1.3 absolute PER and also beat HuBERT and WavLM models of similar size.
• Dev PER, with substitutions as the most common error type:
  • Iqra'Eval overall: 7.69%
  • Qur'anic subset: 3.88%
  • MSA subset: 9.50%
  • held-out professional reciters: 1.55%
• Test (QuranMB): Hafs2Vec had the lowest false acceptance rate of the compared systems (20.80%) and a TAR of 88.40%. Its precision (32.92%) was below the winner's (37.13%). Across 1,643 test utterances, Tajweed phonemes almost never appeared wrongly in the output.
• Error analysis:
  • Vowels cause ~63% of misrecognitions, with a category PER of 11.08%.
  • Shaddah phonemes are rare but error-prone (category PER 8.45%). The rarest, "EE" and "HH", occur only 90 and 94 times in training.

Interesting ideas

• I1. The model is trained on a mix of clean, error-free reference data and noisy, error-rich learner data, then checked for one specific risk: features from the clean domain leaking into predictions for the other domain.
• I2. When a corpus has no annotation, a context-sensitive, rule-based labeller produces the labels and then maps them onto the target label inventory with a few pre- and post-processing rules.
• I3. Errors are reported per category in two ways: the error rate inside the category and the category's share of the total error. This separates frequent-but-easy classes from rare-but-hard ones.
• I4. Since near-identical vowel variants dominate confusions, the paper proposes collapsing them into fewer, broader classes. It also suggests oversampling, balanced batches or a weighted loss for rare classes.
• I5. The author argues for separate dev sets per style, accepting a small loss in one domain for a large gain in another.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Silver training data could mix edited prose with messy learner text, and a negative-control test could check that edited-text constructions are not hallucinated in plain inputs.
• I2 (tree): The same pattern fits deterministic converters that turn other treebank dialects into Mova's UD flavour, which would cheaply enlarge training data.
• I3 (tree): Parser errors could be broken down by relation family with both numbers, showing whether rare relations or common ones deserve the next effort.
• I4 (tokens): Reporting tagger accuracy at both fine and coarse tag levels, and oversampling sentences with rare morphological features, could show which distinctions the tagger truly cannot learn.
• I5 (answer): Fables, school tests and maths problems should each have their own evaluation set, so a gain in one genre does not hide a regression in another.