📄 Metapseud at Iqra'Eval: Domain Adaptation with Multi-Stage Fine-Tuning for Phoneme-Level Qur'anic Mispronunciation Detection

Summary

The paper describes Metapseud, a submission to the Iqra'Eval shared task at ARABIC-NLP 2025. The task is to spot pronunciation mistakes in Qur'anic recitation at the level of individual phonemes. Qur'anic Arabic follows strict Tajweed rules and differs from both MSA and dialects, so general Arabic ASR models do not fit it well. The author starts from wav2vec2-large-xlsr-53-arabic and adapts it in two CTC fine-tuning steps, ordered as a curriculum:

• Stage 1: a large, varied corpus of recitations, Tarteel-ai EA-DI. The author phonemized it and released it on Hugging Face.
• Stage 2: the official Iqra train set, about 79 hours of MSA mixed with Qur'anic speech.

Phoneme sequences are produced with CTC beam search through PyCTCDecode. Phoneme error rate falls to about 0.21 on the development data. On the blind open test, beam search raises accuracy from 76.9% to 82.1%, but F1 (42.36) remains below the stronger official baseline. The work is a simple, reproducible recipe for adapting speech models to a narrow, low-resource domain.

Key points

• The base model is wav2vec2-large-xlsr-53-arabic (Grosman), which was itself tuned on Common Voice 6.1 and the Arabic Speech Corpus.

• PER is about 0.54 after Stage 1 on an EA-DI dev split and about 0.21 after Stage 2. The two figures come from different development sets.

• The splits are 79 h for training, 3.4 h for development, and about 2 h for a blind open test containing deliberate errors.

• Scores on the open test:

  System	F1	Precision	Recall	Accuracy
  Stage-2 + beam search	42.36	80.12	28.79	82.13
  Baseline 1	44.14	77.07	30.93	82.34

• Evaluation uses the hierarchical TA/TR/FA/FR scheme plus CD/ED diagnosis classes.

• A manual check of 100 dev samples found:

  • 11 perfect outputs;
  • beam search helped in 11 cases, repeated the same errors in 68 and made things worse in 10;
  • deletions were the most common error type;
  • error counts grew with utterance length.

Interesting ideas

• I1. Train in two steps: first on a big, varied corpus close to the domain, then on the small official annotated set, instead of jumping straight from a general model to the narrow task.
• I2. Training targets for the big corpus were generated automatically by a rule-based phonetizer, turning plain recordings into supervised data at scale.
• I3. The scoring keeps two questions apart: was an error correctly detected (TR versus FA/FR), and was it correctly diagnosed (CD versus ED).
• I4. Instead of only reporting an overall score gain, the author counted the samples where the decoder improved, kept the same errors, or got worse, which showed that beam search mostly repeats the base model's mistakes.
• I5. Errors were tied to utterance length and broken down by type (deletions dominated), which points to where the model fails rather than just how often.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could be pretrained on large silver-parsed English text and then fine-tuned on gold UD treebanks, which may give more stable training than gold-only fine-tuning.
• I2 (tokens): deterministic morphology and lemma rules could label large raw corpora as pretraining targets for the tagger before it is corrected on hand-annotated data.
• I3 (learn): self-induced repair rules could be scored the same way, separately measuring whether a flagged parse arc is really wrong and whether the proposed fix is the right one.
• I4 (learn): any reranker or beam over parses or frames should be reported with these per-sentence buckets, so that small average gains do not hide regressions.
• I5 (answer): wrong answers on school tests or word problems could be grouped by text length and failure type (a dropped entity versus a wrong value) to steer repairs to the right pipeline stage.