📄 ANPLers at IqraEval Shared task: Adapting Whisper-large-v3 as Speech-to-Phoneme for Qur'anic Recitation Mispronunciation Detection

Summary

The paper addresses phoneme-level mispronunciation detection in Qur'anic recitation for the IqraEval shared task. Word-level speech recognition cannot capture fine articulation and Tajweed errors, so the authors treat the problem as speech-to-phoneme (STP) transcription. Earlier systems mostly used encoder-only or CTC-style models. This team instead fine-tuned an encoder–decoder model, Whisper-large-v3, on the IqraEval training and evaluation splits. They added 68 phoneme symbols to Whisper's tokenizer and resized the embedding layer, so the decoder emits phoneme sequences directly from log-mel features.

On the test set, the system has high recall (0.7624) but poor precision (0.2045), giving an F1 of 0.3224. In practice, it flags many correctly pronounced phonemes as errors. A qualitative check shows that recordings with intended pronunciation differences sometimes get identical transcriptions. The authors also audit the dataset and report several quality problems.

The work matters as a documented baseline for encoder–decoder STP in this domain. It also gives a candid account of where such a system and its data fall short.

Key points

• Data and model: The system is Whisper-large-v3 (about 1550M parameters), chosen for its strong standing on an Arabic ASR leaderboard. It was trained on 73,990 IqraEval records, roughly 82.4 hours of audio, using only the audio and phoneme fields.

• Tokenizer change: The vocabulary was extended with 68 phoneme tokens taken from Halabi and Wald (2016), and the embedding matrix was resized to match.

• Training setup:

  • 3 × A100 80GB GPUs
  • batch size 4 per device
  • gradient accumulation over 3 steps
  • fp16
  • checkpoint selection by per-epoch WER

• Test results:

  Metric	Value
  F1	0.3224
  Precision	0.2045
  Recall	0.7624
  Accuracy	0.6894
  Correct rate	0.7682
  TA	0.7868
  FR	0.2132
  FA	0.2376
  CD	0.5418

• Dataset problems found:

  • truncated or cut audio
  • wrong vowel marking (tahreek) at sentence ends
  • non-canonical Tajweed in the reference recitations
  • irregular speaking rates
  • grammatical errors in the spoken content
  • transcripts misaligned with their audio

Interesting ideas

• I1. A pretrained sequence model can be repurposed for a new output inventory by adding the new symbols to its vocabulary and growing the embedding table, while keeping the learned encoder intact. This lets a large general model produce domain-specific units without starting from scratch.
• I2. Reporting the full accept/reject breakdown (TA, FR, FA, TR, CD) alongside F1 exposes asymmetries that a single score hides. Here it showed that the low F1 comes almost entirely from false rejections, which tells you exactly what to fix.
• I3. The authors used a minimal-pair probe: several recordings of the same verse with intended differences, checking whether the outputs actually diverge. This reveals whether a model is sensitive to the contrast or just echoes the most likely canonical form.
• I4. Before trusting the metrics, the team audited the training data and found misaligned references, truncated audio and incorrect gold forms. This frames weak scores partly as a data-integrity issue rather than purely a model issue.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could gain dedicated tokens for UD relation labels or dialect-conversion markers on top of a pretrained transformer, instead of decoding them indirectly.
• I2 (learn): evaluation could split parser or reader errors into false alarms versus misses per category, showing whether a repair rule over-fires or under-fires.
• I3 (learn): negative controls could feed near-identical sentences differing in one word or attachment and verify that trees and frames change exactly where they should, catching prior-driven "default" outputs.
• I4 (learn): silver treebank data agreed by witness parsers could be routinely screened for systematic label or alignment faults before training, so self-induced repair rules don't learn from corrupted gold.