Elamur Research⌕
☰
home / papers / 2025 / Phoneme-level mispronunciation detection in Quranic recitation using ShallowTransformer

Phoneme-level mispronunciation detection in Quranic recitation using ShallowTransformer

Paper · 2025 · Mohamed Nadhir Daoud and Mohamed Anouar Ben Messaoud

Summary

Errors in Qur'anic recitation can change the meaning of the text, so learners need automatic feedback at the level of individual phonemes. The authors built ShallowTransformer for the Iqra'Eval shared task. It runs a small Transformer encoder on frozen features from a wav2vec 2.0 model adapted to Arabic. The model is trained with CTC loss, so it predicts the phoneme sequence that was actually spoken without needing frame-level alignments. Mispronunciations are then found by comparing that sequence with the canonical one.

Training used CMV-Ar, a Common Voice–based Arabic corpus extended with Qur'anic recitation. Testing used QuranMB.v2, whose labels are hidden behind a scoring API. The model detected errors with very high recall. Its F1 was higher than the published BiLSTM systems reported on QuranMB.v1, but lower than the two official challenge baselines, mainly because its precision was weaker.

The work matters because it shows that a compact model, small enough for phones or embedded devices, can be competitive on a newly opened Arabic pronunciation benchmark. Such open benchmarks were missing until recently.

Key points

  • Data: CMV-Ar provides 71,391 training utterances (about 79 h) and 2,588 dev utterances (3.33 h). QuranMB.v2 contains 98 utterances from 18 native speakers (about 2 h). Phoneme labels follow the Halabi & Wald dictionary.
  • Features: the iqra_wav2vec2_base checkpoint is loaded through S3PRL. Its 13 layer outputs are combined by a learned weighted sum, then projected from 768 to 256 dimensions.
  • Model: 3 Transformer layers with 4 heads each, a 1024-dim feed-forward block, and sinusoidal positions. Decoding is greedy CTC.
  • Training: Adam at 3e-4 with cosine decay to 1.5e-5, dropout 0.15, gradient clipping at 1.0, and 15 epochs (3 of them warmup) with batch size 64.
  • Results: recall 84.56, precision 22.05, F1 34.94.
    • The best BiLSTM baseline (mHuBERT) reached F1 28.64. That figure comes from QuranMB.v1, and the authors themselves caution that the two versions are not identical.
    • The Iqra'Eval baselines reached F1 44.14 and 40.42.

Interesting ideas

  • I1. Instead of picking one layer of a frozen pretrained encoder, the model learns a weighted mix of all its layers and trains only a thin head on top, which keeps the system cheap and lets the data decide which depth matters.
  • I2. CTC lets a model learn to emit a short symbol sequence from a long input without ever being told which input positions produce which symbol.
  • I3. Errors are detected by comparing the predicted realization with the canonical form and sorting every position into true/false accept/reject. This keeps "missed errors" and "false alarms" as separate, visible quantities.
  • I4. The test labels are kept hidden and scores come only from an external API, which prevents tuning on the test set.
  • I5. The authors openly warn that their headline gain compares two benchmark versions with undocumented differences.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could mix all hidden layers of its frozen transformer before the arc/label scorers, and inspecting the learned weights would show which depth carries syntax.
  • I2 (tokens): morphological segmentation or lemma-affix labelling could be trained from word–analysis pairs without hand-aligned character spans.
  • I3 (learn): self-induced repair rules could be scored the same way, counting separately how often a rule wrongly rejects a correct parse and how often it lets a wrong one through.
  • I4 (learn): a held-out school-test or fable QA set scored only by a sealed evaluator would keep answer-stage numbers honest against overfitting.
  • I5 (learn): every reported metric should pin the exact dataset and treebank version, so that UD release changes are never mistaken for parser improvements.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tunis El Manar UniversityTunis · Tunisia2 works · 245 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Mohamed Nadhir Daoud and Mohamed Anouar Ben Messaoud. 2025. Phoneme-level mispronunciation detection in Quranic recitation using ShallowTransformer. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.63

@inproceedings{daoud2025phoneme,
  title = {{Phoneme-level mispronunciation detection in Quranic recitation using ShallowTransformer}},
  author = {Mohamed Nadhir Daoud and Mohamed Anouar Ben Messaoud},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.63},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.63}
}

References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammed Algabri, Hassan Mathkour, Mansour Alsulaiman, and Mohamed A Bencherif. 2022. Mispronunciation detection and diagnosis with articulatory-level feedback generation for non-native arabic speech. Mathematics, 10(15):2727. ↗
  2. Yousef Ajami Alotaibi and Ghulam Muhammad. 2010. Study on pharyngeal and uvular consonants in foreign accented arabic for asr. Computer Speech & Language, 24(2):219–231. ↗
  3. Norah Alrashoudi, Hend Al-Khalifa, and Yousef Alotaibi. 2025. Improving mispronunciation detection and diagnosis for non-native learners of the arabic language. Discover Computing, 28(1):1. ↗
  4. Salah A Aly, Abdelrahman Salah, and Hesham M Er-aqi. 2021. Asmdd: Arabic speech mispronunciation detection dataset. arXiv preprint arXiv:2111.01136. ↗
  5. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670. ↗
  6. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.
  7. Fatma Zahra Besdouri, Inès Zribi, and Lamia Hadrich Belguith. 2024. Arabic automatic speech recognition: challenges and progress. Speech Communication, 163:103110. ↗
  8. Yassine El Kheir, Amit Meghanani, Hawau Olamide Toyin, Nada Almarwani, Omnia Ibrahim, Youssef Elshahawy, Mostafa Shahin, and Ahmed Ali. 2025. Iqra’eval: A shared task on qur’anic pronunciation assessment. In Proceedings of the Third Arabic Natural Language Processing Conference. Association for Computational Linguistics.
  9. John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett. 1993. Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1. NASA STI/Recon technical report n, 93:27403. find
  10. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369–376. find
  11. Nawar Halabi and Mike Wald. 2016. Phonetic inventory for an Arabic speech corpus. pages 734–738, Portorož, Slovenia. European Language Resources Association (ELRA).
  12. Alissa M Harrison, Wai-Kit Lo, Xiaojun Qian, and Helen Meng. 2009. Implementation of an extended recognition network for mispronunciation detection and diagnosis in computer-assisted pronunciation training. In SLaTE, pages 45–48. find
  13. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460. ↗
  14. Shao-Wei Fan Jiang, Bi-Cheng Yan, Tien-Hong Lo, Fu-An Chao, and Berlin Chen. 2021. Towards robust mispronunciation detection and diagnosis for l2 english learners with accent-modulating methods. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1065–1070. IEEE.
  15. Yassine El Kheir, Omnia Ibrahim, Amit Meghanani, Nada Almarwani, Hawau Olamide Toyin, Sadeen Alharbi, Modar Alfadly, Lamya Alkanhal, Ibrahim Selim, Shehab Elbatal, Salima Mdhaffar, Thomas Hain, Yasser Hifny, Mostafa Shahin, and Ahmed Ali. 2025. Towards a unified benchmark for arabic pronunciation assessment: Quranic recitation as case study. Preprint, arXiv:2506.07722. ↗
  16. Wai-Kim Leung, Xunying Liu, and Helen Meng. 2019. Cnn-rnn-ctc based end-to-end mispronunciation detection and diagnosis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8132–8136. IEEE. find
  17. Kun Li, Xiaojun Qian, and Helen Meng. 2016. Mispronunciation detection and diagnosis in l2 english speech using multidistribution deep neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(1):193–207. ↗
  18. Helen Meng, Yuen Yee Lo, Lan Wang, and Wing Yiu Lau. 2007. Deriving salient learners’ mispronunciations from cross-language phonological comparisons. In 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU), pages 437– 442. IEEE. find
  19. Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, and 1 others. 2022. Self-supervised speech representation learning: A review. IEEE Journal of Selected Topics in Signal Processing, 16(6):1179– 1210. find
  20. Faria Nazir, Muhammad Nadeem Majeed, Mustansar Ali Ghazanfar, and Muazzam Maqsood. 2019. Mispronunciation detection using deep convolutional neural network features and transfer learning-based model for arabic phonemes. IEEE Access, 7:52589– 52608. ↗
  21. Ambra Neri, Ornella Mich, Matteo Gerosa, and Diego Giuliani. 2008. The effectiveness of computer assisted pronunciation training for foreign language learning by children. Computer Assisted Language Learning, 21(5):393–408. ↗
  22. Linkai Peng, Kaiqi Fu, Binghuai Lin, Dengfeng Ke, and Jinsong Zhang. 2021. A study on fine-tuning wav2vec2. 0 model for the task of mispronunciation detection and diagnosis. In Interspeech, volume 2021, pages 4448–4452. find
  23. Xiaojun Qian, Helen Meng, and Frank Soong. 2010. Capturing l2 segmental mispronunciations with joint-sequence models in computer-aided pronunciation training (capt). In 2010 7th International Symposium on Chinese Spoken Language Processing, pages 84– 88. IEEE. find
  24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  25. Silke M Witt and Steve J Young. 2000. Phone-level pronunciation scoring and assessment for interactive language learning. Speech communication, 30(2-3):95–108. ↗
  26. Minglin Wu, Kun Li, Wai-Kim Leung, and Helen Meng. 2021. Transformer based end-to-end mispronunciation detection and diagnosis. In Interspeech, pages 3954–3958. find
  27. Shu-wen Yang, Heng-Jui Chang, Zili Huang, Andy T Liu, Cheng-I Lai, Haibin Wu, Jiatong Shi, Xuankai Chang, Hsiang-Sheng Tsai, Wen-Chin Huang, and 1 others. 2024. A large-scale evaluation of speech foundation models. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:2884–2899. find
  28. Nishmia Ziafat, Hafiz Farooq Ahmad, Iram Fatima, Muhammad Zia, Abdulaziz Alhumam, and Kashif Rajpoot. 2021. Correct pronunciation detection of the arabic alphabet using deep learning. Applied Sciences, 11(6):2508. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/daoud-2025-phoneme-level-mispronunciation-detection-quranic/