Elamur Research⌕
☰
home / papers / 2025 / Iqra'Eval: A Shared Task on Qur'anic Pronunciation Assessment

Iqra'Eval: A Shared Task on Qur'anic Pronunciation Assessment

Paper · 2025 · Yassine El Kheir, Amit Meghanani, Hawau Olamide Toyin et al.

Summary

The paper describes Iqra'Eval 2025, the first shared task, which also serves as a publicly available benchmark, targeting the detection and diagnosis of mispronunciations (MDD) in Qur'anic recitation, with MSA-style reading as the case study. Arabic has hard contrasts, such as emphatic versus plain consonants and pharyngeal or uvular sounds, and recitation adds Tajweed rules. Despite this, the field had no shared evaluation. The organizers cast the task as phoneme recognition: given audio and its reference text, a system outputs the phonemes actually spoken, and errors are found by comparing that output with the canonical sequence. They provided:

  • a vowelized Common Voice subset for training;
  • TTS-generated speech, both correct and with injected errors;
  • the QuranMB test set of deliberately mispronounced verses;
  • a 62-phoneme inventory;
  • SSL baselines and a public Hugging Face leaderboard.

Of 29 registered teams, 11 were ranked and 5 had their papers accepted. Their systems mostly fine-tuned Whisper or wav2vec2-family models, often with synthetic or Qur'anic data. The benchmark gives future work on Arabic pronunciation tutoring a common reference point.

Key points

  • Training data comes from two sources:
    • an 82.37-hour CV-Ar subset of Common Voice 12.0 with fully vowelized transcripts, plus Qur'anic recitation samples;
    • 52 hours of TTS speech from seven single-speaker voices, half canonical and half with errors drawn from a phoneme confusion-pair matrix.
  • QuranMB, the test set, has 98 verses read by 18 native speakers (about 2.2 hours). Speakers produced scripted errors, and the recordings were checked by 3 linguistic annotators.
  • Scoring aligns three sequences: the annotated verbatim phones, the canonical phones and the predicted phones. Each phone is labelled TA, TR, FR or FA, and these labels yield FRR, FAR, DER, precision, recall and F1.
  • Baselines freeze an SSL encoder (mHuBERT or WavLM), combine its layers with a weighted sum in the SUPERB style, and feed the result to a 2-layer Bi-LSTM with CTC and greedy decoding.
  • The authors name several limitations: test speakers all speak the Saudi dialect, the errors are elicited rather than natural, and there is no children's corpus. They also built a crowdsourcing platform to collect more diverse data.

Interesting ideas

  • I1. Scoring against both the canonical reference and the human verbatim transcript separates two questions: whether a deviation was noticed (detection) and whether it was named correctly (diagnosis).
  • I2. Training data with errors is produced by corrupting clean inputs along a similarity-based confusion matrix, which gives controlled negative examples where real annotated errors are scarce.
  • I3. The test set uses scripted errors from a tool that highlights exactly what to alter, and the authors admit such errors may not match natural mistakes. This keeps the labels clean but shows the cost of the method.
  • I4. The baselines freeze a pretrained encoder, learn only a weighting over its layers and a small head, and so allow cheap, comparable probing of different backbones.
  • I5. The organizers released a tiny dummy sample set purely for checking formats and debugging submissions before full evaluation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when testing self-induced repair rules, label each token by original parse, gold and repaired output, so that "flagged the error" and "fixed it correctly" are measured apart.
  • I2 (learn): perturb sentences with known tagger or parser confusions (e.g., VBD/VBN, PP-attachment) to build negative controls that the pipeline must reject or repair.
  • I3 (answer): add deliberately altered fable or problem texts to the test suites, while keeping a separate set of naturally occurring failures to check that the gains carry over.
  • I4 (tree): gparse could compare transformer backbones by training only layer weights and a light biaffine head, which also shows which layers carry syntactic information.
  • I5 (learn): keep a minimal smoke-test corpus that every pipeline stage must pass in seconds, to catch interface breakage before costly training runs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SheffieldSheffield · United Kingdom113 works · 21,542 citations🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany77 works · 29,479 citations🎓Alexandria UniversityAlexandria · Egypt39 works · 287 citations🎓UNSW SydneySydney · Australia34 works · 1,509 citations🎓Taibah UniversityMedina · Saudi Arabia12 works · 64 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Yassine El Kheir, Amit Meghanani, Hawau Olamide Toyin, Nada Almarwani, Omnia Ibrahim, Yousseif Ahmed Elshahawy, Mostafa Shahin and Ahmed Ali. 2025. Iqra'Eval: A Shared Task on Qur'anic Pronunciation Assessment. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.61

@inproceedings{kheir2025iqra,
  title = {{Iqra'Eval: A Shared Task on Qur'anic Pronunciation Assessment}},
  author = {Yassine {El Kheir}, Amit Meghanani, Hawau Olamide Toyin, Nada Almarwani, Omnia Ibrahim, Yousseif Ahmed Elshahawy, Mostafa Shahin and Ahmed Ali},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.61},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.61}
}

References

24 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fadzil Ahmad, Saiful Zaimy Yahya, Zuraidi Saad, and Abdul Rahim Ahmad. 2018. Tajweed classification using artificial neural network. In 2018 International Conference on Smart Communications and Networking (SmartNets), pages 1–4. IEEE. find
  2. Ali M Alagrami and Maged M Eljazzar. 2020. Smar-tajweed automatic recognition of arabic quranic recitation rules. arXiv preprint arXiv:2101.04200. https://www.aiwinterschool.com/ ↗
  3. Mohamed Nadhir DAOUD and Mohamed Anouar BEN MESSAOUD. 2025. Phoneme-level mispronunciation detection in quranic recitation using shallowtransformer. In The Third Arabic Natural Language Processing Conference (Arabic-NLP 2025), Suzhou. Association for Computational Linguistics.
  4. Yassine El Kheir, Omnia Ibrahim, Amit Meghanani, Nada Almarwani, Hawau Toyin, Sadeen Alharbi, Modar Alfadly, Lamya Alkanhal, Ibrahim Selim, Shehab Elbatal, Salima Mdhaffar, Thomas Hain, Yasser Hifny, Mostafa Shahin, and Ahmed Ali. 2025. Towards a Unified Benchmark for Arabic Pronunciation Assessment: Qur’anic Recitation as Case Study. In Interspeech 2025, pages 2410–2414.
  5. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, ICML ’06, page 369–376, NY, USA. Association for Computing Machinery. find
  6. Nawar Halabi and Mike Wald. 2016. Phonetic inventory for an arabic speech corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 734–738. 9 451
  7. Ahmed Ibrahim. 2025. Hafs2vec: A system for the iqraeval arabic and qur’anic phoneme-level pronunciation assessment shared task. In The Third Arabic Natural Language Processing Conference (Arabic-NLP 2025), Suzhou. Association for Computational Linguistics.
  8. Yassine El Kheir, Ahmed Ali, and Shammur Absar Chowdhury. 2023a. Automatic pronunciation assessment–a review. arXiv preprint arXiv:2310.13974. ↗
  9. Yassine El Kheir, Shammur Absar Chowdhury, Ahmed Ali, Hamdy Mubarak, and Shazia Afzal. 2022. Speechblender: Speech augmentation framework for mispronunciation data generation. arXiv preprint arXiv:2211.00923. ↗
  10. Yassine El Kheir, Fouad Khnaisser, Shammur Absar Chowdhury, Hamdy Mubarak, Shazia Afzal, and Ahmed Ali. 2023b. Qvoice: Arabic speech pronunciation learning application. arXiv preprint arXiv:2305.07445. ↗
  11. Yassine El Kheir, Hamdy Mubarak, Ahmed Ali, and Shammur Absar Chowdhury. 2024. Beyond orthography: Automatic recovery of short vowels and dialectal sounds in arabic. arXiv preprint arXiv:2408.02430. ↗
  12. Daniel Korzekwa, Jaime Lorenzo-Trueba, Thomas Drugman, and Bozena Kostek. 2022. Computer-assisted pronunciation training—speech synthesis is almost all you need. Speech Communication, 142:22–33. ↗
  13. Wai-Kim Leung, Xunying Liu, and Helen Meng. 2019. Cnn-rnn-ctc based end-to-end mispronunciation detection and diagnosis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8132–8136. IEEE. find
  14. Kun Li, Xiaojun Qian, and Helen Meng. 2016. Mispronunciation detection and diagnosis in l2 english speech using multidistribution deep neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(1):193–207. ↗
  15. Ambra Neri, Ornella Mich, Matteo Gerosa, and Diego Giuliani. 2008. The effectiveness of computer assisted pronunciation training for foreign language learning by children. Computer Assisted Language Learning, 21(5):393–408. ↗
  16. Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779. ↗
  17. Nour Qandos, Serry Sibaee, Samar Ahmad, OMER NACAR, Adel Ammar, Wadii Boulila, and Yasser Alhabashi. 2025. Anplers at iqraeval shared task: Adapting whisper-large-v3 as speech-to-phoneme for qur’anic recitation mispronunciation detection. In The Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou. Association for Computational Linguistics.
  18. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR. ↗
  19. Munirah Ab Rahman, Izatul Anis Azwa Kassim, Tasir-ansurini Ab Rahman, and Siti Zarina Mohd Muji. 2021. Development of automated tajweed checking system for children in learning quran. Evolution in Electrical and Electronic Engineering, 2(1):165– 176. find
  20. Pamela M Rogerson-Revell. 2021. Computer-assisted pronunciation training (capt): Current issues and future directions. Relc Journal, 52(1):189–205. ↗
  21. Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, and Kushal Lakhotia et al. 2021. Superb: Speech processing universal performance benchmark. In Interspeech 2021, pages 1194–1198. ↗
  22. Marcely Zanon Boito, Vivek Iyer, Nikolaos Lagos, Laurent Besacier, and Ioan Calapodescu. 2024. mhubert-147: A compact multilingual hubert model. In Interspeech 2024, pages 3939–3943. find
  23. Ayman Mansour. 2025. Metapseud at iqra’eval: Domain adaptation with multi-stage fine-tuning for phoneme-level qur’anic mispronunciation detection. In The Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou. Association for Computational Linguistics.
  24. Bassam Mattar, Mohamed Fayed, and Ayman Khalafallah. 2025. Aras2p: Arabic speech-to-phonemes system. In The Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou. Association for Computational Linguistics. 10 452

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kheir-2025-iqra-eval-shared-task-qur/