Elamur Research⌕
☰
home / papers / 2025 / Enhancing Quranic Learning: A Multimodal Deep Learning Approach for Arabic Phoneme Recognition

Enhancing Quranic Learning: A Multimodal Deep Learning Approach for Arabic Phoneme Recognition

Summary

The paper addresses automatic detection of mispronounced Arabic phonemes in the context of Qur'anic recitation. In this domain, small phonetic slips can change meaning, and qualified teachers are not always available. The authors build a transformer-based multimodal classifier that treats each recording in two ways. UniSpeech encodes the raw audio. Whisper transcribes the same clip, and BERT embeds that transcript as a textual view. The two embeddings are combined using three fusion strategies: early, intermediate and late, which are then compared head to head. Training and testing use two datasets of 29 Arabic phonemes, including eight hafiz sounds, recorded by 11 native speakers. Extra YouTube recordings are added for variety. Evaluation reports accuracy, precision, recall and F1. The UniSpeech–BERT combination beats unimodal transformer baselines, gaining up to about 1.5 percentage points over the previous best on Dataset B. The authors argue that careful fusion design can be a cheaper route to better performance than enlarging a single model. They present the work as a step toward speaker-independent CALL tools for pronunciation training.

Key points

  • Two inputs per clip: UniSpeech acoustic embeddings, and BERT embeddings of Whisper transcriptions of the same audio.
  • Three fusion strategies (early, intermediate/mid, late) are implemented and compared systematically.
  • Data: two datasets covering 29 Arabic phonemes, including eight hafiz sounds, from 11 native speakers, supplemented with public YouTube recordings.
  • The work builds on the authors' earlier studies:
    • An ensemble classifier on Mel-spectrogram features reached 95.9% accuracy.
    • In a comparison of SEW, HuBERT, Wav2Vec and UniSpeech, UniSpeech came out strongest.
  • The best result is a gain of up to ~1.5 pp over the SOTA baseline on Dataset B.
  • Acknowledged limitations:
    • The audio is mostly clean.
    • Dialect coverage is limited.
    • There is no expert human evaluation.
    • There is no analysis of how much each modality contributes.

Interesting ideas

  • I1. Second view from the same signal: run an ASR model on the audio and encode its transcript as a separate view, which gives a second modality without collecting new data. — For Mova (learn): a derived representation of the same sentence could be fed as an extra witness view in silver-data agreement, for example an LLM paraphrase reparsed or a lemma-only rendering.
  • I2. Testing the fusion point: instead of assuming where to merge information, build early, mid and late fusion variants and compare them under identical conditions. — For Mova (learn): when combining witness parsers, compare arc-level voting (late) with merging their features before scoring (early/mid), and keep whichever gives better silver trees.
  • I3. Specialists plus a fusion layer instead of scaling: two domain-specialized encoders joined by a light fusion network outperformed single bigger-style transformers without adding depth. — For Mova (tree): gparse could fuse representations from the perceptron parser's features and the transformer encoder rather than growing the transformer.
  • I4. Mixing curated and in-the-wild data: clean in-house recordings were supplemented with noisier public recordings to widen variety and help generalization. — For Mova (learn): mix clean UD treebank sentences with noisy web text in training and evaluation to expose parser brittleness.
  • I5. Measuring each modality's contribution: the authors flag per-modality attribution and expert review as missing, which marks them as necessary follow-ups for any fused system. — For Mova (learn): ablate each witness or feature source as a negative control, so the gain of a fused component is explained rather than assumed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2511.17477 · arXiv 2511.17477

@misc{kucukmanisa2025enhancing,
  title = {{Enhancing Quranic Learning: A Multimodal Deep Learning Approach for Arabic Phoneme Recognition}},
  author = {Kucukmanisa, Ayhan and Gelmez, Derya and Calik, Sukru Selim and Kilimci, Zeynep Hilal},
  year = {2025},
  doi = {10.48550/arxiv.2511.17477},
  eprint = {2511.17477},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2511.17477}
}

References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akhtar, S., Hussain, F., Raja, F.R., Ehatisham-ul haq, M., Baloch, N.K., Ishmanov, F., Zikria, Y.B., 2020. Improving mispronunciation detection of arabic words for non-native learners using deep convolutional neural network features. Electronics 9, 963. find
  2. Algabri, M., Mathkour, H., Alsulaiman, M., Bencherif, M.A., 2022a. Mispronunciation detection and diagnosis with articulatory-level feedback generation for non-native arabic speech. Mathematics 10. doi:10.3390/math10152727. ↗
  3. Algabri, M., Mathkour, H., Alsulaiman, M., Bencherif, M.A., 2022b. Mispronunciation detection and diagnosis with articulatory-level feedback generation for non-native arabic speech. Mathematics 10, 2727. find
  4. Alqadheeb, F., Asif, A., Ahmad, H.F., 2021. Correct pronunciation detection for classical arabic phonemes using deep learning, in: 2021 International Conference of Women in Data Science at Taif University (WiDSTaif), IEEE. pp. 1–6. find
  5. Alrashoudi, N., Al-Khalifa, H., Alotaibi, Y., 2025. Improving mispronunciation detection and diagnosis for non-native learners of the arabic language. Discover Computing 28, 1. find
  6. Asif, A., Mukhtar, H., Alqadheeb, F., Ahmad, H.F., Alhumam, A., 2021. An approach for pronunciation classification of classical arabic phonemes using deep learning. Applied Sciences 12, 238. find
  7. Bahi, H., 2024. Automatic pronunciation assessment and feedback for learners. World Scientific . find
  8. Calık, S.S., Kucukmanisa, A., Kilimci, Z.H., 2023. An ensemble-based framework for mispronunciation detection of arabic phonemes. Applied Acoustics 212, 109593. find our card
  9. Calik, S.S., Kucukmanisa, A., Kilimci, Z.H., 2024. A novel framework for mispronunciation detection of arabic phonemes using audio-oriented transformer models. Applied Acoustics 215. find
  10. Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. ↗
  11. El Kheir, Y., Ibrahim, O., Meghanani, A., Almarwani, N., et al., 2025. Towards a unified benchmark for arabic pronunciation assessment: Qur’anic recitation as case study. Interspeech 2025 . find our card
  12. Farooq, J., Imran, M., 2021. Mispronunciation detection in articulation points of arabic letters using machine learning, in: 2021 International Conference on Computing, Electronic and Electrical Engineering (ICE Cube), IEEE. pp. 1–6. find
  13. Haouhat, A., Bellaouar, S., Nehar, A., Cherroun, H., Abdelali, A., 2025. Arabic multimodal machine learning: Datasets, applications, approaches, and challenges. arXiv preprint arXiv:2508.12227 . Page 10 of 11 ↗ our card
  14. Harere, A.A., Jallad, K.A., 2023. Mispronunciation detection of basic quranic recitation rules using deep learning. arXiv preprint arXiv:2305.06429 . ↗
  15. Li, S., Tang, H., 2024. Multimodal alignment and fusion: A survey. arXiv:2411.17040. ↗
  16. Lo, T.H., Weng, S.Y., Chang, H.J., Chen, B., 2020. An effective end-to-end modeling approach for mispronunciation detection. arXiv preprint arXiv:2005.08440 . ↗
  17. Lounis, M., Dendani, B., Bahi, H., 2025. One-class convolutional neural network for arabic mispronunciation detection, in: Intelligent Systems and Pattern Recognition, Springer Nature Switzerland. pp. 311–320. find
  18. Nazir, F., Majeed, M.N., Ghazanfar, M.A., Maqsood, M., 2019. Mispronunciation detection using deep convolutional neural network features and transfer learning-based model for arabic phonemes. IEEE Access 7, 52589–52608. find
  19. Radford, A., Kim, J.W., Xu, T., Brockman, G., Mcleavey, C., Sutskever, I., 2022. Robust speech recognition via large-scale weak supervision. arXiv:2212.04356. ↗ our card
  20. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I., 2017. Attention is all you need, in: Advances in Neural Information Processing Systems (NeurIPS). find
  21. Wang, C., Wu, Y., Qian, Y., Kumatani, K., Liu, S., Wei, F., Zeng, M., Huang, X., 2021. Unispeech: Unified speech representation learning with labeled and unlabeled data. arXiv:2101.07597. ↗
  22. Yan, B.C., Wang, H.W., Chen, B., 2023. Peppanet: Effective mispronunciation detection and diagnosis leveraging phonetic, phonological, and acoustic cues, in: 2022 IEEE Spoken Language Technology Workshop (SLT), IEEE. pp. 1045–1051. find
  23. Zhu, D., Liu, J., Yang, Y., Lin, Z., 2023. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence . Page 11 of 11 find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kucukmanisa-2025-enhancing-quranic-learning-multimodal-deep/