Elamur Research⌕
☰
home / papers / 2026 / A Fusion-Aware Two-Stage Framework for Mispronunciation Detection and Diagnosis in Low-Resource Modern Standard Arabic

A Fusion-Aware Two-Stage Framework for Mispronunciation Detection and Diagnosis in Low-Resource Modern Standard Arabic

Summary

The paper tackles phoneme-level mispronunciation detection and diagnosis (MDD) for Modern Standard Arabic, where learner recordings are scarce and synthetic error data does not match real learner speech well. The authors join a wav2vec2-XLS-R encoder (xls-r-300m), fine-tuned with a low learning rate, to a stack of causal dilated temporal convolutional networks (TCNs) and train it with a CTC objective. The convolutional head is chosen for its local focus. It is meant to keep subtle contrasts that global context tends to smooth away, such as emphatic versus plain consonants and gemination. Training runs in two stages. The first uses native and TTS-generated data. The second adapts the model on a small set of genuine learner recordings and saves several checkpoints along the way. At inference, these checkpoints are merged through a confusion network, and the result is rescored with a modified Kneser-Ney N-gram model during beam search. On the blind QuranMB.v2 test set, the system reaches an F1 of 0.7201, against 0.4414 for the official baseline. This placed it first in the IqraEval.2 Challenge. The work shows how to build a strong pronunciation assessor from very little real data.

Key points

  • Architecture: wav2vec2-XLS-R encoder, then L causal dilated TCN blocks, then a CTC phoneme classifier. The authors report that it beats LSTM-based baselines in the low-resource setting.
  • Stage 1 data, as shown in Figure 1: Iqra_train (79h) native speech and Iqra_TTS (80h) synthetic speech. The real-learner adaptation set is Iqra_Extra_IS26, about 2h.
  • Phoneme F1 on QuranMB.v2 is 0.7201, against a baseline of 0.4414. This is a 63.1% relative gain and the top result in IqraEval.2 (Interspeech 2026).
  • Combining multiple checkpoints through confusion-network voting plus MKN N-gram rescoring adds 5.5% relative over the best single checkpoint.
  • CTC removes the need for frame-level alignment. This also helps with phonemes whose length varies, such as geminates like /bb/.

Interesting ideas

  • I1. A diagnostic system should report what was actually produced, not silently "fix" it toward the expected form; the authors pick a model design that resists such normalization so that errors stay visible. — For Mova (tokens): The tokenizer and morphology layer could keep the raw non-standard form (typos, learner errors) as a first-class token alongside any normalized reading, so later steps can explain deviations instead of hiding them.
  • I2. Learn broad mappings first on plentiful native plus synthetic data, then shift decision boundaries on a tiny set of authentic examples, so the model is not overfit to artifacts of generated data. — For Mova (learn): Train parsers on large silver or LLM-generated data first, then run a short final adaptation on a small gold set of real texts, and compare against a model trained on silver only as a control.
  • I3. Save several checkpoints from different points of the final training phase and treat them as an ensemble, since each has a slightly different sensitivity to unusual input. — For Mova (learn): Snapshots from one training run could serve as extra cheap "witness parsers" when building agreed silver data, without training separate models.
  • I4. Align competing hypotheses into a confusion network, vote position by position, then let a lightweight N-gram model break ties during beam search. — For Mova (tree): Disagreeing parser outputs could be aligned per token and fused by head and label voting, with a small dependency-pattern model resolving ties, and the vote record kept as an explanation.
  • I5. Choose an architecture with a deliberately local bias when the target signal is fine-grained and easily washed out by wide context. — For Mova (tree): For decisions that depend mostly on nearby words, such as attaching punctuation or particles, a locally restricted feature window in the perceptron parser may be more robust than broad context features.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2606.24086 · arXiv 2606.24086

@misc{yang2026fusion,
  title = {{A Fusion-Aware Two-Stage Framework for Mispronunciation Detection and Diagnosis in Low-Resource Modern Standard Arabic}},
  author = {Yang, Jing and Zhang, Shuqing and Deng, Yongyi and Li, Pan and Dang, Ting and Huang, Gongping and Chen, Jingdong and Benesty, Jacob},
  year = {2026},
  doi = {10.48550/arxiv.2606.24086},
  eprint = {2606.24086},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.24086}
}

References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K. Li, X. Qian, and H. Meng, “Mispronunciation detection and diagnosis in l2 english speech using multidistribution deep neural networks,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 25, no. 1, pp. 193–207, 2017. find
  2. T.-H. Lo, Y.-T. Sung, and B. Chen, “Improving end-to-end modeling for mispronunciation detection with effective augmentation mechanisms,” in 2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2021, pp. 1049–1055. find
  3. W.-K. Leung, X. Liu, and H. Meng, “Cnn-rnn-ctc based end-to-end mispronunciation detection and diagnosis,” in Proc. IEEE ICASSP, 2019, pp. 8132–8136. find
  4. Y. S. Alsahafi and M. Asad, “Empirical study on mispronunciation detection for tajweed rules during quran recitation,” in 2024 6th ICCI, 2024, pp. 39–45. find
  5. S. Mao, Z. Wu, R. Li, X. Li, H. Meng, and L. Cai, “Applying multitask learning to acoustic-phonemic model for mispronunciation detection and diagnosis in l2 english speech,” in Proc. IEEE ICASSP, 2018, pp. 6254–6258. find
  6. S. Mao, X. Li, K. Li, Z. Wu, X. Liu, and H. Meng, “Unsupervised discovery of an extended phoneme set in l2 english speech for mispronunciation detection and diagnosis,” in Proc. IEEE ICASSP, 2018, pp. 6244–6248. find
  7. Y. E. Kheir, O. Ibrahim, A. Meghanani, N. Almarwani, H. O. Toyin, S. Alharbi, M. Alfadly, L. Alkanhal, I. Selim, S. Elbatal, S. Mdhaffar, T. Hain, Y. Hifny, M. Shahin, and A. Ali, “Towards a unified benchmark for arabic pronunciation assessment: Quranic recitation as case study,” ArXiv, vol. abs/2506.07722, 2025. ↗ our card
  8. E. Variani, T. Bagby, K. Lahouel, E. McDermott, and M. Bacchiani, “Sampled connectionist temporal classification,” in Proc. IEEE ICASSP, 2018, pp. 4959–4963. find
  9. H. Li and W. Wang, “Reinterpreting ctc training as iterative fitting,” Pattern Recognition, vol. 105, p. 107392, 2020. find
  10. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd International Conference on Machine Learning, ser. ICML ’06. New York, NY, USA: Association for Computing Machinery, 2006, p. 369–376. find
  11. K. Audhkhasi, B. Kingsbury, B. Ramabhadran, G. Saon, and M. Picheny, “Building competitive direct acoustics-to-word models for english conversational speech recognition,” in Proc. IEEE ICASSP, 2018, pp. 4759–4763. find
  12. H. Sak, A. Senior, K. Rao, and F. Beaufays, “Fast and accurate recurrent neural network acoustic models for speech recognition,” in Interspeech 2015, 2015, pp. 1468–1472. find
  13. W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 29, pp. 3451–3460, 2021. find our card
  14. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 12 449–12 460. find our card
  15. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning. PMLR, 2023, pp. 28 492–28 518. find our card
  16. B.-C. Yan, M.-C. Wu, H.-T. Hung, and B. Chen, “An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling,” in Interspeech 2020, 2020, pp. 3032–3036. find
  17. N. Alrashoudi, H. Al-Khalifa, and Y. Alotaibi, “Improving mispronunciation detection and diagnosis for non-native learners of the Arabic language,” Discover Computing, vol. 28, no. 1, p. 1, Jan. 2025. find
  18. M. Wu, K. Li, W.-K. Leung, and H. Meng, “Transformer Based End-to-End Mispronunciation Detection and Diagnosis,” in Interspeech 2021, 2021, pp. 3954–3958. find
  19. S. A.-H. Mohammed Hadwan, Hamzah A. Alsayadi, “An end-to-end transformer-based automatic speech recognition for qur’an reciters,” Computers, Materials & Continua, vol. 74, no. 2, pp. 3471–3487, 2023. find
  20. F.-R. Stöter, S. Chakrabarty, B. Edler, and E. A. P. Habets, “Temporal convolutional networks for speech and music detection,” in Proceedings of the 20th International Society for Music Information Retrieval Conference (ISMIR 2019). Delft, Netherlands: International Society for Music Information Retrieval (ISMIR), 2019, pp. 458–465. find
  21. A. Haque, M. Guo, P. Verma, and L. Fei-Fei, “Audio-linguistic embeddings for spoken sentences,” Proc. IEEE ICASSP, pp. 7355–7359, 2019. find
  22. R. Kneser and H. Ney, “Improved backing-off for m-gram language modeling,” in 1995 International Conference on Acoustics, Speech, and Signal Processing, vol. 1, 1995, pp. 181–184 vol.1. find
  23. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-fusion-aware-two-stage-framework/