Elamur Research⌕
☰
home / papers / 2023 / An ensemble-based framework for mispronunciation detection of Arabic phonemes

An ensemble-based framework for mispronunciation detection of Arabic phonemes

Summary

The paper addresses automatic detection of mispronounced Arabic phonemes, aimed at computer-assisted language learning (CALL) and especially at Quranic (Classical Arabic) recitation, where pronunciation rules are strict. The authors recorded their own dataset: 29 Arabic letters voiced by 11 speakers, 8 of them hafiz. They enlarged it with four audio augmentations: added noise, time shifting, time stretching and pitch shifting. Two acoustic representations, MFCC and Mel spectrogram, were each fed to classical classifiers (SVM, k-NN, decision tree, naive Bayes, random forest). These were then combined through voting, bagging, boosting and three stacking variants. Mel spectrogram features clearly beat MFCC across the board. A voting ensemble on Mel spectrograms reached 95.9% accuracy, which outperformed two reimplemented deep sound-classification networks on the same data and also ran fastest. The work matters because it shows that cheap, low-complexity models can match or beat deep learning on a small speech task. That makes them suitable for embedded or low-resource deployment.

Key points

  • Own dataset: 29 Arabic letters, 11 speakers (8 hafiz), augmented with noise, time shift, time stretch and pitch shift. The data uses an 80/20 train/test split plus 5-fold cross-validation.
  • With MFCC input, the best single model was SVM (accuracy 0.758), and the best ensembles stayed below about 0.78. With Mel spectrogram input, k-NN (k=3) reached 0.953 and SVM reached 0.952.
  • The best system was a voting ensemble on Mel spectrograms, with accuracy 0.959, precision 0.969, recall 0.957 and F1 0.962. Stacking with random forest followed at 0.951.
  • Two deep baselines were reimplemented on the same dataset: Abdoli et al. 2019 reached 0.910 accuracy and Choi et al. 2018 reached 0.900. Their processing times were 0.470 s and 0.612 s, compared with 0.091 s for the proposed method.
  • The text reports the headline figure inconsistently: 95.9% in the abstract and tables, but 95.3% in the introduction.

Interesting ideas

  • I1. The choice of input representation shifted accuracy by roughly 20 points, far more than the choice of classifier, so features deserve tuning before the learner does. — For Mova (tokens): Before swapping tagger algorithms, compare alternative feature templates for the en tagger and morphology on the same split to find where the real gain lies.
  • I2. A simple majority vote over several cheap, diverse classifiers matched or beat heavier stacking and deep models while staying fast. — For Mova (learn): A plain vote over the witness parsers is a strong, explainable baseline for building silver trees, and any fancier aggregation should have to beat it.
  • I3. Baselines were reimplemented from their original specifications and run on the very same dataset, with inference time reported next to accuracy. — For Mova (tree): Compare the perceptron parser and gparse on identical treebank splits, reporting LAS together with parse time, so the cost of extra accuracy is visible.
  • I4. The authors broke results down per letter to show that their method stays stable where the deep models collapse on certain classes. — For Mova (answer): Report test-reader scores per question type or per fable rather than one aggregate, so weak categories are not hidden by a good average.
  • I5. Label-preserving perturbations (noise, shifts, stretching) were used to multiply a tiny hand-recorded dataset. — For Mova (learn): Meaning-preserving text perturbations, such as name swaps or number changes in word problems, could enlarge small gold sets and also serve as consistency checks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

3 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.01378 · arXiv 2301.01378

@misc{calik2023ensemble,
  title = {{An ensemble-based framework for mispronunciation detection of Arabic phonemes}},
  author = {Calik, Sukru Selim and Kucukmanisa, Ayhan and Kilimci, Zeynep Hilal},
  year = {2023},
  doi = {10.48550/arxiv.2301.01378},
  eprint = {2301.01378},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.01378}
}

References

49 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdoli, S., Cardinal, P., Lameiras Koerich, A., 2019. End-to-end environmental sound classification using a 1d convolutional neural network. Expert Systems with Applications 136, 252–263. find
  2. Abufanas, O., 2013. Computer aided language learning system for arabic for second language learners. International Journal of Educational and Pedagogical Sciences 7, 3189–3193. find
  3. Adnan, A.H.M., Zamari, Z.M., 2012. Computer-aided self-access language learning: Views of indonesian, malaysian & new zealand practitioners. Procedia-Social and Behavioral Sciences 67, 49–60. find
  4. Akhtar, S., Hussain, F., Raja, F.R., Ehatisham-ul haq, M., Baloch, N.K., Ishmanov, F., Zikria, Y.B., 2020. Improving mispronunciation detection of arabic words for non-native learners using deep convolutional neural network features. Electronics 9, 963. find
  5. Algabri, M., Mathkour, H., Alsulaiman, M., Bencherif, M.A., 2022. Mispronunciation detection and diagnosis with articulatory-level feedback generation for non-native arabic speech. Mathematics 10, 2727. find
  6. Alp, S., O.E., 2019. Makine öğrenmesinde sınıflandırma yöntemleri ve R uygulamaları. Nobel Akademik Yayıncılık, Istanbul, Turkey. find
  7. Arafa, M.N., Elbarougy, R., Ewees, A., Behery, G., 2018. A dataset for speech recognition to support arabic phoneme pronunciation. International Journal of Image, Graphics and Signal Processing 10, 31. find
  8. Arias, J.P., Yoma, N.B., Vivanco, H., 2010. Automatic intonation assessment for computer aided language learning. Speech communication 52, 254–267. find
  9. Asif, A., Mukhtar, H., Alqadheeb, F., Ahmad, H.F., Alhumam, A., 2021. An approach for pronunciation classification of classical arabic phonemes using deep learning. Applied Sciences 12, 238. find
  10. Breiman, L., 2001. Random forests. Machine Learning 45, 5–32. find
  11. Choi, K., Fazekas, G., Sandler, M., Cho, K., 2018. A comparison of audio signal preprocessing methods for deep neural networks on music tagging, pp. 1870–1874. find
  12. Cristianini, N., Shawe-Taylor, J., 2000. An introduction to support vector machines. Cambridge University Press, Cambridge, UK. find
  13. Cucchiarini, C., Wet, F.d., Strik, H., Boves, L., 1998. Assessment of dutch pronunciation by means of automatic speech recognition technology . find
  14. Dave, N., 2013. Feature extraction methods lpc, plp and mfcc in speech recognition. International Journal for Advance Research in Engineering and Technology 1, 1–4. find
  15. Džeroski, S., Z.B., 2004. Is combining classifiers with stacking better than selecting the best one? Machine Learning 54, 255–273. find
  16. Ehsani, F., Knodt, E., 1998. Speech technology in computer-aided language learning: Strengths and limitations of a new call paradigm. Language Learning & Technology 2, 54–73. find
  17. Elsayed, E.K., Fathy, D., 2019. Evaluation of quran recitation via owl ontology based system. Int. Arab J. Inf. Technol. 16, 970–977. find
  18. Farooq, J., Imran, M., 2021. Mispronunciation detection in articulation points of arabic letters using machine learning, in: 2021 International Conference on Computing, Electronic and Electrical Engineering (ICE Cube), IEEE. pp. 1–6. find
  19. Freund, Y., Schapire, R., 1996. Experiments with a new boosting algorithm, in: International Conference on Machine Learning; Italy, pp. 148–156. find
  20. Gopika, D., Azhagusundari, B., 2014. An analysis on ensemble methods in classification tasks . find
  21. Gopika D. Azhagusundari, B., 2014. An analysis on ensemble methods in classification tasks. International Journal of Advanced Research in Computer and Communication Engineering 3, 7423–7427. find
  22. Hu, W., Qian, Y., Soong, F.K., 2013. A new dnn-based high quality pronunciation evaluation for computer-aided language learning (call)., in: Interspeech, pp. 1886–1890. find
  23. Khan, A.F.A., Mourad, O., Mannan, A.M.K.B., Dahan, H.B.A.M., Abushariah, M.A., 2013. Automatic arabic pronunciation scoring for computer aided language learning, in: 2013 1st International Conference on Communications, Signal Processing, and their Applications (ICCSPA), IEEE. pp. 1–6. find
  24. Kilimci, Z.H., Akyokus, S., Omurca, S.I., 2016. The effectiveness of homogenous ensemble classifiers for turkish and english texts, in: 2016 International Symposium on INnovations in Intelligent SysTems and Applications (INISTA), IEEE. pp. 1–7. find
  25. L., B., 1996. Bagging predictors. Machine learning 24, 123–140. find
  26. L., R., 2010. Ensemble-based classifiers. Artificial Intelligence Review 33, 1–39. find
  27. Lane, J., 2019. The 10 most spoken languages in the world. Babbel Magazine 6. find
  28. Maqsood, M., Habib, H., Anwar, S., Ghazanfar, M.A., Nawaz, T., 2017. A comparative study of classifier based mispronunciation detection system for confusing. The Nucleus 54, 114–120. find
  29. Maqsood, M., Habib, H.A., Nawaz, T., Haider, K.Z., 2016. A complete mispronunciation detection system for arabic phonemes using svm. International Journal of Computer Science and Network Security (IJCSNS) 16, 30. find
  30. Minematsu, N., 2004. Pronunciation assessment based upon the compatibility between a learner’s pronunciation structure and the target language’s lexical structure, in: Eighth International Conference on Spoken Language Processing. find
  31. Muaad, A.Y., Jayappa, H., Al-antari, M.A., Lee, S., 2021. Arcar: a novel deep learning computer-aided recognition for character-level arabic text representation and recognition. Algorithms 14, 216. find
  32. Mucherino, A., Papajorgji, P.J., PM., P., 2009. K-nearest neighbor classification. Data Mining in Agriculture 34, 83–106. find
  33. Muhammad, W.M., Muhammad, R., Muhammad, A., Martinez-Enriquez, A., 2010. Voice content matching system for quran readers, in: 2010 Ninth Mexican International Conference on Artificial Intelligence, IEEE. pp. 148–153. find
  34. Nazir, F., Majeed, M.N., Ghazanfar, M.A., Maqsood, M., 2019. Mispronunciation detection using deep convolutional neural network features and transfer learning-based model for arabic phonemes. IEEE Access 7, 52589–52608. find
  35. Nazir, F., Majeed, M.N., Ghazanfar, M.A., Maqsood, M., 2021. An arabic mispronunciation detection system based on the frequency of mistakes for asian speakers. Mehran University Research Journal Of Engineering & Technology 40, 279–297. find
  36. Neumeyer, L., Franco, H., Digalakis, V., Weintraub, M., 2000. Automatic scoring of pronunciation quality. Speech communication 30, 83–93. find
  37. Peng, L., Gao, Y., Lin, B., Ke, D., Xie, Y., Zhang, J., 2022. Text-aware end-to-end mispronunciation detection and diagnosis. arXiv preprint arXiv:2206.07289 . ↗
  38. Polikar, R., 2006a. Ensemble based systems in decision making. IEEE Circuits and systems magazine 6, 21–45. find
  39. Polikar, R., 2006b. Ensemble based systems in decision making. IEEE Circuits and Systems Magazine 6, 21–45. find
  40. Putra, B., Atmaja, B., Prananto, D., 2012. Developing speech recognition system for quranic verse recitation learning software. IJID (International Journal on Informatics for Development) 1, 14–21. find
  41. Ren, Y., Zhang, L., Suganthan, P.N., 2016. Ensemble classification and regression-recent developments, applications and future directions. IEEE Computational intelligence magazine 11, 41–53. find
  42. Ren Y., Zhang L., S.P.N., 2016. Ensemble classification and regression-recent developments applications and future directions. IEEE Computational Intelligence Magazine 11, 41–53. find
  43. Rokach, L., 2010. Ensemble-based classifiers. Artificial intelligence review 33, 1–39. find
  44. Shaalan 1, K.F., 2005. An intelligent computer assisted language learning system for arabic learners. Computer Assisted Language Learning 18, 81–109. find
  45. Shareef, S.R., Al-Irhayim, Y.F.M., 2022. Comparison between features extraction techniques for impairments arabic speech. Al-Rafidain Engineering Journal (AREJ) 27, 190–197. find
  46. Stevens, S.S., Volkmann, J., B., N.E., 1937. A scale for the measurement of the psychological magnitude pitch. Journal of the Acoustical Society of America 8, 185–190. find
  47. Webb, G.I., Keogh, E., R., M., 2010. Naive bayes. Encyclopedia of Machine Learning 15, 713–714. find
  48. Yang, M., Hirschi, K., Looney, S.D., Kang, O., Hansen, J.H., 2022. Improving mispronunciation detection with wav2vec2-based momentum pseudo-labeling for accentedness and intelligibility assessment. arXiv preprint arXiv:2203.15937 . ↗
  49. Ziafat, N., Ahmad, H.F., Fatima, I., Zia, M., Alhumam, A., Rajpoot, K., 2021. Correct pronunciation detection of the arabic alphabet using deep learning. Applied Sciences 11, 2508. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/calik-2023-ensemble-based-framework-mispronunciation-detection/