Elamur Research⌕
☰
home / papers / 2025 / Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment

Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment

Paper · 2025 · Kwanghee Choi, Eunjung Yeo, Kalvin Chang et al.

Summary

Assessing atypical pronunciation, such as dysarthric or non-native speech, requires telling unusual realizations of a phoneme apart from normal variation. One phoneme can legitimately surface in several forms depending on context; English /t/, for example, can be aspirated, flapped or glottalized. Recent classifier-based Goodness of Pronunciation (GoP) methods ignore this allophony. They treat each phoneme as one cluster and rely on a softmax, which assumes the test speech resembles the training data.

The authors propose MixGoP, which fits one Gaussian mixture model per phoneme on frozen self-supervised speech model (S3M) features. Each phoneme is modelled as several subclusters, standing in for allophones. Scoring uses the phoneme likelihood directly rather than a softmax posterior, which makes it behave like an out-of-distribution detector.

Models are trained only on healthy or native speech and tested on atypical speech. Across five datasets and eight compared methods, MixGoP reaches state-of-the-art results on four. Further analysis indicates that S3M features represent allophonic variation better than MFCCs or Mel spectrograms. This supports pairing mixture-based scoring with S3Ms for clinical and language-learning assessment.

Key points

  • Datasets: three dysarthric corpora (UASpeech, TORGO, SSNCE, the last in Tamil) and two non-native corpora (speechocean762, L2-ARCTIC).
  • Features:
    • Frozen S3M features from XLS-R-300M and WavLM-Large, taken from every layer (convolutional layer 0 plus Transformer layers 1–24).
    • Baselines: MFCCs, Mel spectrograms and, for speechocean762, TDNN-F features.
  • Training: each phoneme gets a GMM with a fixed number of components (e.g., C = 32). Components are initialized with k-means and fitted by EM in scikit-learn. The Gaussian quadratic term relates the score to the Mahalanobis distance.
  • Baselines: classifier-based GoP variants (GMM-GoP, NN-GoP, DNN-GoP, MaxLogit-GoP) and OOD detectors (kNN, one-class SVM, among others).
  • Limitations and availability:
    • The data is mostly English.
    • Alignment quality was not verified.
    • The allophony analysis relies mainly on TORGO's manual alignments.
    • Code is released on GitHub (juice500ml/acoustic-units-for-ood).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksMedicine · 241 worksPhysiology · 29 works

Cite

Cite as Kwanghee Choi, Eunjung Yeo, Kalvin Chang, Shinji Watanabe and David R Mortensen. 2025. Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.naacl-long.132

@inproceedings{choi2025leveraging,
  title = {{Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment}},
  author = {Kwanghee Choi, Eunjung Yeo, Kalvin Chang, Shinji Watanabe and David R Mortensen},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.132},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.132}
}

References

64 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B. M Abdullah, M. M. Shaik, B. Möbius, and D. Klakow. 2023. An information-theoretic analysis of self-supervised discrete representations of speech. In Proc. Interspeech. find
  2. T-H. Feng, A. Dong, C-F Yeh, et al. 2023. SUPERB @ SLT 2022: Challenge on generalization and efficiency of self-supervised speech representation learning. In Proc. SLT. find
  3. A. Babu, C. Wang, A. Tjandra, et al. 2022. XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale. In Proc. Interspeech.
  4. Y. Gong, Z. Chen, I-H. Chu, P. Chang, and J. Glass. 2022. Transformer-based multi-aspect multi-granularity non-native english speaker pronunciation assessment. In Proc. ICASSP.
  5. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Proc. NeurIPS.
  6. D. Hendrycks and K. Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proc. ICLR. find
  7. J.A. Bilmes et al. 1998. A gentle tutorial of the em algorithm and its application to parameter estimation for gaussian mixture and hidden markov models. International Computer Science Institute. find
  8. A. Hernandez, E.J. Yeo, S. Kim, and M. Chung. 2020. Dysarthria detection and severity assessment using rhythm-based metrics. In Proc. Interspeech. find
  9. M. Blondel, O. Teboul, Q. Berthet, and J. Djolonga. 2020. Fast differentiable sorting and ranking. In Proc. ICML. find
  10. X. Cao, Z. Fan, T Svendsen, and G. Salvi. 2024. A framework for phoneme-level pronunciation assessment using ctc. In Proc. Interspeech. find
  11. W-N. Hsu, B. Bolte, Yao-Hung H. Tsai, et al. 2021. Hu-BERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM TASLP. find
  12. W. Hu, Y. Qian, and F. K. Soong. 2015a. An improved dnn-based approach to mispronunciation detection and diagnosis of l2 learners’ speech. In SLaTE. find
  13. X. Chang, B. Yan, K. Choi, et al. 2024. Exploring speech recognition, translation, and understanding with discrete speech units: A comparative study. In Proc. ICASSP. find
  14. W. Hu, Y. Qian, F. K Soong, and Y. Wang. 2015b. Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiers. Speech Communication. find
  15. F-A. Chao, T-H. Lo, T-I. Wu, et al. 2022. 3m: An effective multi-view, multi-granularity, and multi-aspect modeling approach to english pronunciation assessment. In Proc. APSIPA ASC. find
  16. O. Jokisch, A. Wagner, R. Sabo, et al. 2009. Multilingual speech data collection for the assessment of pronunciation and prosody in a language learning system. In Proc. of SPECOM. find
  17. S. Chen, C. Wang, Z. Chen, et al. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE JSTSP. find
  18. H. Kim, M. Hasegawa-Johnson, A. Perlman, et al. 2008. Dysarthric speech database for universal access research. In Proc. Interspeech. find
  19. S. Cheng, Z. Liu, L. Li, et al. 2020. ASR-Free Pronunciation Assessment. In Proc. Interspeech. find
  20. H. Kim, K. Martin, M. Hasegawa-Johnson, and A. Perlman. 2010. Frequency of consonant articulation errors in dysarthric speech. Clinical linguistics & phonetics. find
  21. K. Choi, J-W. Jung, and S. Watanabe. 2024a. Understanding probe behaviors through variational bounds of mutual information. In Proc. ICASSP. find
  22. K. Choi, A. Pasad, T. Nakamura, et al. 2024b. Self-supervised speech representations are more phonetic than semantic. In Proc. Interspeech. find
  23. D. P. Kingma and J. Ba. 2015. Adam: A method for stochastic optimization. In Proc. ICLR.
  24. J. Kominek and A. W. Black. 2004. The cmu arctic speech databases. In Proc. Speech Synthesis Workshop. find
  25. D. Korzekwa, J. Lorenzo-Trueba, S. Zaporowski, S. Calamaro, T. Drugman, and B. Kostek. 2021. Mispronunciation detection in non-native (l2) english with uncertainty modeling. In Proc. ICASSP.
  26. P. Ladefoged. 1965. The nature of general phonetic theories. Monograph Series on Languages and Linguistics, pages 27–42. find
  27. J. Lee, K. Lee, H. Lee, and J. Shin. 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Proc. NeurIPS. find
  28. K-F. Lee, S. Hayamizu, H-W. Hon, et al. 1990. Allophone clustering for continuous speech recognition. In Proc. ICASSP. find
  29. K. Li, X. Qian, and H. Meng. 2016. Mispronunciation detection and diagnosis in l2 english speech using multidistribution deep neural networks. IEEE/ACM TASLP. find
  30. O. D. Liu, H. Tang, and S. Goldwater. 2023. Self-supervised predictive coding models encode speaker and phonetic information in orthogonal subspaces. In Proc. Interspeech. find
  31. L. MacKenzie and D. Turton. 2020. Assessing the accuracy of existing forced alignment software on varieties of british english. Linguistics Vanguard, 6(s1):20180061. find
  32. K. Martin, J. Gauthier, C. Breiss, and R. Levy. 2023. Probing Self-supervised Speech Models for Phonetic and Phonemic Information: A Case Study in Aspiration. In Proc. Interspeech. find
  33. M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger. 2017. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Proc. Interspeech. find
  34. B. McFee, C. Raffel, D. Liang, et al. 2015. librosa: Audio and music signal analysis in python. In Proc. Python in Science Conference. find
  35. L. McInnes, J. Healy, N. Saul, and L. Großberger. 2018. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software. find
  36. B. K. Ng and P. S. Chiew. 2023. L1 influence on stop consonant production: A case study of malaysian mandarin-english bilinguals. Asian Englishes. find
  37. T. Nguyen, H. D. Nguyen, F. Chamroukhi, and G. J. McLachlan. 2020. Approximation by finite mixtures of continuous density functions that vanish at infinity. Cogent Mathematics & Statistics. find
  38. A. Pasad, B. Shi, and K. Livescu. 2023. Comparative layer-wise analysis of self-supervised speech models. In Proc. ICASSP. find
  39. F. Pedregosa, G. Varoquaux, A. Gramfort, et al. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research. find
  40. D. Povey, G. Cheng, Y. Wang, et al. 2018. Semi-orthogonal low-rank matrix factorization for deep neural networks. In Proc. Interspeech, pages 3743– 3747. find
  41. F. Rudzicz, A. K. Namasivayam, and T. Wolff. 2012. The torgo database of acoustic and articulatory speech from speakers with dysarthria. Language Resources and Evaluation. find
  42. S. Sagayama. 1989. Phoneme environment clustering for speech recognition. In Proc. ICASSP. find
  43. B. Schölkopf, J. C Platt, J. Shawe-Taylor, et al. 2001. Estimating the support of a high-dimensional distribution. Neural computation. find
  44. M. Shahin and B. Ahmed. 2019. Anomaly detection based pronunciation verification approach using speech attribute features. Speech Communication. find
  45. M. Shahin, J. Epps, and B. Ahmed. 2024. Phonological level wav2vec2-based mispronunciation detection and diagnosis method. In Proc. Interspeech. find
  46. J. Shi, N. Huo, and Q. Jin. 2020. Context-aware goodness of pronunciation for computer-assisted pronunciation training. In Proc. Interspeech. find
  47. A. Sicherman and Y. Adi. 2023. Analysing discrete self supervised speech representation for spoken language modeling. In Proc. ICASSP. find
  48. S. Sudhakara, M. K. Ramanathi, C. Yarra, and P. K. Ghosh. 2019. An improved goodness of pronunciation (gop) measure for pronunciation evaluation with dnn-hmm system considering hmm transition probabilities. In Proc. Interspeech. find
  49. Y. Sun, Y. Ming, X. Zhu, and Y. Li. 2022. Out-of-distribution detection with deep nearest neighbors. In Proc. ICML. find
  50. M. C. TA, T. Nagarajan, and P. Vijayalakshmi. 2016. Dysarthric speech corpus in tamil for rehabilitation research. In Proc. TENCON. find
  51. William F Twaddell. 1952. Phonemes and allophones in speech analysis. Journal of the Acoustical Society of America. find
  52. A. Pasad, C-M. Chien, S. Settle, et al. 2024. What Do Self-Supervised Speech Models Know About Words? TACL.
  53. J. Vidal, L. Ferrer, and L. Brambilla. 2019. Epadb: A database for development of pronunciation assessment systems. In Proc. Interspeech. find
  54. A. Pasad, J-C Chou, and K. Livescu. 2021. Layer-wise analysis of a self-supervised speech representation model. In Proc. ASRU, pages 914–921. find
  55. Z. Wang, Q. Gu, Y. Ning, and H. Liu. 2015. High dimensional em algorithm: Statistical optimization and asymptotic normality. In Proc. NeurIPS. find
  56. D. Wells, H. Tang, and K. Richmond. 2022. Phonetic Analysis of Self-supervised Representations of English Speech. In Proc. Interspeech. find
  57. S. M. Witt and S. J. Young. 2000. Phone-level pronunciation scoring and assessment for interactive language learning. Speech Communication, 30:95–108. find
  58. X. Xu, Y. Kang, S. Cao, et al. 2021. Explore wav2vec 2.0 for Mispronunciation Detection. In Proc. Interspeech. find
  59. S-W. Yang, P-H. Chi, Y-S. Chuang, et al. 2021. SU-PERB: Speech Processing Universal PERformance Benchmark. In Proc. Interspeech, pages 1194–1198. find
  60. E-J. Yeo, K. Choi, S. Kim, and M. Chung. 2023a. Speech Intelligibility Assessment of Dysarthric Speech by using Goodness of Pronunciation with Uncertainty Quantification. In Proc. Interspeech. find
  61. E-J. Yeo, H. Ryu, J. Lee, et al. 2023b. Comparison of l2 korean pronunciation error patterns from five l1 backgrounds by using automatic phonetic transcription. In Proc. ICPhS. find
  62. S. J. Young, J. J. Odell, and P. C. Woodland. 1994. Tree-based state tying for high accuracy modelling. In Proc. HLT. find
  63. J. Zhang, Z. Zhang, Y. Wang, et al. 2021. speechocean762: An open-source non-native english speech corpus for pronunciation assessment. In Proc. Interspeech.
  64. G. Zhao, S. Sonsaat, A. Silpachai, et al. 2018. L2-arctic: A non-native english speech corpus. In Proc. Interspeech. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/choi-2025-leveraging-allophony-self-supervised-speech/