Elamur Research⌕
☰
home / papers / 2021 / Experiments of ASR-based mispronunciation detection for children and adult English learners

Experiments of ASR-based mispronunciation detection for children and adult English learners

Paper · 2021 · Nina Hosseini-Kivanani; Roberto Gretter; Marco Matassoni; and Giuseppe Daniele Falavigna

Summary

The paper builds a phone-level mispronunciation detector for Italian learners of English. It also aims to make speech recognition more robust to the non-native pronunciations that inflate recognition errors. The system is a Kaldi phone recognizer with a hybrid acoustic model: GMM stages bootstrap TDNNs trained with LF-MMI on i-vector-augmented MFCCs. Two decoding setups are compared. One uses free phone n-grams (orders 1–5). The other is an "error model": the lexicon of known words is expanded with variant pronunciations produced by hand-written rules, which were derived from the errors the 5-gram recognizer made most often. Evaluation uses two phonetically hand-annotated corpora, ChildEn (Italian children) and the Italian part of ISLE (adults). Each utterance is analyzed through three aligned phone transcriptions: canonical, manually corrected, and ASR output. The authors report that the error model separates correct from incorrect sounds and lowers phone error rate compared with the plain n-gram setup. They argue such detectors could support self-directed pronunciation practice in CAPT tools, provided they are adapted to the learner's native language.

Key points

  • Data:
    • ChildEn: 5,268 utterances (about 3h28m) from 78 children around ten years old, with 3–4 years of school English.
    • ISLE (adult Italian learners): the abstract reports 5,867 utterances from 46 speakers, while the data section mentions 23 Italian speakers. The full ISLE audio totals about 17h54m.
  • Acoustic model (standard Kaldi recipe):
    • GMM stages on MFCCs with LDA and speaker adaptive training.
    • 100-dimensional i-vectors appended to the features.
    • TDNNs trained with LF-MMI on mixed child and adult speech.
  • Phone inventory: it combines English and Italian phones, because the two languages reportedly share only about 40% of their phones. This lets the recognizer output L1-coloured realizations.
  • Error types: errors are classified as substitution, insertion or deletion. The most frequent errors involve English phones that Italian lacks (e.g., /ax/), which speakers replace with the nearest Italian sound or drop.
  • Results: the error model improves PER and accuracy over plain phone n-grams on both corpora.

Interesting ideas

  • I1. Comparing three transcriptions pairwise gives three different measurements: canonical vs. manual maps the true errors, canonical vs. system shows what the system flags on unannotated input, and manual vs. system scores detection quality.
  • I2. The error rules were bootstrapped in two steps: an unconstrained phone recognizer first revealed the most frequent deviations, which were then frozen into explicit rewrite rules for a constrained decoder.
  • I3. Instead of free decoding, the search space is limited to the canonical form plus a small set of rule-generated plausible variants. This keeps outputs interpretable as named deviations from the norm.
  • I4. The symbol inventory is widened to cover both the target and the interfering system (English plus Italian phones), so interference can be represented rather than forced into the nearest target category.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver-data evaluation could compare the expected/converted analysis, the gold tree and parser output pairwise, separating real annotation divergence from parser mistakes.
  • I2 (learn): Self-induced repair rules could follow the same loop, mining the commonest disagreements of an unconstrained pass and promoting them to explicit, inspectable rules.
  • I3 (tokens): Normalizing noisy or learner text could generate rule-based variant spellings of known lemmas, so each correction carries the rule that explains it.
  • I4 (tree): During UD dialect conversion, a union label inventory could keep source-specific relations visible until they are explicitly mapped, instead of silently collapsing them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 worksSignal Processing · 213 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksSpeech & spoken language · 233 works

Cite

Cite as Nina Hosseini-Kivanani; Roberto Gretter; Marco Matassoni; and Giuseppe Daniele Falavigna. 2021. Experiments of ASR-based mispronunciation detection for children and adult English learners. arXiv:2104.05980.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2104.05980 · arXiv 2104.05980

@misc{matassoni2021experiments,
  title = {{Experiments of ASR-based mispronunciation detection for children and adult English learners}},
  author = {Nina Hosseini-Kivanani; Roberto Gretter; Marco Matassoni; and Giuseppe Daniele Falavigna},
  year = {2021},
  doi = {10.48550/arxiv.2104.05980},
  eprint = {2104.05980},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2104.05980}
}

References

26 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Huensch, “The pronunciation teaching practices of university-level graduate teaching assistants of French and Spanish introductory language courses,” Foreign Language Annals, vol. 52, no. 1, pp. 13–31, 2019. ↗
  2. S. M. McCrocklin, “Pronunciation learner autonomy: The potential of Automatic Speech Recognition,” System, vol. 57, no. February, pp. 25–42, 2016. ↗
  3. M. A. Peabody, “Methods for pronunciation assessment in computer aided language learning,” Ph.D. dissertation, Massachusetts Institute of Technology, 2011. find
  4. G. Fant, Speech sounds and features. The MIT Press, 1973. find
  5. S. M. Witt, “Automatic error detection in pronunciation training: Where we are and where we need to go,” Proceedings of the International Symposium on Automatic Detection of Errors in Pronunciation Training (IS ADEPT), vol. 6, pp. 1–8, 2012. find
  6. A. Neri, C. Cucchiarini, and W. Strik, “Automatic speech recognition for second language learning: how and why it actually works.” in In Proc. ICPhS, 2003, pp. 1157–1160. find
  7. G. Kawai and K. Hirose, “A call system using speech recognition to teach the pronunciation of japanese tokushuhaku,” in STiLL-Speech Technology in Language Learning, 1998. find
  8. K. Zechner, “What did they actually say? Agreement and disagreement among transcribers of non-native spontaneous speech responses in an English proficiency test,” in International Workshop on Speech and Language Technology in Education, 2009. find
  9. R. Downey, H. Farhady, R. Present-Thomas, M. Suzuki, and A. Van Moere, “Evaluation of the usefulness of the versant for english test: A response,” Language Assessment Quarterly, vol. 5, no. 2, pp. 160–167, 2008. find
  10. S. Witt and S. Young, “Phone-level pronunciation scoring and assessment for interactive language learning,” Speech Communication, vol. 30, no. 2-3, pp. 95–108, feb 2000. ↗
  11. H. Ryu, H. Hong, S. Kim, and M. Chung, “Automatic pronunciation assessment of Korean spoken by L2 learners using best feature set selection,” 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2016, 2017. find
  12. S. M. Witt, “Use of speech recognition in computer-assisted language learning,” Ph.D. dissertation, University of Cambridge Cambridge, United Kingdom, 1999. find
  13. O. Franco, Horacio and Neumeyer, Leonardo and Kim, Yoon and Ronen, “Automatic pronunciation scoring for language instruction organization,” in 1997 IEEE international conference on acoustics, speech, and signal processing. IEEE, 1997, pp. 1471–1474. find
  14. L. Kim, Yoon and Franco, Horacio and Neumeyer, “Automatic pronunciation scoring of specific phone segments for language instruction,” in Fifth European Conference on Speech Communication and Technology, 1997. find
  15. M. Eskenazi, “Using automatic speech processing for foreign language pronunciation tutoring: Some issues and a prototype,” Language Learning and Technology, vol. 2, no. 2, pp. 62–76, 1999. ↗
  16. J. Tao, S. Ghaffarzadegan, L. Chen, and K. Zechner, “Exploring deep learning architectures for automatically grading non-native spontaneous speech,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016, pp. 6140–6144. find
  17. P. Boersma and D. Weenink, “Praat: Doing phonetics by computer (Version 6.0. 14),” Retrieved from (last access: 29.04. 2018), 2016. find
  18. D. Jurafsky and J. H. Martin, “N-gram language models,” Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 2018. find
  19. F. Miao, Yajie and Zhang, Hao and Metze, “Speaker adaptive training of deep neural network acoustic models using i-vectors,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 23, no. 11, pp. 1938–1949, 2015. ↗
  20. M. Madikeri, Srikanth and Dey, Subhadeep and Motlicek, Petr and Ferras, “Implementation of the standard i-vector system for the kaldi speech recognition toolkit,” Idiap, Tech. Rep., 2016. find
  21. D. Povey, V. Peddinti, D. Galvez, P. Ghahremani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur, “Purely sequence-trained neural networks for asr based on lattice-free mmi.” in Interspeech, 2016, pp. 2751–2755. find
  22. A. Batliner, M. Blomberg, S. D’Arcy, D. Elenius, D. Giuliani, M. Gerosa, C. Hacker, M. Russell, S. Steidl, and M. Wong, “The PF STAR children’s speech corpus,” in Ninth European Conference on Speech Communication and Technology, 2005. find
  23. W. Menzel, E. Atwell, P. Bonaventura, D. Herron, P. Howarth, R. Morton, and C. Souter, “The ISLE corpus of non-native spoken English,” in 2nd International Conference on Language Resources and Evaluation, LREC 2000, vol. 2. European Language Resources Association, 2000, pp. 957–964. find
  24. S. Browning, “Analysis of Italian children’s English pronunciation,” Report contributed to the EU FP5 PF STAR Project, 2004. find
  25. K. Kawai, Goh and Hirose, “A CALL system using speech recognition to teach the pronunciation of Japanese tokushuhaku,” in STiLL-Speech Technology in Language Learning, 1998. find
  26. L. Chen, K. Zechner, S. Y. Yoon, K. Evanini, X. Wang, A. Loukina, J. Tao, L. Davis, C. M. Lee, M. Ma, R. Mundkowsky, C. Lu, C. W. Leong, and B. Gyawali, “Automated Scoring of Nonnative Speech Using the SpeechRaterSM v. 5.0 Engine,” ETS Research Report Series, vol. 2018, no. 1, pp. 1–31, dec 2018. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hosseini-kivanani-2021-experiments-asr-based-mispronunciation-detection/