Elamur Research⌕
☰
home / papers / 2021 / Mispronunciation Detection in Non-native (L2) English with Uncertainty Modeling

Mispronunciation Detection in Non-native (L2) English with Uncertainty Modeling

Paper · 2021 · Daniel Korzekwa, Jaime Lorenzo-Trueba, Szymon Zaporowski et al.

Summary

Computer Assisted Pronunciation Training (CAPT) systems usually transcribe a learner's phonemes and flag any mismatch with one canonical native pronunciation. This produces many false alarms for two reasons. Phoneme recognition is error-prone even on native speech, and native speakers themselves pronounce many words in more than one acceptable way. The authors build a three-part pipeline to address both problems:

  • a CNN-GRU-CTC Phoneme Recognizer (PR) that outputs several beam hypotheses with phoneme posteriors;
  • a seq2seq Pronunciation Model (PM) that estimates how likely a native speaker is to produce the recognized phonemes, given the canonical ones;
  • a Pronunciation Error Detector (PED) that aligns the sequences and scores each word.

Training used about 125 hours of L1 and L2 English from TIMIT, LibriTTS, Isle and GUT Isle. Evaluation used held-out German, Italian and Polish speakers. At matched recall, word-level precision rose by up to 18% relative to a CTC baseline in the style of Leung et al. Modeling native pronunciation variability helped precision more than modeling recognizer uncertainty. This matters because learners are less likely to be confused by spurious corrections.

Key points

  • Training data: 125.28 h of speech, 983 speakers and 102,812 sentences. Phonetic transcriptions came from an Amazon proprietary grapheme-to-phoneme model.
  • Annotation: five native American English annotators marked word-level errors. Isle has 3,624 mispronounced words out of 13,191, and GUT Isle has 1,046 out of 5,064.
  • Test set: 28 held-out L2 speakers, namely 11 Italian and 11 German speakers from Isle and 6 Polish speakers from GUT Isle.
  • Isle results: at a recall of about 40%, PR-PM reaches 54.20% precision versus 49.39% for PR-LIK.
  • GUT Isle results: at the same recall, PR-PM reaches 61.21% versus 54.91% for PR-LIK.
  • Implementation: N = 4 recognition hypotheses were used. Alignment relies on the Needleman-Wunsch algorithm. Everything was built in MxNet.

Interesting ideas

  • I1. The PM's training pairs are produced by running native speech through the recognizer, so the model learns to treat the recognizer's habitual mistakes as normal rather than as errors.
  • I2. Instead of comparing against a single reference, the system scores how plausible an observed variant is under a model of acceptable realizations.
  • I3. A word is flagged only when every one of the N top hypotheses agrees that it is wrong, which trades recall for precision in a controllable way.
  • I4. The two sequences are aligned with Needleman-Wunsch, and a word's error score comes from its weakest aligned unit, which yields an explainable, localized verdict.
  • I5. The ablation design switches off one component at a time (no PM, then also no posteriors) and compares the variants at matched recall. This shows which assumption actually matters.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): train the self-induced repair rules on our own parser's output for text known to be well-formed, so the rules stop "fixing" the parser's systematic quirks.
  • I2 (answer): when grading school-course or word-problem answers, score each student answer by its likelihood under a model of valid phrasings rather than by exact match to one key, which would cut false "wrong" verdicts.
  • I3 (learn): accept a silver UD arc or label as an error only when all witness parsers' top analyses disagree with it, which keeps silver data conservative.
  • I4 (back): align original and regenerated tokens in the text → graph → text compressor and blame the word with the worst-matching unit, pinpointing where meaning was lost.
  • I5 (learn): evaluate pipeline changes by disabling one stage's uncertainty at a time and comparing precision at a fixed recall, as a principled negative-control setup.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AmazonSeattle · United States70 works · 39,977 citations🎓Gdańsk University of TechnologyGdansk · Poland3 works · 58 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works Signal Processing · 213 works

🧭 Topics 1 English · 1,373 works

Cite

Cite as Daniel Korzekwa, Jaime Lorenzo-Trueba, Szymon Zaporowski, Shira Calamaro, Thomas Drugman and Bozena Kostek. 2021. Mispronunciation Detection in Non-native (L2) English with Uncertainty Modeling. arXiv:2101.06396.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2101.06396 · arXiv 2101.06396

@misc{korzekwa2021mispronunciation,
  title = {{Mispronunciation Detection in Non-native (L2) English with Uncertainty Modeling}},
  author = {Daniel Korzekwa, Jaime Lorenzo-Trueba, Szymon Zaporowski, Shira Calamaro, Thomas Drugman and Bozena Kostek},
  year = {2021},
  doi = {10.48550/arxiv.2101.06396},
  eprint = {2101.06396},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2101.06396}
}

References

22 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Neri, O. Mich, M. Gerosa, and D. Giuliani, “The effectiveness of computer assisted pronunciation training for foreign language learning by children,” Computer Assisted Language Learning, vol. 21, no. 5, pp. 393– 408, 2008. ↗
  2. C. Tejedor-Garcı́a, D. Escudero, E. Cámara-Arenas, C. González-Ferreras, and V. Cardeñoso-Payo, “Assessing pronunciation improvement in students of english using a controlled computer-assisted pronunciation tool,” IEEE Transactions on Learning Technologies, 2020. ↗
  3. S. M. Witt and S. J. Young, “Phone-level pronunciation scoring and assessment for interactive language learning,” Speech communication, vol. 30, no. 2-3, pp. 95– 108, 2000. ↗
  4. K. Li, X. Qian, and H. Meng, “Mispronunciation detection and diagnosis in l2 english speech using multidistribution deep neural networks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 1, pp. 193–207, 2016. ↗
  5. S. Sudhakara, M. K. Ramanathi, C. Yarra, and P. K. Ghosh, “An improved goodness of pronunciation (gop) measure for pronunciation evaluation with dnn-hmm system considering hmm transition probabilities.,” in INTERSPEECH, 2019, pp. 954–958. find
  6. W. Leung, X. Liu, and H. Meng, “Cnn-rnn-ctc based end-to-end mispronunciation detection and diagnosis,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 8132–8136. find
  7. J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” in Advances in neural information processing systems, 2015, pp. 577–585. ↗ our card
  8. S. Cheng et al., “Asr-free pronunciation assessment,” arXiv preprint arXiv:2005.11902, 2020. ↗
  9. A. M. Harrison, W. Lo, X. Qian, and H. Meng, “Implementation of an extended recognition network for mispronunciation detection and diagnosis in computer-assisted pronunciation training,” in Intl. Workshop on Speech and Language Technology in Education, 2009. find
  10. Y. Xiao and W. Soong, F. K .and Hu, “Paired phone-posteriors approach to esl pronunciation quality assessment,” in bdl, vol. 1, p. 3. 2018. find
  11. M. Nicolao, A. V. Beeston, and T. Hain, “Automatic assessment of english learner pronunciation using discriminative classifiers,” in 2015 IEEE Intl. Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, pp. 5351–5355. find
  12. J. Wang, Y. Qin, Z. Peng, and T. Lee, “Child speech disorder detection with siamese recurrent network using speech attribute features.,” in INTERSPEECH, 2019, pp. 3885–3889. find
  13. X. Qian, H. Meng, and F. Soong, “Capturing l2 segmental mispronunciations with joint-sequence models in computer-aided pronunciation training (capt),” in 2010 7th Intl. Symposium on Chinese Spoken Language Processing. IEEE, 2010, pp. 84–88. find
  14. A. Graves, A. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in 2013 IEEE Intl. conference on acoustics, speech and signal processing. IEEE, 2013, pp. 6645–6649. ↗
  15. I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in neural information processing systems, 2014, pp. 3104– 3112. ↗
  16. T. et al. Chen, “Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems,” arXiv preprint arXiv:1512.01274, 2015. ↗ our card
  17. S. B. Needleman and C. D. Wunsch, “A general method applicable to the search for similarities in the amino acid sequence of two proteins,” Journal of molecular biology, vol. 48, no. 3, pp. 443–453, 1970. ↗
  18. J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, and David S. Pallett, “Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1,” STIN, vol. 93, pp. 27403, 1993. find
  19. H. Zen et al., “LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech,” in Proc. Interspeech 2019, 2019, pp. 1526–1530. ↗ our card
  20. E. S. Atwell, P. A. Howarth, and D. C. Souter, “The isle corpus: Italian and german spoken learner’s english,” ICAME Journal: Intl. Computer Archive of Modern and Medieval English Journal, vol. 27, pp. 5–18, 2003. find
  21. D. Weber, S. Zaporowski, and D. Korzekwa, “Constructing a dataset of speech recordings with lombard effect,” in 24th IEEE SPA, 2020. find
  22. A. E. Hieke, “Linking as a marker of fluent speech,” Language and Speech, vol. 27, no. 4, pp. 343–354, 1984. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/korzekwa-2021-mispronunciation-detection-non-native-l2/