Elamur Research⌕
☰
home / papers / 2021 / Weakly-supervised word-level pronunciation error detection in non-native English speech

Weakly-supervised word-level pronunciation error detection in non-native English speech

Paper · 2021 · Daniel Korzekwa, Jaime Lorenzo-Trueba, Thomas Drugman et al.

Summary

The paper addresses automatic detection of mispronounced words in learner (L2) English. Most prior systems need L2 recordings that humans have transcribed phoneme by phoneme, which is slow, costly and sometimes ill-defined. The authors propose WEAKLY-S, which needs only a flag on each word saying whether it was mispronounced. The model has two parts. A Mispronunciations Detection Network (MDN) reads the mel-spectrogram and the expected (canonical) phonemes and outputs a mispronunciation probability per word. A Phoneme Recognition Network (PRN), trained on easily transcribed native (L1) speech, shares the audio encoder and attention decoder with the MDN. This shared second task acts as a regularizer, so the MDN overfits less on the small L2 data. The training set also includes L1 utterances with artificially corrupted transcriptions. Against two baselines, PR and PR-PM, WEAKLY-S raises word-level AUC by 30% on GUT Isle (Polish speakers) and by 21.5% on Isle (German and Italian speakers). This shows that useful pronunciation feedback can be learned without phonetic transcription of learner speech.

Key points

  • Training data combines L1 English from TIMIT and LibriTTS with L2 English from Isle (German and Italian speakers) and GUT Isle (Polish speakers): 102,812 utterances in total. 28 L2 speakers are held out for testing only.
  • The L2 portion has 35,555 words, of which 8,035 are mispronounced. Five native English annotators labeled GUT Isle and assigned each error a severity (MINOR, MEDIUM or MAJOR). For Isle, the word labels were derived automatically: a word counts as wrong if any of its phonemes was marked wrong.
  • The loss adds a word-level mispronunciation term to a phoneme recognition term. L1 examples train both terms, while untranscribed L2 examples train only the first.
  • The training set gains 292,242 synthetic L1 utterances, in which each phoneme of the transcription is swapped for a random one with probability 0.2.
  • The model was built in MxNet, tuned with AutoGluon and uses Location Sensitive Attention. It was pretrained on L1+L2 data, and then only the MDN was fine-tuned on L2. Evaluation reports AUC plus precision at a fixed recall of 0.4, chosen to match baselines that cannot cover the full recall range.

Interesting ideas

  • I1. Supervise the target decision at a coarse level (word right or wrong) and add a cheap, abundant auxiliary task that shares the encoder and attention; this lets scarce weak labels train a deep model without overfitting.
  • I2. Create negative training examples by randomly corrupting the reference labels of clean data, rather than collecting real errors.
  • I3. Derive coarse labels mechanically from finer ones (any bad phoneme makes the word bad), so that corpora annotated at different granularities can be merged into one training set.
  • I4. Condition the detector on the expected canonical sequence, so it judges a deviation from a known target instead of recognizing everything from scratch.
  • I5. When some baselines cannot reach every operating point, compare precision at one matched recall in addition to AUC.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): A frame or answer checker could learn from sentence-level "reading correct or incorrect" flags, while a shared encoder keeps learning from UD parsing on large silver treebanks.
  • I2 (learn): Perturbing gold UD trees (wrong heads, swapped deprels) would yield negative controls and training data for a tree-error detector that proposes self-induced repair rules.
  • I3 (tree): During UD dialect conversion, treebanks with richer subtype labels could be collapsed to a shared coarse layer, which allows them to be pooled for the perceptron parser and gparse.
  • I4 (answer): A student-answer checker could compare the parsed answer against the expected frame or graph and score mismatches per word, which also explains why an answer was marked wrong.
  • I5 (learn): Rule-based components that abstain could be compared with neural ones at equal coverage, which avoids unfair comparisons in evaluation reports.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AmazonSeattle · United States70 works · 39,977 citations🎓Gdańsk University of TechnologyGdansk · Poland3 works · 58 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works Signal Processing · 213 works

🧭 Topics 1 English · 1,373 works

Cite

Cite as Daniel Korzekwa, Jaime Lorenzo-Trueba, Thomas Drugman, Shira Calamaro and Bozena Kostek. 2021. Weakly-supervised word-level pronunciation error detection in non-native English speech. arXiv:2106.03494.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2106.03494 · arXiv 2106.03494

@misc{korzekwa2021weakly,
  title = {{Weakly-supervised word-level pronunciation error detection in non-native English speech}},
  author = {Daniel Korzekwa, Jaime Lorenzo-Trueba, Thomas Drugman, Shira Calamaro and Bozena Kostek},
  year = {2021},
  doi = {10.48550/arxiv.2106.03494},
  eprint = {2106.03494},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2106.03494}
}

References

37 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Neri, O. Mich, M. Gerosa, and D. Giuliani, “The effectiveness of computer assisted pronunciation training for foreign language learning by children,” Computer Assisted Language Learning, vol. 21, no. 5, pp. 393–408, 2008. ↗
  2. C. Tejedor-Garcı́a, D. Escudero, E. Cámara-Arenas, C. González-Ferreras, and V. Cardeñoso-Payo, “Assessing pronunciation improvement in students of english using a controlled computer-assisted pronunciation tool,” IEEE Transactions on Learning Technologies, 2020. ↗
  3. W.-K. Leung, X. Liu, and H. Meng, “Cnn-rnn-ctc based end-to-end mispronunciation detection and diagnosis,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 8132–8136. find
  4. D. Korzekwa, J. Lorenzo-Trueba, S. Zaporowski, S. Calamaro, T. Drugman, and B. Kostek, “Mispronunciation detection in non-native (l2) english with uncertainty modeling,” arXiv preprint arXiv:2101.06396, accepted to ICASSP 2021, 2021. ↗ our card
  5. Z. Zhang, Y. Wang, and J. Yang, “Text-conditioned transformer for automatic pronunciation error detection,” arXiv preprint arXiv:2008.12424, 2020. ↗
  6. H. Li, S. Huang, S. Wang, and B. Xu, “Context-dependent duration modeling with backoff strategy and look-up tables for pronunciation assessment and mispronunciation detection,” in IN-TERSPEECH 2011, 12th Annual Conference of the International Speech Communication Association, Florence, Italy, August 27-31, 2011. ISCA, 2011, pp. 1133–1136. find
  7. K. Li, X. Qian, and H. Meng, “Mispronunciation detection and diagnosis in l2 english speech using multidistribution deep neural networks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 1, pp. 193–207, 2016. ↗
  8. S. Sudhakara, M. K. Ramanathi, C. Yarra, and P. K. Ghosh, “An improved goodness of pronunciation (gop) measure for pronunciation evaluation with dnn-hmm system considering hmm transition probabilities.” in INTERSPEECH, 2019, pp. 954–958. find
  9. S. Cheng, Z. Liu, L. Li, Z. Tang, D. Wang, and T. F. Zheng, “Asr-free pronunciation assessment,” arXiv preprint arXiv:2005.11902, 2020. ↗
  10. H. Franco, L. Neumeyer, Y. Kim, and O. Ronen, “Automatic pronunciation scoring for language instruction,” in 1997 IEEE international conference on acoustics, speech, and signal processing, vol. 2. IEEE, 1997, pp. 1471–1474. find
  11. S. M. Witt and S. J. Young, “Phone-level pronunciation scoring and assessment for interactive language learning,” Speech communication, vol. 30, no. 2-3, pp. 95–108, 2000. ↗
  12. N. Minematsu, “Pronunciation assessment based upon the phonological distortions observed in language learners’ utterances,” in INTERSPEECH, 2004. find
  13. A. M. Harrison, W.-K. Lo, X.-j. Qian, and H. Meng, “Implementation of an extended recognition network for mispronunciation detection and diagnosis in computer-assisted pronunciation training,” in Intl. Workshop on Speech and Language Technology in Education, 2009. find
  14. A. Lee and J. R. Glass, “Pronunciation assessment via a comparison-based system,” in SLaTE, 2013. find
  15. P. Plantinga and E. Fosler-Lussier, “Towards real-time mispronunciation detection in kids’ speech,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 690–696. find
  16. S. Sudhakara, M. K. Ramanathi, C. Yarra, A. Das, and P. Ghosh, “Noise robust goodness of pronunciation measures using teacher’s utterance,” in SLaTE, 2019. find
  17. Y. Xiao, F. K. Soong, and W. Hu, “Paired phone-posteriors approach to esl pronunciation quality assessment,” in bdl, 2018, vol. 1, no. 782d, p. 3. find
  18. M. Nicolao, A. V. Beeston, and T. Hain, “Automatic assessment of english learner pronunciation using discriminative classifiers,” in 2015 IEEE Intl. Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, pp. 5351–5355. find
  19. J. Wang, Y. Qin, Z. Peng, and T. Lee, “Child speech disorder detection with siamese recurrent network using speech attribute features.” in INTERSPEECH, 2019, pp. 3885–3889. find
  20. X. Qian, H. Meng, and F. Soong, “Capturing l2 segmental mispronunciations with joint-sequence models in computer-aided pronunciation training (capt),” in 2010 7th Intl. Symposium on Chinese Spoken Language Processing. IEEE, 2010, pp. 84–88. find
  21. J. Chorowski, D. Bahdanau, K. Cho, and Y. Bengio, “End-to-end continuous speech recognition using attention-based recurrent nn: First results,” arXiv preprint arXiv:1412.1602, 2014. ↗
  22. J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” in Advances in neural information processing systems, 2015, pp. 577– 585. ↗ our card
  23. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2016, pp. 4945– 4949. find
  24. S. B. Needleman and C. D. Wunsch, “A general method applicable to the search for similarities in the amino acid sequence of two proteins,” Journal of molecular biology, vol. 48, no. 3, pp. 443– 453, 1970. ↗
  25. I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in neural information processing systems, 2014, pp. 3104–3112. ↗
  26. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems, 2017, pp. 5998–6008. our card
  27. T. e. a. Chen, “Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems,” arXiv preprint arXiv:1512.01274, 2015. ↗ our card
  28. N. Erickson, J. Mueller, A. Shirkov, H. Zhang, P. Larroy, M. Li, and A. Smola, “Autogluon-tabular: Robust and accurate automl for structured data,” arXiv preprint arXiv:2003.06505, 2020. ↗
  29. J. Guo et al., “Gluoncv and gluonnlp: Deep learning in computer vision and natural language processing.” Journal of Machine Learning Research, vol. 21, no. 23, pp. 1–7, 2020. ↗
  30. J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, and D. S. Pallett, “Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1,” STIN, vol. 93, p. 27403, 1993. find
  31. H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu, “LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech,” in Proc. Interspeech 2019, 2019, pp. 1526–1530. ↗ our card
  32. E. Atwell, P. Howarth, and D. Souter, “The isle corpus: Italian and german spoken learner’s english,” ICAME Journal: Intl. Computer Archive of Modern and Medieval English Journal, vol. 27, pp. 5–18, 2003. find
  33. D. Weber, S. Zaporowski, and D. Korzekwa, “Constructing a dataset of speech recordings with lombard effect,” in 24th IEEE SPA, 2020. find
  34. D. Korzekwa, B. Kostek et al., “Detection of lexical stress errors in non-native (l2) english with data augmentation and attention,” arXiv preprint arXiv:2012.14788, 2020. ↗ our card
  35. A. Graves, “Connectionist temporal classification,” in Supervised Sequence Labelling with Recurrent Neural Networks. Springer, 2012, pp. 61–93. ↗
  36. J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord, “Unsupervised speech representation learning using wavenet autoencoders,” IEEE/ACM transactions on audio, speech, and language processing, vol. 27, no. 12, pp. 2041–2053, 2019. ↗
  37. A. Van Den Oord, O. Vinyals et al., “Neural discrete representation learning,” Advances in Neural Information Processing Systems, vol. 30, pp. 6306–6315, 2017. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/korzekwa-2021-weakly-supervised-word-level-pronunciation/