Elamur Research⌕
☰
home / papers / 2020 / An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling

An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling

Paper · 2020 · Bi-Cheng Yan, Meng-Che Wu, Hsiao-Tsung Hung et al.

Summary

Computer-assisted pronunciation training (CAPT) depends on mispronunciation detection and diagnosis (MDD). Most MDD systems only handle categorical errors, where a learner substitutes, deletes or inserts a whole phone. They miss distortions, where the sound produced lies somewhere between the target L2 phone and an L1 phone. The authors build an end-to-end MDD system on a hybrid CTC-Attention recognizer: a BLSTM encoder, an attention-based ULSTM decoder, and an auxiliary CTC branch, trained with an interpolated loss. They double the phone inventory by adding an "anti-phone" (marked with #) for every canonical phone, so the model can label a distorted realization of a phone without forcing it into another category. Two further steps support this. Training data is augmented by randomly swapping reference labels for anti-phones in correctly pronounced utterances, and training proceeds from a native-speech model to an accented one through transfer learning, with no hand-written phonological rules. On L2-ARCTIC the system reaches 56.02% F1, well above GOP and a coarse single-label variant. The work matters because it shows distortion errors can be represented inside an end-to-end model.

Key points

  • Data:
    • Native-speech pretraining uses TIMIT plus a small part of Librispeech.
    • Accented training and evaluation use the L2-ARCTIC non-native English corpus.
    • The label-shuffling augmentation is applied only to L2-ARCTIC utterances that contain no pronunciation errors.
  • Mispronunciation detection F1:
    • CTC-ATT (Anti): 56.02%, with precision 46.57% and recall 70.28%.
    • CTC-ATT (Unk), which uses one coarse anti-label: 44.97%.
    • GOP: 28.31%.
    • These gaps are the 11.05 and 27.71 points reported in the abstract.
  • Architecture comparison with anti-phones:
    • Detection F1 is 53.52% for CTC alone, 52.25% for Attention alone and 56.02% for the hybrid.
    • Diagnosis accuracy (DAR) is 32.46%, 37.02% and 40.66% respectively.
  • Error breakdown:
    • The ground truth contains 771 non-categorical and 3,310 categorical errors.
    • Phone-specific anti-phones correctly diagnose 9.4% (73) of non-categorical and 33.02% (1,093) of categorical errors.
    • The coarse Unk variant manages 8.4% (65) and 19.63% (650).
  • The authors note that diagnosis remains much harder than detection, and that distortion errors are still largely unsolved.

Interesting ideas

  • I1. Give every canonical label its own "near-miss" twin (anti-label), so that a deviant input is marked as a distorted version of a specific category rather than forced into a wrong category or a single catch-all bucket.
  • I2. Create training examples for the "deviant" classes by randomly relabeling parts of clean, correct data with anti-labels at a fixed probability, which yields abundant synthetic negatives without needing annotated errors.
  • I3. Compare phone-specific anti-labels against one shared unknown label, and break results down by error type; this isolates whether fine-grained rejection classes actually pay off.
  • I4. Train in stages: first a model on clean "standard" data, then reuse its encoder to start training on the shifted domain, instead of hand-writing rules for every variety.
  • I5. Report detection (is something wrong?) and diagnosis (what exactly is wrong?) as separate metrics, since a system can do well on the first and poorly on the second.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The tagger could emit labels such as "#VERB" or a "#lemma" flag for nonstandard or misspelled words, saying which category was intended while marking that the form deviates from it.
  • I2 (learn): Silver treebanks could be duplicated with randomly flipped "anti-relation" labels to train a module that flags suspicious arcs, giving cheap negative controls for evaluation.
  • I3 (learn): Whenever a fallback label is added (an unknown frame, an unknown role), it should be tested against per-category fallbacks and scored separately for categorical and distortion-like failures.
  • I4 (tree): gparse could be pretrained on standard UD treebanks, and the encoder then reused to start training on dialect-converted or learner-English data, instead of writing conversion rules for every dialect.
  • I5 (answer): When grading student answers, Mova could score "found an error" separately from "named the right error", making it explicit how well its explanations hold up.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Taiwan Normal UniversityTaipei · Taiwan3 works · 117 citations🏢AsusTaipei · Taiwan2 works · 1,627 citations

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 worksSignal Processing · 213 works

🧭 Topics 1Research subjects and languages the work is about.English · 1,373 works

Cite

Cite as Bi-Cheng Yan, Meng-Che Wu, Hsiao-Tsung Hung and Berlin Chen. 2020. An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling. arXiv:2005.11950.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2005.11950 · arXiv 2005.11950

@misc{yan2020end,
  title = {{An End-to-End Mispronunciation Detection System for L2 English Speech Leveraging Novel Anti-Phone Modeling}},
  author = {Bi-Cheng Yan, Meng-Che Wu, Hsiao-Tsung Hung and Berlin Chen},
  year = {2020},
  doi = {10.48550/arxiv.2005.11950},
  eprint = {2005.11950},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.11950}
}

References

19 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. M. Witt and S. J. Young, “Phone-level pronunciation scoring and assessment for interactive language learning,” Speech Communication, vol. 30, pp. 95–108, 2000. ↗
  2. W. Hu, et al, “Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiers,” Speech Communication, vol. 67, pp.154–166, 2015. ↗
  3. S. Sudhakara, et al, “An improved goodness of pronunciation (GoP) measure for pronunciation evaluation with DNN-HMM system considering HMM transition probabilities,” Proceedings of the INTERSPEECH, pp. 954–958, 2019. find
  4. W. Li, et al, “Improving non-native mispronunciation detection and enriching diagnostic feedback with DNN-based speech attribute modeling,” Proceedings of the ICASSP, pp. 6135–6139, 2016. find
  5. W. Lo, S. Zhang, and H. Meng, “Automatic derivation of phonological rules for mispronunciation detection in a computer-assisted pronunciation training system,” Proceedings of the INTERSPEECH, pp.765–768, 2010. find
  6. X. Qian, F. K. Soong, and H. Meng, “Discriminative acoustic model for improving mispronunciation detection and diagnosis in computer-aided pronunciation training (CAPT),” in International Speech Communication Association, 2010. find
  7. W. Leung, X. Liu and H. Meng, “CNN-RNN-CTC Based End-to-end Mispronunciation Detection and Diagnosis,” Proceedings of the ICASSP, pp. 8132–8136, 2019. find
  8. Y. Feng, et al, “SED-MDD: towards sentence dependent end-to-end mispronunciation detection and diagnosis,” Proceedings of the ICASSP, pp. 3492-3496, 2020 find
  9. L. Chen, et al, “End-to-End neural network based automated speech scoring,” Proceedings of the ICASSP, pp. 6234–6238, 2018. find
  10. S. Mao, et al, “Unsupervised Discovery of an Extended Phoneme Set in L2 English Speech for Mispronunciation Detection and Diagnosis,” Proceedings of the ICASSP, pp. 6244–6248, 2018. find
  11. X. Li, et al, “Unsupervised discovery of non-native phonetic patterns in L2 English speech for mispronunciation detection and diagnosis” Proceedings of the INTERSPEECH, pp. 2554–2558, 2018. find
  12. S. Watanabe, et al, “Hybrid CTC/attention architecture for end-to-end speech recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017. ↗
  13. O. Ronen, L. Neumeyer, and H. Franco, “Automatic detection of mispronunciation for language instruction,” Proceedings of Speech Communication and Technology, pp. 649–652, 1997. find
  14. J. K. Chorowski, et al, “Attention-based models for speech recognition,” Proceedings of NIPS, pp. 577–585, 2015. ↗
  15. G. Zhao, et al. “L2-ARCTIC: A Non-native English Speech Corpus,” Proceedings of the INTERSPEECH, pp. 2783–2787, 2018. find
  16. V. Panayotov, et al, “Librispeech: an asr corpus based on public domain audio books,” Proceedings of the ICASSP, pp. 5206–5210, 2015. find
  17. J. Cho, et al, “Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling,” Proceedings of the SLT, pp. 521–527, 2018. find
  18. K. Li, X. Qian and H. Meng, “Mispronunciation Detection and Diagnosis in L2 English Speech Using Multidistribution Deep Neural Networks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, 193–207, 2016. ↗
  19. S. Mao, et al, “Applying multitask learning to acoustic-phonemic model for mispronunciation detection and diagnosis in L2 English speech,” Proceedings of the ICASSP, pp. 6254–6258, 2018. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yan-2020-end-end-mispronunciation-detection-system/