Elamur Research⌕
☰
home / papers / 2021 / Exploring Non-Autoregressive End-To-End Neural Modeling For English Mispronunciation Detection And Diagnosis

Exploring Non-Autoregressive End-To-End Neural Modeling For English Mispronunciation Detection And Diagnosis

Paper · 2021 · Hsin-Wei Wang, Bi-Cheng Yan, Hsuan-Sheng Chiu et al.

Summary

The paper targets computer-assisted pronunciation training. The task is to find phone-level mispronunciations in L2 English speech and say what went wrong (MD&D). Common end-to-end systems such as CTC-ATT decode phones left to right with beam search. That makes them slow, and they also suffer when non-native training data is scarce.

The authors replace the autoregressive recognizer with a non-autoregressive "dictation model" built on Mask-CTC. A Conformer encoder with CTC produces an initial phone string. Phones with low confidence are then masked and refilled in parallel by a conditional masked language model that sees context on both sides.

On top of this they add a separate "pronunciation model". It takes the dictated phones and their confidence scores and checks them against the canonical phones of the prompt text, using a prompt-modulated gating mechanism. The system is trained on L2-ARCTIC plus TIMIT.

With wav2vec 2.0 features, the dictation model comes close to the phone error rate of CTC-ATT at a small fraction of its real-time factor. This shows that fast, deployable feedback for learners is feasible.

Key points

  • Data. L2-ARCTIC covers 24 non-native speakers with six mother tongues: Hindi, Korean, Mandarin, Spanish, Arabic and Vietnamese.

    • Split by speakers: 17 train, 1 dev and 6 test, with 900 test utterances.
    • TIMIT native speech was pooled into training.

  • Phone inventory. Training uses 48 phones, each paired with an "anti-phone" that marks distorted, in-between realizations. At test time the inventory is collapsed to 39 phones to stay comparable with earlier work.

  • Dictation model. It is trained with a joint CTC plus mask-predict objective, using Mask-CTC with a Conformer encoder and a Transformer-based CMLM.

  • Pronunciation model. Each phone embedding is concatenated with its confidence score and passed through a Transformer encoder. A GRU turns the canonical prompt phones into one embedding, which gates every phone vector through sigmoid gating and ReLU. A final FFN with softmax makes the decision.

  • Speed and accuracy. On L2-ARCTIC, accuracy is measured as PER (the share of phones recognized wrongly) and speed as RTF (the real-time factor):

    ModelPERRTF
    CTC-ATT, beam 115.2%3.10
    CTC-ATT, beam 1014.9%13.85
    Mask-CTC, FBANK features22.6%0.05
    Mask-CTC, wav2vec 2.0 features15.9%0.05

    The choice of acoustic features clearly matters for accuracy, while speed stays the same.

Interesting ideas

  • I1. Draft, mask, refill. Make a fast first draft of the whole sequence, mask only the low-confidence items, and refill them in parallel from confident context on both sides; this avoids slow sequential decoding while still fixing weak spots.
  • I2. Separate recognizing from judging. Instead of directly diffing the recognized output against the reference, a second model decides correctness from the output, its confidence and the reference, which makes it robust to a recognizer that overfits.
  • I3. Anti-labels for in-between errors. Each canonical class gets a companion label for "distorted version of X", so non-categorical errors are not forced into a wrong class.
  • I4. Reference-conditioned gating. A single summary vector of the expected reference gates every element of the observed sequence, so each item is judged relative to what should have been there.
  • I5. Train fine, evaluate coarse. Train on a fine-grained label set but collapse it to a standard coarser set for evaluation, keeping detail internally while staying comparable to prior work.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could emit a full parse, mask arcs or labels below a confidence threshold, and re-predict only those given the confident arcs, logging each revision as an explainable repair.
  • I2 (learn): a judge that sees a parse, per-arc confidence and the witness parsers' outputs could decide which arcs enter silver data, rather than relying on plain agreement counts.
  • I3 (tokens): the tagger and morphology could carry an "off-form of X" tag for misspellings, dialect or learner forms, keeping the intended lemma while flagging the deviation.
  • I4 (answer): when grading student answers in school-course tests, the frames of the expected answer could gate how each extracted student frame is scored, with the gate values kept as an explanation.
  • I5 (learn): train on Mova's richer internal relation labels but report on collapsed plain UD labels, so benchmark numbers stay comparable without losing internal distinctions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Taiwan Normal UniversityTaipei · Taiwan5 works · 119 citations

Also on the byline: Chunghwa Telecom.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Hsin-Wei Wang, Bi-Cheng Yan, Hsuan-Sheng Chiu, Yung-Chang Hsu and Berlin Chen. 2021. Exploring Non-Autoregressive End-To-End Neural Modeling For English Mispronunciation Detection And Diagnosis. arXiv:2111.00844.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2111.00844 · arXiv 2111.00844

@misc{wang2021exploring,
  title = {{Exploring Non-Autoregressive End-To-End Neural Modeling For English Mispronunciation Detection And Diagnosis}},
  author = {Hsin-Wei Wang, Bi-Cheng Yan, Hsuan-Sheng Chiu, Yung-Chang Hsu and Berlin Chen},
  year = {2021},
  doi = {10.48550/arxiv.2111.00844},
  eprint = {2111.00844},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2111.00844}
}

References

30 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K. Evanini and X. Wang, “Automated speech scoring for Non-native middle school students with multiple task types,” in Proceedings of INTERSPEECH, pp. 2435–2439, 2013. find
  2. A. Metallinou and J. Cheng, “Using deep neural networks to improve proficiency assessment for children English language learners,” in Proceedings of INTERSPEECH, pp. 1468–1472, 2014. find
  3. H. Li, S. Huang, S. Wang, and B. Xu, “Context-dependent duration modeling with backoff strategy and look-up tables for pronunciation assessment and mispronunciation detection,” in Proceedings of INTERSPEECH, pp. 1133–1136, 2011. find
  4. K. Li, X. Qian, and H. Meng, “Mispronunciation detection and diagnosis in l2 English speech using multidistribution deep neural networks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 1, pp. 193–207, 2017. ↗
  5. S. Sudhakara, M. K. Ramanathi, C. Yarra, and P. K. Ghosh, “An improved goodness of pronunciation (GOP) measure for pronunciation evaluation with DNN-HMM system considering hmm transition probabilities.” in Proceedings of INTERSPEECH, 2019, pp. 954–958. find
  6. S. Cheng, Z. Liu, L. Li, Z. Tang, D. Wang, and T. F. Zheng, “ASR-free pronunciation assessment,” in arXiv preprint arXiv:2005.11902, 2020. ↗
  7. Z. Wang, John H.L. Hansen, and Y. Xie, “A multi-view approach for Mandarin non-native mispronunciation verification,” in Proceedings of ICASSP, pp. 8079–8083, 2020. ↗
  8. N. Minematsu, “Pronunciation assessment based upon the phonological distortions observed in language learners' utterances,” in Proceedings of INTERSPEECH, pp. 1669–1672, 2004. find
  9. A. Lee, and J. R. Glass, “Pronunciation assessment via a comparison-based system,” in Proceedings of IEEE SLaTE Workshop, pp. 122–126, 2013. find
  10. W. Leung, X. Liu, and H. Meng, “CNN-RNN-CTC based end-to-end mispronunciation detection and diagnosis,” in Proceedings of ICASSP, pp. 8132–8136, 2019. find
  11. K. Fu, J. Lin, D. Ke, Y. Xie, J. Zhang, and B. Lin, “A full text-dependent end to end mispronunciation detection and diagnosis with easy data augmentation techniques,” in arXiv preprint arXiv:2104.08428, 2021. ↗
  12. Y. Feng, G. Fu, Q. Chen, and K. Chen, “SED-MDD: Towards sentence dependent end-to-end mispronunciation detection and diagnosis,” in Proceedings of ICASSP, pp. 3492–3496, 2020. find
  13. T.-H. Lo, S.-Y. Weng, H.-J. Chang, and B. Chen, “An effective end-to-end modeling approach for mispronunciation detection,” in Proceedings of INTERSPEECH, pp. 3027–3031, 2020. find
  14. B.-C. Yan, M.-C. Wu, H.-T. Hung, and B. Chen, “An end-to-end mispronunciation detection system for l2 English speech leveraging novel anti-phone modeling,” in Proceedings of INTERSPEECH, pp. 3032-3036, 2020.
  15. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist Temporal Classification: Labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of ICML, pp. 369–376, 2006. find
  16. J. Chorowski, D. Bahdanau, K. Cho, and Y. Bengio, “End-to-end continuous speech recognition using attention-based recurrent nn: First results,” in arXiv preprint arXiv:1412.1602, 2014. ↗
  17. J. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” in Proceedings of NIPS, pp. 577–585, 2015. ↗
  18. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in Proceedings of ICASSP, pp. 4945-4949, 2016. find
  19. S. Kim, T. Hori, and S. Watanabe, “Joint CTC-attention based end-to-end speech recognition using multi-task learning,” in Proceedings of ICASSP, pp. 4835–4839, 2017. find
  20. A. Vaswani, N. Shazeer, N. Parmar, and J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser and I. Polosukhin, “Attention is all you need,” in Proceedings of NIPS, pp. 5998–6008., 2017.
  21. Y. Higuchi, S. Watanabe, N. Chen, T. Ogawa, T. Kobayashi, “Mask CTC: Non-autoregressive end-to-end ASR with CTC and mask predict,” in Proceedings of INTERSPEECH, pp. 3655–3659, 2020. find
  22. Y. Bai, J. Yi, J. Tao, Z. Tian, Z. Wen, and S. Zhang, “Listen attentively, and spell once: whole sentence generation via a non-autoregressive architecture for low-latency speech recognition,” in Proceedings of INTERSPEECH, pp. 3381– 3385, 2020. find
  23. E. A. Chi, J. Salazar, and K. Kirchhoff, “Align-Refine: Non-autoregressive speech recognition via iterative realignment,” in Proceedings of NAACL, pp. 1920–1927, 2021. ↗
  24. A. Gulati, J. Qin, C. C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented Transformer for speech recognition,” in Proceedings of INTERSPEECH, pp. 4036–5040, 2020.
  25. M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer, “Mask-Predict: Parallel decoding of conditional masked language models,” in Proceedings of EMNLP-IJCNLP, pp. 6112–6121, 2019.
  26. G. Zhao, S. Sonsaat, A. O. Silpachai, I. Lucic, E. Chukharev-Hudilainen, J. Levis, and R. Gutierrez-Osuna, “L2-ARCTIC: A non-native English speech corpus,” Perception Sensing Instrumentation Lab, 2018. find
  27. J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, and D. S. Pallett, “DARPA TIMIT acoustic-phonetic continuous speech corpus CD-Rom: NIST speech disc 1-1.1,” NASA STI/Recon Technical Report, vol. 93, pp. 27403, 1993. find
  28. A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” arXiv preprint arXiv:2006.11477, 2020. ↗
  29. S.-W. Fan Jiang, B.-C. Yan, T.-H. Lo, F.-A. Chao, and B. Chen, “Towards robust mispronunciation detection and diagnosis for L2 English learners with accent-modulating methods,” in Proceedings of ASRU, pp. 1065–1070, 2021.
  30. T.-H. Lo, Y.-T. Sung, and B. Chen, “Improving end-to-end modeling for mispronunciation detection with effective augmentation mechanisms,” in Proceedings of APSIPA ASC, pp. 1049–1055, 2021. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2021-exploring-non-autoregressive-end-end/