Elamur Research⌕
☰
home / papers / 2021 / Towards Robust Mispronunciation Detection and Diagnosis for L2 English Learners with Accent-Modulating Methods

Towards Robust Mispronunciation Detection and Diagnosis for L2 English Learners with Accent-Modulating Methods

Paper · 2021 · Shao-Wei Fan Jiang, Bi-Cheng Yan, Tien-Hong Lo et al.

Summary

Computer-assisted pronunciation training needs mispronunciation detection and diagnosis (MDD) that works for learners of English with many different native-language accents. Common end-to-end systems pool speech from all accents into one model, and the authors argue this weakens the model's ability to tell correct pronunciations from errors. They start from a hybrid CTC-ATT phone recognizer and make its encoder accent-aware in two ways. Accent-modulated concatenation (AMC) appends an accent embedding to every encoder frame. Accent-modulated gating (AMG) uses that embedding to compute a sigmoid gate that rescales each frame. Both versions need the accent label at test time, so they also add "soft" variants (AMC-S, AMG-S). In these, an auxiliary accent classifier is trained jointly with the recognizer, and its hidden summary replaces the true label. Accents then only have to be known for the training data. On L2-ARCTIC, with TIMIT as native-speech data, the accent-aware models beat a plain CTC-ATT baseline and a GOP scoring baseline. AMG-S was the strongest, which makes accent-robust diagnosis practical without asking learners for their background.

Key points

  • The base system is a hybrid CTC-ATT model with a shared encoder, attention decoder and CTC branch. Its loss interpolates the CTC and attention log-posteriors with weight α = 0.3.
  • Two fusion mechanisms are proposed:
    • AMC: concatenation of an utterance-level accent embedding with each frame, followed by a linear projection.
    • AMG: a sigmoid gate computed from the frame and accent vector, applied elementwise, followed by ReLU and a projection.
  • The soft variants (AMC-S, AMG-S) add a multi-layer bidirectional GRU accent classifier over the encoder states. It is trained jointly with the recognizer, with weight β balancing the two losses.
  • The data come from two corpora:
    • TIMIT: 462 training speakers, 3696 utterances, 3.15 h of native speech.
    • L2-ARCTIC: 17 training speakers, 2549 utterances, 2.66 h, plus a test set of 6 speakers, 900 utterances and 0.88 h.
  • Evaluation is based on a confusion matrix over correct versus mispronounced phones (CP/MP). For diagnostic accuracy (DAR), the neural models fall roughly between 70.7% and 73.75%. AMG-S is the best of them, though the authors admit DAR is still far from satisfactory.

Interesting ideas

  • I1. Condition every frame on one utterance-level attribute by gating, rather than only concatenating it, so the attribute can amplify or suppress individual feature dimensions.
  • I2. Replace a side label that is unavailable at inference with the hidden state of a jointly trained auxiliary classifier, so labels are needed only for training data.
  • I3. Recognize what was actually produced without constraints, then compare it with the expected prompt to yield typed errors (substitution, deletion, insertion) rather than a single score.
  • I4. The authors warn that merging many sub-populations into one model can blur the distinctions the model needs to make.
  • I5. Pair a monotonic-alignment objective (CTC) with a flexible attention decoder, interpolated by a fixed weight, so the rigid objective stabilizes the flexible one.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): A gate in gparse computed from a treebank/dialect embedding could let one parser adapt its token representations to each UD dialect instead of averaging conventions.
  • I2 (learn): Train a source/genre classifier alongside the parser on silver data, then feed its hidden summary to the parser so unlabeled input text still gets domain-aware analysis.
  • I3 (answer): When grading student answers, align the produced frame/graph against the expected one and report which roles were swapped, missing or extra, keeping feedback explainable.
  • I4 (learn): Before pooling treebanks or registers into one training set, measure whether pooling hurts per-source accuracy, and add explicit source conditioning when it does.
  • I5 (back): During text generation from the meaning graph, an auxiliary monotonic alignment loss between graph order and output tokens could curb dropped or repeated content.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Taiwan Normal UniversityTaipei · Taiwan3 works · 117 citations

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 worksSignal Processing · 213 works

🧭 Topics 1Research subjects and languages the work is about.English · 1,373 works

Cite

Cite as Shao-Wei Fan Jiang, Bi-Cheng Yan, Tien-Hong Lo, Fu-An Chao and Berlin Chen. 2021. Towards Robust Mispronunciation Detection and Diagnosis for L2 English Learners with Accent-Modulating Methods. arXiv:2108.11627.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2108.11627 · arXiv 2108.11627

@misc{jiang2021robust,
  title = {{Towards Robust Mispronunciation Detection and Diagnosis for L2 English Learners with Accent-Modulating Methods}},
  author = {Shao-Wei Fan Jiang, Bi-Cheng Yan, Tien-Hong Lo, Fu-An Chao and Berlin Chen},
  year = {2021},
  doi = {10.48550/arxiv.2108.11627},
  eprint = {2108.11627},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2108.11627}
}

References

30 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K. Evanini and X. Wang, “Automated speech scoring for nonnative middle school students with multiple task types,” in Proceedings of Interspeech, pp. 2435–2439, 2013. find
  2. A. Metallinou and J. Cheng, “Using deep neural networks to improve proficiency assessment for children English language learners,” in Proceedings of Interspeech, pp. 1468–1472, 2014. find
  3. K. Li et al., “Mispronunciation detection and diagnosis in l2 English speech using multidistribution deep neural networks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, 193–207, 2016. ↗
  4. H. Franco et al., “Automatic pronunciation scoring for language instruction,” in Proceedings of ICASSP, pp. 1471–1474, 1997. find
  5. S.M. Witt and S.J.Young, “Phone-level pronunciation scoring and assessment for interactive language learning,” Speech Communication, pp. 95–108, 2000. ↗
  6. S. Sudhakara et al., “An improved goodness of pronunciation (GoP) measure for pronunciation evaluation with DNN-HMM system considering HMM transition probabilities,” in Proceedings of Interspeech, pp. 954–958, 2019. find
  7. Z. Wang et al., “A multi-view approach for Mandarin non-native mispronunciation verification,” in Proceedings of ICASSP, pp. 8079–8083, 2020. ↗
  8. W. K. Lo et al., “Automatic derivation of phonological rules for mispronunciation detection in a computer-assisted pronunciation training system,” in Proceedings of Interspeech, 2010. find
  9. H. Meng et al., “Deriving salient learners’ mispronunciations from cross-language phonological comparisons,” in Proceedings of ASRU, 2007. find
  10. W. Li et al., “Improving non-native mispronunciation detection and enriching diagnostic feedback with DNN-based speech attribute modeling,” in Proceedings of ICASSP, pp. 6135–6139, 2016. find
  11. WK. Leung et al., “CNN-RNN-CTC based end-to-end mispronunciation detection and diagnosis,” in Proceedings of ICASSP, pp. 8132–8136, 2019. find
  12. Y. Feng et al., “SED-MDD: Towards sentence dependent end-to-end mispronunciation detection and diagnosis,” in Proceedings of ICASSP, pp. 3492–3496, 2020. find
  13. B. C. Yan et al., “An end-to-end mispronunciation detection system for L2 English speech leveraging novel anti-phone modeling,” in Proceedings of Interspeech, pp. 3032–3036, 2020.
  14. A. Jain et al., “A multi-accent acoustic model using mixture of experts for speech recognition,” in Proceedings of Interspeech, pp. 779–783, 2019. find
  15. V. Pratap et al., “Massively multilingual ASR: 50 languages, 1 model, 1 billion parameters,” in Proceedings of Interspeech, pp. 4751–4755,2020. find
  16. D. Vergyri et al., “Automatic speech recognition of multiple accented English data,” in Proceedings of Interspeech, 2010. find
  17. S. Tong et al., “Cross-lingual adaptation of a CTC-based multilingual acoustic model,” Speech Communication, vol. 104, pp. 39–46, 2018. ↗
  18. X. Yang et al., “Joint modeling of accents and acoustics for multi-accent speech recognition,” in Proceedings of ICASSP, pp.1–5, 2018.
  19. A. Jain et al., “Improved accented speech recognition using accent embeddings and multi-task learning,” in Proceedings of Interspeech, pp. 2454–2458, 2018. find
  20. S. Punjabi et al., “Streaming end-to-end bilingual ASR systems with joint language identification,” arXiv preprint arXiv:2007.03900, 2020. ↗
  21. S. Watanabe et al., “Hybrid CTC/attention architecture for end-to-end speech recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017. ↗
  22. A. Graves et al., “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of ICML, vol. 148, 2006. find
  23. G. Zhao et al., “L2-ARCTIC: A non-native English speech corpus,” in Proceedings of Interspeech, pp. 2783–2787, 2018. find
  24. J.S. Garofolo et al., “DARPA TIMIT: acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1–1.1,” STIN, vol. 93, 1993. find
  25. G. Pundak et al., “Deep context: end-to-end contextual speech recognition,” in Proceedings of SLT, pp. 418–425, 2018. find
  26. B.-C. Yan and B. Chen, “End-to-end mispronunciation detection and diagnosis from raw waveforms,” in Proceedings of EUSIPCO, 2021. find
  27. M. Wu et al., “Transformer based end-to-end mispronunciation detection and diagnosis,” in Proceedings of Interspeech, pp. 3954–3958, 2021. find
  28. X. Xu et al., “Explore wav2vec 2.0 for mispronunciation detection,” in Proceedings of Interspeech, pp. 4428–4432, 2021. find
  29. T.-H. Lo et al., “Improving end-to-end modeling for mispronunciation detection with effective augmentation mechanisms,” in Proceedings of APSIPA ASC, 2021. ↗
  30. F.-A. Chao et al., “TENET: A time-reversal enhancement network for noise-robust ASR,” in Proceedings of ASRU, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jiang-2021-towards-robust-mispronunciation-detection-diagnosis/