Elamur Research⌕
☰
home / papers / 2022 / Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment

Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment

Paper · 2022 · Yuan Gong, Ziyi Chen, Iek-Heng Chu et al.

Summary

Tools that score the pronunciation of English learners usually predict a single quality, most often phoneme accuracy, at a single level of detail. Gong et al. instead train one model that predicts eight labels together. These cover phoneme accuracy, word accuracy, word stress and word total, and five utterance-level aspects: accuracy, fluency, completeness, prosody and total.

The model is GOPT, a small Transformer encoder. Its input is a sequence of Goodness of Pronunciation (GOP) features. These come from a public Kaldi TDNN-F acoustic model trained on Librispeech and are force-aligned to the known canonical transcription. Each phoneme position also receives an embedding of the expected phoneme. Five learnable [cls] tokens supply the utterance-level outputs, and every label has its own regression head, so all granularities are learned jointly.

On speechocean762, GOPT beats random forest, SVR, deep-feature transfer and a matched LSTM on most phoneme- and utterance-level metrics. The authors also give the first word-level results on this benchmark. The work matters for two reasons: it runs on fully public resources, and it shows that joint training helps each individual task rather than only bundling them together.

Key points

  • Data: speechocean762 has 5,000 utterances from 250 non-native speakers, split 2,500/2,500 between training and test. Five experts annotated every score. Word and utterance scores (0–10) were rescaled to match the phoneme range of 0–2.
  • Input features:
    • GOP features are 84-dimensional: log phone posterior plus log posterior ratio over 42 phones.
    • They are projected to 24 dimensions and summed with a canonical-phoneme embedding and a positional embedding.
    • The encoder has 3 layers and width 24.
  • Phoneme results (Librispeech acoustic model):
    • GOPT reaches MSE 0.085 and PCC 0.612.
    • SVR scores 0.160 / 0.450, RF scores 0.130 / 0.440, and the matched LSTM scores 0.089 / 0.591.
  • Utterance and word results:
    • Utterance PCC is 0.742 for total, 0.753 for fluency and 0.760 for prosody.
    • Utterance completeness stays weak at 0.155.
    • Word-level PCC is 0.533 for accuracy, 0.291 for stress and 0.549 for total.
  • Ablations and robustness:
    • Joint training beats single-task training.
    • Removing the canonical-phoneme embedding hurts every task.
    • Making the model wider or deeper brings no gain.
    • GOPT improves over SVR with every acoustic model tested, including the PAII internal models PAII-A and PAII-B.

Interesting ideas

  • I1. Rather than building a hierarchy to pool phonemes into an utterance, GOPT prepends one trainable [cls] token per utterance-level label and lets attention link them to the sequence; this is a cheap, flat way to get several sentence-level outputs from a single encoder.
  • I2. The loss is averaged inside each granularity and the granularity losses are then summed with equal weight, so levels with many labels or many positions do not drown out the others.
  • I3. Word-level labels are copied down to each phoneme during training and averaged back over the word's phonemes at inference, so the unit of labelling is decoupled from the unit the model sees.
  • I4. Adding an embedding of what should be there (the canonical phoneme) to the measured evidence proved essential, because the same measurement means different things for different expected units.
  • I5. The comparison is kept fair: every baseline uses the same acoustic model and features, the LSTM matches GOPT in depth and width, each result is averaged over five seeds, and PCC is used because the labels lean towards high scores.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could prepend a few aspect tokens to predict sentence-level properties (mood, root type, clause count) alongside arcs, giving a sanity signal that is checked against the tree.
  • I2 (learn): when training joint heads for tags, arcs and sentence labels, this per-level averaging is a simple default that keeps the rare sentence-level objective from being swamped by token-level losses.
  • I3 (tokens): a character- or subword-level morphology model could learn word-level features the same way, broadcasting the word label to pieces and averaging their predictions back into one explainable word decision.
  • I4 (tree): feeding the lemma or expected UPOS as an explicit identity embedding next to contextual features could help the parser judge arcs per word type, especially with small treebanks.
  • I5 (learn): evaluation of parser or reader variants should likewise fix the shared inputs, match model capacity and report the spread across seeds, so that a claimed gain is not an artefact of setup or luck.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States211 works · 69,000 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States69 works · 30,953 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Yuan Gong, Ziyi Chen, Iek-Heng Chu, Peng Chang and James Glass. 2022. Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment. arXiv:2205.03432.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1109/icassp43922.2022.9746743 · arXiv 2205.03432

@misc{gong2022transformer,
  title = {{Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment}},
  author = {Yuan Gong, Ziyi Chen, Iek-Heng Chu, Peng Chang and James Glass},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746743},
  eprint = {2205.03432},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/icassp43922.2022.9746743}
}

References

27 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [1] Maxine Eskenazi, “An overview of spoken language technology for education,” Speech Communication, 2009. ↗
  2. [14] Joseph Tepperman and Shrikanth Narayanan, “Automatic syllable stress detection using prosodic features for pronunciation evaluation of language learners,” in ICASSP, 2005. find
  3. [2] Klaus Zechner, Derrick Higgins, et al., “Automatic scoring of non-native spontaneous speech in tests of spoken english,” Speech Communication, 2009. ↗
  4. [15] Juan Pablo Arias, Nestor Becerra Yoma, and Hiram Vivanco, “Automatic intonation assessment for computer aided language learning,” Speech Communication, 2010. ↗
  5. [3] Silke M Witt, “Automatic error detection in pronunciation training: Where we are and where we need to go,” in International Symposium on Automatic Detection on Errors in Pronunciation Training, 2012. find
  6. [16] Kun Li, Xixin Wu, and Helen Meng, “Intonation classification for l2 english speech using multi-distribution deep neural networks,” Computer Speech and Language, 2017. ↗
  7. [4] S.M Witt and S.J Young, “Phone-level pronunciation scoring and assessment for interactive language learning,” Speech Communication, 2000. ↗
  8. [17] Ashish Vaswani, Noam Shazeer, et al., “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017.
  9. [5] Feng Zhang, Chao Huang, et al., “Automatic mispronunciation detection for mandarin,” in ICASSP, 2008. find
  10. [18] Junbo Zhang, Zhiwen Zhang, et al., “speechocean762: An open-source non-native english speech corpus for pronunciation assessment,” in Interspeech, 2021.
  11. [6] Dean Luo, Yu Qiao, et al., “Analysis and utilization of mllr speaker adaptation technique for learners’ pronunciation evaluation,” in Interspeech, 2009. find
  12. [19] Binghuai Lin, Liyuan Wang, Xiaoli Feng, and Jinsong Zhang, “Automatic scoring at multi-granularity for l2 pronunciation.,” in Interspeech, 2020. find
  13. [7] Yow-Bang Wang and Lin-Shan Lee, “Improved approaches of modeling and detecting error patterns with empirical analysis for computer-aided pronunciation training,” in ICASSP, 2012. find
  14. [20] Xiaoshuo Xu, Yueteng Kang, Songjun Cao, Binghuai Lin, et al., “Explore wav2vec 2.0 for mispronunciation detection,” Interspeech, 2021. find
  15. [8] Wenping Hu, Yao Qian, et al., “Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiers,” Speech Communication, 2015. ↗
  16. [9] Jiatong Shi, Nan Huo, and Qin Jin, “Context-aware goodness of pronunciation for computer-assisted pronunciation training,” in Interspeech, 2020. find
  17. [10] Joost van Doremalen, C Cucchiarini, and H Strik, “Using non-native error patterns to improve pronunciation verification,” in Interspeech, 2010. find
  18. [11] Catia Cucchiarini, Helmer Strik, and LWJ Boves, “Quantitative assessment of second language learners’ fluency: an automatic approach,” in ICSLP, 1998. find
  19. [12] Catia Cucchiarini, Helmer Strik, and Lou Boves, “Quantitative assessment of second language learners’ fluency by means of automatic speech recognition technology,” The Journal of the Acoustical Society of America, 2000. find
  20. [13] Paul Christopher Bagshaw, Automatic prosodic analysis for computer aided pronunciation teaching, Ph.D. thesis, 1994. find
  21. [21] Binghuai Lin and Liyuan Wang, “Deep feature transfer learning for automatic pronunciation assessment,” Interspeech, 2021. find
  22. [22] Tobias Cincarek, Rainer Gruhn, et al., “Automatic pronunciation scoring of words and sentences independent from the non-native’s first language,” Computer Speech and Language, 2009. find
  23. [23] Jacob Devlin, Ming-Wei Chang, et al., “Bert: Pre-training of deep bidirectional transformers for language understanding,” ACL, 2018.
  24. [24] Vassil Panayotov, Guoguo Chen, et al., “Librispeech: an asr corpus based on public domain audio books,” in ICASSP, 2015. find
  25. [25] Ming Tu, Anna Grabek, et al., “Investigating the role of l1 in automatic pronunciation evaluation of l2 speech,” in Interspeech, 2018. find
  26. [26] François Hernandez, Vincent Nguyen, et al., “Ted-lium 3: twice as much data and corpus repartition for experiments on speaker adaptation,” in SPECOM, 2018. find
  27. [27] Mitchell P Marcus, Mary Ann Marcinkiewicz, et al., “Building a large annotated corpus of english: the penn treebank,” Computational Linguistics, 1993. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gong-2022-transformer-based-multi-aspect-multi/