Elamur Research⌕
☰
home / papers / 2020 / Detection of Lexical Stress Errors in Non-Native (L2) English with Data Augmentation and Attention

Detection of Lexical Stress Errors in Non-Native (L2) English with Data Augmentation and Attention

Paper · 2020 · Daniel Korzekwa, Roberto Barra-Chicote, Szymon Zaporowski et al.

Summary

Pronunciation-training software usually drills individual sounds and seldom checks whether a learner stresses the right syllable, although misplaced stress makes L2 English harder to understand. The authors build a system that flags English words spoken with the wrong stress pattern. Earlier detectors measured hand-picked acoustic cues at fixed spots, such as the vowel nucleus. This model instead uses two dot-product attention modules. One learns which frames (pitch, loudness) matter for each syllable, and the other learns which phonemes (durations) matter. Each syllable is then compared with its neighbours through simple ratios. Real recordings contain very few stress mistakes, so the authors add words that a Neural TTS voice deliberately mispronounces. They justify this by treating the audio of synthetic examples as a hidden variable under a variational bound. The model is trained on the authors' own Slavic/Baltic recordings plus public corpora. With both techniques, it reaches 94.8% precision at 49.2% recall. A practice tool therefore rarely raises false alarms, even though it catches only about half of the real errors.

Key points

  • The model has three parts: a Feature Extractor, an Attention-based Classification Model and a Lexical Stress Error Detector.
    • Alignment comes from the Montreal forced aligner with a LibriSpeech acoustic model.
    • F0 and intensity are computed with Praat (10 ms step, 40 ms window).
  • The authors recorded their own L2 corpus of 25 speakers (23 Polish, 1 Ukrainian, 1 Lithuanian), each reading 200 multi-syllable words.
    • Half the list came from an English teacher and covers words these learners often get wrong.
    • The other half are frequent words from Google's Trillion Word Corpus.
    • TIMIT, Arctic, L2-Arctic and Porzuczek data were added.
  • Training data:
    • 8223 human-spoken words, of which only 425 have stress errors.
    • 3937 TTS words, of which 2005 have errors.
    • The test set has 2108 human words with 189 errors.
  • The relative-prominence layer computes ratios between each syllable and its two neighbours and has no trainable parameters.
    • The network is small: GRU and dense layers with 4 units each.
    • It was built in MxNet and tuned with Bayesian optimization.
  • The paper reports precision and recall on erroneous words rather than accuracy. The final result is 94.8% precision and 49.2% recall.

Interesting ideas

  • I1. Rare error examples are manufactured by a controllable generator (Neural TTS told to misplace stress), which rebalances a dataset dominated by correct cases.
  • I2. Prominence is judged relative to the neighbours through a parameter-free ratio layer, rather than by absolute feature values, which makes the signal robust to speaker or context scale.
  • I3. Two granularities are joined by plain replication: each phoneme's encoding is copied across all frames aligned to it, so attention can work at the finer level.
  • I4. Evaluation targets detecting the rare "wrong" class, reporting precision and recall instead of overall accuracy, which would hide poor error detection under heavy imbalance.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): the back generator could emit deliberately corrupted sentences (wrong agreement, swapped roles) to give repair rules and negative controls many labelled error cases.
  • I2 (tree): perceptron parser features could include ratios or differences between a token's scores and those of its adjacent tokens, keeping decisions explainable without extra weights.
  • I3 (tokens): word-level morphology or tag features could be copied onto every character or subword of the word, so character-level tokenizer decisions can see them transparently.
  • I4 (answer): when readers flag a test item as unanswerable or contradictory, scoring precision/recall on those flags shows whether they are trustworthy rather than lucky.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AmazonSeattle · United States70 works · 39,977 citations🎓Gdańsk University of TechnologyGdansk · Poland3 works · 58 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works Signal Processing · 213 works

🧭 Topics 1 English · 1,373 works

Cite

Cite as Daniel Korzekwa, Roberto Barra-Chicote, Szymon Zaporowski, Grzegorz Beringer, Jaime Lorenzo-Trueba, Alicja Serafinowicz, Jasha Droppo, Thomas Drugman and Bozena Kostek. 2020. Detection of Lexical Stress Errors in Non-Native (L2) English with Data Augmentation and Attention. arXiv:2012.14788.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2012.14788 · arXiv 2012.14788

@misc{korzekwa2020detection,
  title = {{Detection of Lexical Stress Errors in Non-Native (L2) English with Data Augmentation and Attention}},
  author = {Daniel Korzekwa, Roberto Barra-Chicote, Szymon Zaporowski, Grzegorz Beringer, Jaime Lorenzo-Trueba, Alicja Serafinowicz, Jasha Droppo, Thomas Drugman and Bozena Kostek},
  year = {2020},
  doi = {10.48550/arxiv.2012.14788},
  eprint = {2012.14788},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2012.14788}
}

References

36 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. M. Witt and S. J. Young, “Phone-level pronunciation scoring and assessment for interactive language learning,” Speech communication, vol. 30, no. 2-3, pp. 95–108, 2000. ↗
  2. W.-K. Leung, X. Liu, and H. Meng, “Cnn-rnn-ctc based end-to-end mispronunciation detection and diagnosis,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 8132–8136. find
  3. D. Korzekwa, J. Lorenzo-Trueba, S. Zaporowski, S. Calamaro, T. Drugman, and B. Kostek, “Mispronunciation detection in non-native (l2) english with uncertainty modeling,” in Accepted to ICASSP 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021. our card
  4. J. Field, “Intelligibility and the listener: The role of lexical stress,” TESOL quarterly, vol. 39, no. 3, pp. 399–423, 2005. ↗
  5. A. Lepage and M. G. Busà, “Intelligibility of english l2: The effects of incorrect word stress placement and incorrect vowel reduction in the speech of french and italian learners of english,” in Proc. of the Intl. Symposium on the Acquisition of Second Language Speech, vol. 5, no. 2014, 2014, pp. 387–400. find
  6. Y.-J. Jung, S.-C. Rhee et al., “Acoustic analysis of english lexical stress produced by korean, japanese and taiwanese-chinese speakers,” Phonetics and Speech Sciences, vol. 10, no. 1, pp. 15–22, 2018. ↗
  7. D. R. v. Bergem, “Acoustic and lexical vowel reduction,” in Phonetics and Phonology of Speaking Styles, 1991. find
  8. J. Latorre, J. Lachowicz, J. Lorenzo-Trueba, T. Merritt, T. Drugman, S. Ronanki, and V. Klimkov, “Effect of data reduction on sequence-to-sequence neural tts,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 7075–7079. find
  9. L. Ferrer, H. Bratt, C. Richey, H. Franco, V. Abrash, and K. Precoda, “Classification of lexical stress using spectral and prosodic features for computer-assisted language learning systems,” Speech Communication, vol. 69, pp. 31–45, 2015. ↗
  10. M. A. Shahin, J. Epps, and B. Ahmed, “Automatic classification of lexical stress in english and arabic languages using deep learning.” in INTERSPEECH, 2016, pp. 175–179. find
  11. J.-Y. Chen and L. Wang, “Automatic lexical stress detection for chinese learners’ of english,” in 2010 7th Intl. Symposium on Chinese Spoken Language Processing. IEEE, 2010, pp. 407–411. find
  12. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems, 2017, pp. 5998–6008. our card
  13. M. I. Posner and S. E. Petersen, “The attention system of the human brain,” Annual review of neuroscience, vol. 13, no. 1, pp. 25–42, 1990. ↗
  14. Y. Ruan, X. Wang, H. Liu, Z. Ou, Y. Gao, J. Cheng, and Y. Qian, “An end-to-end approach for lexical stress detection based on transformer,” arXiv preprint arXiv:1911.04862, 2019. ↗
  15. K. Li, S. Mao, X. Li, Z. Wu, and H. Meng, “Automatic lexical stress and pitch accent detection for l2 english speech using multi-distribution deep neural networks,” Speech Communication, vol. 96, pp. 28–36, 2018. ↗
  16. J. Zhao, H. Yuan, J. Liu, and S. Xia, “Automatic lexical stress detection using acoustic features for computer assisted language learning,” Proc. APSIPA ASC, pp. 247–251, 2011. find
  17. N. Chen and Q. He, “Using nonlinear features in automatic english lexical stress detection,” in 2007 Intl. Conference on Computational Intelligence and Security Workshops (CISW 2007). IEEE, 2007, pp. 328–332. ↗
  18. K. Li, X. Qian, S. Kang, and H. Meng, “Lexical stress detection for l2 english speech using deep belief networks.” in Interspeech, 2013, pp. 1811–1815. find
  19. M. K. Ramanathi, C. Yarra, and P. K. Ghosh, “Asr inspired syllable stress detection for pronunciation evaluation without using a supervised classifier and syllable level features.” in INTER-SPEECH, 2019, pp. 924–928. find
  20. M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using kaldi.” in Interspeech, vol. 2017, 2017, pp. 498–502. find
  21. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE Intl. Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, pp. 5206–5210. find
  22. P. Boersma, “Praat: doing http://www.praat.org/, 2006. phonetics by computer,” ↗
  23. K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” arXiv preprint arXiv:1406.1078, 2014. ↗
  24. I. Elias, H. Zen, J. Shen, Y. Zhang, Y. Jia, R. Weiss, and Y. Wu, “Parallel tacotron: Non-autoregressive and controllable tts,” arXiv preprint arXiv:2010.11439, 2020. ↗
  25. M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul, “An introduction to variational methods for graphical models,” Machine learning, vol. 37, no. 2, pp. 183–233, 1999. ↗
  26. T. e. a. Chen, “Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems,” arXiv preprint arXiv:1512.01274, 2015. ↗ our card
  27. A. Paleyes, M. Pullin, M. Mahsereci, N. Lawrence, and J. Gonzalez, “Emulation of physical processes with emukit,” in Second Workshop on Machine Learning and the Physical Sciences, NeurIPS, 2019. find
  28. J. Guo, H. He, T. He, L. Lausen, M. Li, H. Lin, X. Shi, C. Wang, J. Xie, S. Zha et al., “Gluoncv and gluonnlp: Deep learning in computer vision and natural language processing.” Journal of Machine Learning Research, vol. 21, no. 23, pp. 1–7, 2020. ↗
  29. J.-B. Michel, Y. K. Shen, A. P. Aiden, A. Veres, M. K. Gray, J. P. Pickett, D. Hoiberg, D. Clancy, P. Norvig, J. Orwant et al., “Quantitative analysis of culture using millions of digitized books,” science, vol. 331, no. 6014, pp. 176–182, 2011. ↗
  30. J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, and D. S. Pallett, “Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1,” STIN, vol. 93, p. 27403, 1993. find
  31. J. Kominek and A. W. Black, “The cmu arctic speech databases,” in Fifth ISCA workshop on speech synthesis, 2004. find
  32. G. Zhao, S. Sonsaat, A. O. Silpachai, I. Lucic, E. Chukharev-Hudilainen, J. Levis, and R. Gutierrez-Osuna, “L2-arctic: A non-native english speech corpus,” Perception Sensing Instrumentation Lab, 2018. find
  33. A. Porzuczek and A. Rojczyk, “English word stress in polish learners speech production and metacompetence,” Research in Language, vol. 15, no. 4, pp. 313–323, 2017. ↗
  34. A. Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. Driessche, E. Lockhart, L. Cobo, F. Stimberg et al., “Parallel wavenet: Fast high-fidelity speech synthesis,” in International conference on machine learning. PMLR, 2018, pp. 3918–3926. ↗
  35. A. E. Hieke, “Linking as a marker of fluent speech,” Language and Speech, vol. 27, no. 4, pp. 343–354, 1984. ↗
  36. S. Shattuck-Hufnagel, M. Ostendorf, and K. Ross, “Stress shift and early pitch accent placement in lexical items in american english,” Journal of Phonetics, vol. 22, no. 4, pp. 357–388, 1994. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/korzekwa-2020-detection-lexical-stress-errors-non/