Elamur Research⌕
☰
home / papers / 2021 / Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English

Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English

Paper · 2021 · Toshiko Shibano, Xinyi Zhang, Mia Taige Li et al.

Summary

English speech recognizers work noticeably worse for non-native (L2) speakers, and there is little annotated accented speech to fix this. The authors test whether a strong self-supervised model, the self-trained wav2vec 2.0 Large (LV-60) (Wav2Vec 2.0-ST), can be adapted to L2 English by CTC fine-tuning on the small L2-ARCTIC corpus, with no external language model. They build five data splits that separate two questions: whether the model has seen the test speaker, and whether it has seen the test accent. Using these splits, they compare one model trained on all accents with models trained on a single accent. They also check what fine-tuning does to native speech on L1-ARCTIC and LibriSpeech. Fine-tuning improves L2 recognition in every setting, and the multi-accent model transfers to accents it never saw. Single-accent models, each trained on about five hours of audio, beat the multi-accent model on their own accent while still handling native speech reasonably well. The study suggests that cheap adaptation can narrow the accent gap and gives a clean protocol for measuring it.

Key points

  • L2-ARCTIC data: 24 speakers, about one hour each, covering six L1s (Arabic, Hindi, Korean, Mandarin, Spanish, Vietnamese), with 2 female and 2 male speakers per L1. All speakers read the 1,132 CMU ARCTIC prompts taken from Project Gutenberg, the same source behind LibriSpeech and Libri-Light. Data is split 80:10:10 with no shared utterances.
  • Training setup: five splits (speaker-dependent vs. speaker-independent, multi-accent vs. single-accent, plus a zero-shot condition with one accent held out). Each model is trained with three random seeds and the WERs are averaged. No language model is used.
  • Single-accent models (Model-4) beat the multi-accent Model-1 by about 2 WER points on average. Hindi and Korean were easiest and Vietnamese hardest.
  • Variability: on L2 test data, SD across models was 2.72–7.30, compared with 0.20–0.43 on L1 test data. For unseen speakers, the spread was 1.12–4.87, compared with 0.11–0.38 when the speaker was seen.
  • Transfer and proficiency: fine-tuning on Hindi helped only Hindi and hurt every other accent. Baseline WER correlated strongly and negatively with speakers' TOEFL scores (r ≈ −.77).

Interesting ideas

  • I1. Build evaluation splits along two separate axes, seen/unseen speaker and seen/unseen variety, so that memorizing individuals is not confused with generalizing to a group.
  • I2. Use a parallel corpus in which the same prompts are produced by native and non-native speakers, so the only thing that changes is the speaker variety and not the content.
  • I3. After specializing a model, always re-test on the original domain to catch regressions, as the authors did with L1-ARCTIC and LibriSpeech.
  • I4. Build a cross-variety transfer matrix (train on group A, test on every group B) to find asymmetric cases, like Hindi helping only itself, where adding data can hurt others.
  • I5. Correlate per-speaker errors with an external proficiency score (TOEFL) to explain variance rather than just report it.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Treebank and silver-data evaluation could hold out whole genres or sources as well as sentences, revealing whether the parser generalizes or just memorizes a source's style.
  • I2 (tree): Running the parser on aligned native and learner (or simplified) versions of the same sentences would isolate which constructions break the UD parse because of the variety alone.
  • I3 (learn): Any fine-tune of gparse on a niche domain should be gated by a fixed general-English regression set that acts as a negative control.
  • I4 (learn): A similar matrix across text genres would show which silver-data sources to mix and which to keep apart when training the parser.
  • I5 (answer): Correlating reader errors with a text difficulty measure such as grade level or sentence length would show whether failures track difficulty or a specific component.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSpeech & spoken language · 480 works

Cite

Cite as Toshiko Shibano, Xinyi Zhang, Mia Taige Li, Haejin Cho, Peter Sullivan and Muhammad Abdul-Mageed. 2021. Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English. Proceedings of the 4th International Conference on Natural Language and Speech Processing (ICNLSP 2021).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{shibano2021speech,
  title = {{Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English}},
  author = {Toshiko Shibano, Xinyi Zhang, Mia Taige Li, Haejin Cho, Peter Sullivan and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the 4th International Conference on Natural Language and Speech Processing (ICNLSP 2021)},
  year = {2021},
  url = {https://aclanthology.org/2021.icnlsp-1.2/}
}

References

36 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexei Baevski, Steffen Schneider, and Michael Auli. 2019. vq-wav2vec: Self-supervised learning of discrete speech representations. arXiv preprint arXiv:1910.05453. ↗
  2. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006.11477. ↗
  3. Amy Bearman, Kelsey Josund, and Gawan Fiore. 2017. Accent conversion using artificial neural networks. Technical report, Stanford University, Tech. Rep, Tech. Rep. find
  4. William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4960–4964. IEEE. find
  5. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-based models for speech recognition. In Advances in neural information processing systems, pages 577–585. ↗
  6. Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. 2019. An unsupervised autoregressive model for speech representation learning. arXiv preprint arXiv:1904.03240. ↗
  7. David Crystal. 2003. English as a global language. Ernst Klett Sprachen. ↗
  8. Nilaksh Das, Sravan Bodapati, Monica Sunkara, Sundararajan Srinivasan, and Duen Horng Chau. 2021. Best of both worlds: Robust accented speech recognition with adversarial transfer learning. arXiv preprint arXiv:2103.05834. ↗
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  10. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369–376. find
  11. Alex Graves and Navdeep Jaitly. 2014. Towards end-to-end speech recognition with recurrent neural networks. In International conference on machine learning, pages 1764–1772. find
  12. Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. 2020. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100. ↗
  13. Jingyong Hou, Pengcheng Guo, Sining Sun, Frank K Soong, Wenping Hu, and Lei Xie. 2019. Domain adversarial training for improving keyword spotting performance of esl speech. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8122–8126. IEEE. find
  14. Hu Hu, Xuesong Yang, Zeynab Raeesy, Jinxi Guo, Gokce Keskin, Harish Arsikere, Ariya Rastrow, Andreas Stolcke, and Roland Maas. 2021. Redat: Accent-invariant representation for end-to-end asr by domain adversarial training with relabeling. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6408–6412. IEEE. ↗
  15. Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. 2020. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7669–7673. IEEE. find
  16. John Kominek, Alan W Black, and Ver Ver. 2003. Cmu arctic databases for speech synthesis. find
  17. Shaoshi Ling, Yuzong Liu, Julian Salazar, and Katrin Kirchhoff. 2020. Deep contextualized acoustic representations for semi-supervised speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6429–6433. IEEE. find
  18. Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. 2020a. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6419–6423. IEEE. find
  19. Songxiang Liu, Disong Wang, Yuewen Cao, Lifa Sun, Xixin Wu, Shiyin Kang, Zhiyong Wu, Xunying Liu, Dan Su, Dong Yu, et al. 2020b. End-to-end accent conversion without using native utterances. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6289–6293. IEEE. find
  20. Karen Livescu and James Glass. 2000. Lexical modeling of non-native speech for automatic speech recognition. In 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 00CH37100), volume 3, pages 1683– 1686. IEEE. find
  21. Marco Matassoni, Roberto Gretter, Daniele Falavigna, and Diego Giuliani. 2018. Non-native children speech recognition through transfer learning. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6229–6233. IEEE. find
  22. Zhirong Wang, Tanja Schultz, and Alex Waibel. 2003. Comparison of acoustic model adaptation techniques on non-native speech. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., volume 1, pages I–I. IEEE. ↗
  23. Tomohiro Nakatani. 2019. Improving transformer-based end-to-end speech recognition with connectionist temporal classification and language model integration. In Proc. Interspeech 2019. find
  24. Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. 2017. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 11(8):1240–1253. ↗
  25. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. ↗
  26. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. find
  27. Tan Tien Ping. 2008. Automatic speech recognition for non-native speakers. Ph.D. thesis, Université Joseph-Fourier-Grenoble I. find
  28. Kacper Radzikowski, Le Wang, Osamu Yoshie, and Robert Nowak. 2021. Accent modification for speech recognition of non-native speakers using neural style transfer. EURASIP Journal on Audio, Speech, and Music Processing, 2021(1):1–10. ↗
  29. Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862. ↗
  30. Xian Shi, Fan Yu, Yizhou Lu, Yuhao Liang, Qiangze Feng, Daliang Wang, Yanmin Qian, and Lei Xie. 2021. The accented english speech recognition challenge 2020: open datasets, tracks, baselines, results and methods. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6918–6922. IEEE. ↗
  31. Sining Sun, Ching-Feng Yeh, Mei-Yuh Hwang, Mari Ostendorf, and Lei Xie. 2018. Domain adversarial training for accented speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4854–4858. IEEE. find
  32. Thibault Viglino, Petr Motlicek, and Milos Cernak. 2019. End-to-end accented speech recognition. In Interspeech, pages 2140–2144. find
  33. Yongqiang Wang, Abdelrahman Mohamed, Due Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, et al. 2020. Transformer-based acoustic modeling for hybrid speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6874–6878. IEEE. find
  34. Haihua Xu, Yerbolat Khassanov, Zhiping Zeng, Eng Siong Chng, Chongjia Ni, Bin Ma, Haizhou Li, et al. 2020. Independent language modeling architecture for end-to-end asr. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7059–7063. IEEE. find
  35. Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. 2021. Self-training and pre-training are complementary for speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3030–3034. IEEE. ↗
  36. Guanlong Zhao, Sinem Sonsaat, Alif O Silpachai, Ivana Lucic, Evgeny Chukharev-Hudilainen, John Levis, and Ricardo Gutierrez-Osuna. 2018. L2-arctic: A non-native english speech corpus. Perception Sensing Instrumentation Lab. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shibano-2021-speech-technology-everyone-automatic-speech/