Elamur Research⌕
☰
home / papers / 2021 / Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English with Transfer Learning

Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English with Transfer Learning

Paper · 2021 · Toshiko Shibano, Xinyi Zhang, Mia Taige Li et al.

Summary

English speech recognisers do noticeably worse on non-native (L2) speakers. The main causes are first-language pronunciation habits and the small amount of transcribed accented speech. The authors test whether a strong pretrained model can close this gap through modest fine-tuning alone, with no external language model. They start from the self-trained wav2vec 2.0 Large (LV-60) checkpoint, called Wav2Vec 2.0-ST, and fine-tune it with CTC on L2-ARCTIC. L2-ARCTIC is a read-speech corpus of accented English whose prompts match the native CMU ARCTIC (L1-ARCTIC) recordings.

Five data splits separate several conditions:

  • multi-accent versus single-accent training;
  • speakers seen versus unseen in training;
  • accents seen versus entirely held out (zero-shot).

Models are also checked on L1-ARCTIC and LibriSpeech to detect any loss on native speech. Both multi-accent and single-accent fine-tuning reduce WER for L2 speakers. Single-accent models help their target accent most, even with about five hours of data. Accent transfer to a blind accent also works. For practical systems, this shows that a small labelled corpus can make ASR serve more of the world's English speakers.

Key points

  • L2-ARCTIC has 24 speakers across six L1s: Arabic, Hindi, Korean, Mandarin, Spanish and Vietnamese. Each L1 has two women and two men, and each speaker recorded about one hour of the 1,132 Gutenberg-based ARCTIC prompts. Data are divided 80:10:10 with no utterance overlap.
  • Each setup is trained with three random seeds and the WERs are averaged. Hyper-parameters are chosen on Dev from small grids over feature-masking and time-masking settings, with batch size 16.
  • Accent-specific models (Model-4, about five hours each) beat the multi-accent Model-1 on L2 test data by roughly 2 WER points on average. They also stay reasonable on L1 test sets.
  • Results vary far more across accents than across native test sets (SD 2.72–7.30 versus 0.20–0.43). Hindi and Korean are easiest and Vietnamese is hardest. Fine-tuning on Hindi helps only Hindi and hurts the other accents.
  • With a speaker held out (Model-5), spread across folds is small on the full test set (SD 0.11–0.38) but large on the unseen speaker (SD 1.12–4.87). Baseline WER correlates strongly and negatively with speakers' TOEFL scores (r ≈ −.77).

Interesting ideas

  • I1. The splits form a ladder of held-out conditions: unseen utterances, then unseen speakers within known groups, then an entire unseen group. This tells you which kind of generalisation a model really has.
  • I2. A same-domain native control set (L1-ARCTIC, identical prompts) is kept beside the target data, so any drop after adaptation clearly comes from forgetting rather than a domain shift.
  • I3. The authors train on one group and test on every other group, building a transfer table. This exposed an asymmetric source (Hindi) that helps only itself.
  • I4. The external language model is deliberately left out to measure the acoustic component's own ability, and adding it back is deferred to later work.
  • I5. Per-speaker error is correlated with an independent proficiency score (TOEFL), which grounds the variability in a measurable property of the input.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): evaluation could hold out whole authors or genres, not just sentences, so that silver-data gains are not mistaken for memorising a familiar source.
  • I2 (tree): when gparse is adapted to a new treebank or UD dialect, a fixed control slice of the original treebank would show whether LAS on the old style degrades.
  • I3 (learn): a similar matrix across text genres or witness-parser sources would show which silver sources to combine and which ones pull others down.
  • I4 (answer): readers could be scored once with the LLM teacher tiers switched off, isolating what the explainable pipeline achieves on its own.
  • I5 (answer): per-text error on school tests or tales could be correlated with a readability or grade-level measure, showing whether failures track text difficulty or specific pipeline bugs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada46 works · 33,673 citations

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 worksSignal Processing · 213 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksSpeech & spoken language · 233 works

Cite

Cite as Toshiko Shibano, Xinyi Zhang, Mia Taige Li, Haejin Cho, Peter Sullivan and Muhammad Abdul-Mageed University of British Columbia). 2021. Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English with Transfer Learning. arXiv:2110.00678.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2110.00678 · arXiv 2110.00678

@misc{shibano2021speech,
  title = {{Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English with Transfer Learning}},
  author = {Toshiko Shibano, Xinyi Zhang, Mia Taige Li, Haejin Cho, Peter Sullivan and Muhammad Abdul-Mageed University of British Columbia)},
  year = {2021},
  doi = {10.48550/arxiv.2110.00678},
  eprint = {2110.00678},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2110.00678}
}

References

36 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexei Baevski, Steffen Schneider, and Michael Auli. 2019. vq-wav2vec: Self-supervised learning of discrete speech representations. arXiv preprint arXiv:1910.05453. ↗
  2. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006.11477. ↗
  3. Amy Bearman, Kelsey Josund, and Gawan Fiore. 2017. Accent conversion using artificial neural networks. Technical report, Stanford University, Tech. Rep, Tech. Rep. find
  4. William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4960–4964. IEEE. find
  5. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-based models for speech recognition. In Advances in neural information processing systems, pages 577–585. ↗
  6. Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. 2019. An unsupervised autoregressive model for speech representation learning. arXiv preprint arXiv:1904.03240. ↗
  7. David Crystal. 2003. English as a global language. Ernst Klett Sprachen. ↗
  8. Nilaksh Das, Sravan Bodapati, Monica Sunkara, Sundararajan Srinivasan, and Duen Horng Chau. 2021. Best of both worlds: Robust accented speech recognition with adversarial transfer learning. arXiv preprint arXiv:2103.05834. ↗
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  10. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369–376. find
  11. Alex Graves and Navdeep Jaitly. 2014. Towards end-to-end speech recognition with recurrent neural networks. In International conference on machine learning, pages 1764–1772. find
  12. Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. 2020. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100. ↗
  13. Jingyong Hou, Pengcheng Guo, Sining Sun, Frank K Soong, Wenping Hu, and Lei Xie. 2019. Domain adversarial training for improving keyword spotting performance of esl speech. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8122–8126. IEEE. find
  14. Hu Hu, Xuesong Yang, Zeynab Raeesy, Jinxi Guo, Gokce Keskin, Harish Arsikere, Ariya Rastrow, Andreas Stolcke, and Roland Maas. 2021. Redat: Accent-invariant representation for end-to-end asr by domain adversarial training with relabeling. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6408–6412. IEEE. ↗
  15. Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. 2020. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7669–7673. IEEE. find
  16. John Kominek, Alan W Black, and Ver Ver. 2003. Cmu arctic databases for speech synthesis. find
  17. Shaoshi Ling, Yuzong Liu, Julian Salazar, and Katrin Kirchhoff. 2020. Deep contextualized acoustic representations for semi-supervised speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6429–6433. IEEE. find
  18. Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. 2020a. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6419–6423. IEEE. find
  19. Songxiang Liu, Disong Wang, Yuewen Cao, Lifa Sun, Xixin Wu, Shiyin Kang, Zhiyong Wu, Xunying Liu, Dan Su, Dong Yu, et al. 2020b. End-to-end accent conversion without using native utterances. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6289–6293. IEEE. find
  20. Karen Livescu and James Glass. 2000. Lexical modeling of non-native speech for automatic speech recognition. In 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 00CH37100), volume 3, pages 1683– 1686. IEEE. find
  21. Marco Matassoni, Roberto Gretter, Daniele Falavigna, and Diego Giuliani. 2018. Non-native children speech recognition through transfer learning. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6229–6233. IEEE. find
  22. Zhirong Wang, Tanja Schultz, and Alex Waibel. 2003. Comparison of acoustic model adaptation techniques on non-native speech. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., volume 1, pages I–I. IEEE. ↗
  23. Tomohiro Nakatani. 2019. Improving transformer-based end-to-end speech recognition with connectionist temporal classification and language model integration. In Proc. Interspeech 2019. find
  24. Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. 2017. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 11(8):1240–1253. ↗
  25. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. ↗
  26. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. find
  27. Tan Tien Ping. 2008. Automatic speech recognition for non-native speakers. Ph.D. thesis, Université Joseph-Fourier-Grenoble I. find
  28. Kacper Radzikowski, Le Wang, Osamu Yoshie, and Robert Nowak. 2021. Accent modification for speech recognition of non-native speakers using neural style transfer. EURASIP Journal on Audio, Speech, and Music Processing, 2021(1):1–10. ↗
  29. Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862. ↗
  30. Xian Shi, Fan Yu, Yizhou Lu, Yuhao Liang, Qiangze Feng, Daliang Wang, Yanmin Qian, and Lei Xie. 2021. The accented english speech recognition challenge 2020: open datasets, tracks, baselines, results and methods. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6918–6922. IEEE. ↗
  31. Sining Sun, Ching-Feng Yeh, Mei-Yuh Hwang, Mari Ostendorf, and Lei Xie. 2018. Domain adversarial training for accented speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4854–4858. IEEE. find
  32. Thibault Viglino, Petr Motlicek, and Milos Cernak. 2019. End-to-end accented speech recognition. In Interspeech, pages 2140–2144. find
  33. Yongqiang Wang, Abdelrahman Mohamed, Due Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, et al. 2020. Transformer-based acoustic modeling for hybrid speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6874–6878. IEEE. find
  34. Haihua Xu, Yerbolat Khassanov, Zhiping Zeng, Eng Siong Chng, Chongjia Ni, Bin Ma, Haizhou Li, et al. 2020. Independent language modeling architecture for end-to-end asr. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7059–7063. IEEE. find
  35. Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. 2021. Self-training and pre-training are complementary for speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3030–3034. IEEE. ↗
  36. Guanlong Zhao, Sinem Sonsaat, Alif O Silpachai, Ivana Lucic, Evgeny Chukharev-Hudilainen, John Levis, and Ricardo Gutierrez-Osuna. 2018. L2-arctic: A non-native english speech corpus. Perception Sensing Instrumentation Lab. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shibano-2021-speech-technology-everyone-automatic-speech-recognition/