Elamur Research⌕
☰
home / papers / 2022 / Improving Automatic Speech Recognition for Non-Native English with Transfer Learning and Language Model Decoding

Improving Automatic Speech Recognition for Non-Native English with Transfer Learning and Language Model Decoding

Paper · 2022 · Peter Sullivan, Toshiko Shibano and Muhammad Abdul-Mageed

Summary

ASR systems tuned for native (L1) English perform noticeably worse on non-native (L2) speakers. Accents shift pronunciation, and little annotated accented audio exists for training. The authors start from the self-trained wav2vec 2.0 Large model (Wav2Vec 2.0-ST) and fine-tune it with CTC under many data conditions: L1 speech, pooled L2 accents, single L2 accents, and multi-accent zero-shot setups. L2-ARCTIC is the central L2 corpus. They compare plain best-path decoding with beam search guided by an external KenLM language model. They also study error types through Levenshtein edit operations.

Fine-tuning and LM decoding each give gains of roughly similar size, and combining them closes much of the L1–L2 gap. A key observation concerns the language knowledge the self-trained model absorbs during training. That knowledge is enough for clean L1 audio but does not hold up on accented speech, so an explicit LM still helps. This is a useful warning against assuming a strong end-to-end model needs no external linguistic prior on out-of-domain input.

Key points

  • Base model and fine-tuning:

    • The base is Wav2Vec 2.0-ST (LV-60). It was pre-trained on 960 h of LibriSpeech, then self-trained on 53.2k h of Libri-Light using pseudo-labels produced with LibriSpeech language models.
    • Fine-tuning uses CTC with the CNN feature extractor frozen.
    • Masking and learning-rate settings, which imitate the tri-state schedule, are chosen per Dev set.

  • Accent findings:

    • Single-accent fine-tuning gives the largest gains on its target accent.
    • Multi-accent fine-tuning generalizes more widely.
    • Fine-tuning on domain-specific L1 speech actually hurts L2 performance.

  • Error reduction on the L2-ARCTIC dev set, relative to baseline B1 (best path decoding, no LM):

    ModelSubstitutionsDeletionsInsertions
    Fine-tuned M4−43%−55%−30%
    M4 + LM−64%−65%−52%

  • LM without fine-tuning: adding the LM to the non-fine-tuned B1 lowers WER but increases deletions.

  • Single-edit errors made by B1:

    • 15 of 18 deletions are spelling errors.
    • All 18 substitutions are spelling errors.
    • 7 of 11 insertions are spacing errors, e.g. "to morrow".
    • The LM is especially good at fixing word spacing.

Interesting ideas

  • I1. Isolate the contribution of two interventions with a full ±fine-tuning × ±LM grid, so gains from adaptation and from an external prior are measured separately rather than lumped together.
  • I2. Use cheap automatic edit-operation counts as an error taxonomy, then hand-inspect only the simplest single-edit cases to name the underlying linguistic cause (spelling, spacing, plural, tense).
  • I3. Do not assume knowledge a model internalized on clean in-domain data transfers to shifted input; test that explicitly, since the paper's LM-free strength vanished on accented speech.
  • I4. Contrast training on one target group, a pooled set of groups, and a held-out group to see the trade-off between peak accuracy and generalization.
  • I5. Freeze most weights (adapters, LayerNorm/attention-only tuning) when adapting, to avoid erasing knowledge the base model already holds.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Run the same 2×2 ablation for parser retraining vs. self-induced repair rules, so we know which one actually fixes which errors.
  • I2 (tokens): Categorize tokenizer, lemma and morphology mismatches by edit type, then manually label a small single-edit subset to find systematic rule gaps.
  • I3 (tree): Evaluate gparse on learner or non-standard English separately, and keep explicit lexical and grammatical constraints available as a fallback when the shift appears.
  • I4 (learn): Train silver-data parsers per genre, pooled, and with one genre held out, to choose between specialist and generalist models.
  • I5 (tree): When adapting gparse to a new UD dialect or treebank, tune only small adapter parts so behaviour on the original treebanks is preserved.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada46 works · 33,673 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works Signal Processing · 213 works

🧭 Topics 3 Language models & pretraining · 1,343 works English · 1,373 works Speech & spoken language · 233 works

Cite

Cite as Peter Sullivan, Toshiko Shibano and Muhammad Abdul-Mageed. 2022. Improving Automatic Speech Recognition for Non-Native English with Transfer Learning and Language Model Decoding. arXiv:2202.05209.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2202.05209 · arXiv 2202.05209

@misc{sullivan2022improving,
  title = {{Improving Automatic Speech Recognition for Non-Native English with Transfer Learning and Language Model Decoding}},
  author = {Peter Sullivan, Toshiko Shibano and Muhammad Abdul-Mageed},
  year = {2022},
  doi = {10.48550/arxiv.2202.05209},
  eprint = {2202.05209},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2202.05209}
}

References

59 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baevski A, Schneider S, Auli M (2019) vq-wav2vec: Self-supervised learning of discrete speech representations. arXiv preprint arXiv:191005453 find
  2. Baevski A, Zhou H, Mohamed A, Auli M (2020) wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:200611477 our card
  3. Bearman A, Josund K, Fiore G (2017) Accent conversion using artificial neural networks. Tech. rep., Stanford University, Tech. Rep, Tech. Rep find
  4. Chan W, Jaitly N, Le Q, Vinyals O (2016) Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 4960–4964 find
  5. Chorowski JK, Bahdanau D, Serdyuk D, Cho K, Bengio Y (2015) Attention-based models for speech recognition. In: Advances in neural information processing systems, pp 577–585 ↗ our card
  6. Chung YA, Hsu WN, Tang H, Glass J (2019) An unsupervised autoregressive model for speech representation learning. arXiv preprint arXiv:190403240 find
  7. Crystal D (2003) English as a global language. Ernst Klett Sprachen ↗
  8. Das N, Bodapati S, Sunkara M, Srinivasan S, Chau DH (2021) Best of both worlds: Robust accented speech recognition with adversarial transfer learning. arXiv preprint arXiv:210305834 find
  9. Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805 our card
  10. Futami H, Inaguma H, Ueno S, Mimura M, Sakai S, Kawahara T (2020) Distilling the knowledge of bert for sequence-to-sequence asr. arXiv preprint arXiv:200803822 find
  11. Graves A (2012) Connectionist temporal classification. In: Supervised Sequence Labelling with Recurrent Neural Networks, Springer, pp 61–93 ↗
  12. Graves A, Jaitly N (2014) Towards end-to-end speech recognition with recurrent neural networks. In: International conference on machine learning, pp 1764– 1772 find
  13. Graves A, Fernández S, Gomez F, Schmidhuber J (2006) Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd international conference on Machine learning, pp 369–376 find
  14. Gulati A, Qin J, Chiu CC, Parmar N, Zhang Y, Yu J, Han W, Wang S, Zhang Z, Wu Y, et al. (2020) Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:200508100 our card
  15. Guliani D, Beaufays F, Motta G (2021) Training speech recognition models with federated learning: A quality/cost framework. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 3080–3084 find
  16. Hannun AY, Maas AL, Jurafsky D, Ng AY (2014) First-pass large vocabulary continuous speech recognition using bi-directional recurrent dnns. arXiv preprint arXiv:14082873 find
  17. Heafield K (2011) Kenlm: Faster and smaller language model queries. In: Proceedings of the sixth workshop on statistical machine translation, pp 187– 197 our card
  18. Hori T, Cho J, Watanabe S (2018) End-to-end speech recognition with word-based rnn language models. In: 2018 IEEE Spoken Language Technology Workshop (SLT), IEEE, pp 389–396 find
  19. Hou J, Guo P, Sun S, Soong FK, Hu W, Xie L (2019) Domain adversarial training for improving keyword spotting performance of esl speech. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 8122–8126 find
  20. Houlsby N, Giurgiu A, Jastrzebski S, Morrone B, De Laroussilhe Q, Gesmundo A, Attariyan M, Gelly S (2019) Parameter-efficient transfer learning for nlp. In: International Conference on Machine Learning, PMLR, pp 2790–2799 ↗
  21. Hu H, Yang X, Raeesy Z, Guo J, Keskin G, Arsikere H, Rastrow A, Stolcke A, Maas R (2021) Redat: Accent-invariant representation for end-to-end asr by domain adversarial training with relabeling. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 6408–6412 ↗
  22. Hwang K, Sung W (2016) Character-level incremental speech recognition with recurrent neural networks. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 5335–5339 find
  23. Jain A, Upreti M, Jyothi P (2018) Improved accented speech recognition using accent embeddings and multi-task learning. In: Interspeech, pp 2454–2458 find
  24. Kahn J, Rivière M, Zheng W, Kharitonov E, Xu Q, Mazaré PE, Karadayi J, Liptchinsky V, Collobert R, Fuegen C, et al. (2020) Libri-light: A benchmark for asr with limited or no supervision. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 7669–7673 find
  25. Kominek J, Black AW, Ver V (2003) Cmu arctic databases for speech synthesis find
  26. Kunze J, Kirsch L, Kurenkov I, Krug A, Johannsmeier J, Stober S (2017) Transfer learning for speech recognition on a budget. arXiv preprint arXiv:170600290 find
  27. Li X, Wang C, Tang Y, Tran C, Tang Y, Pino J, Baevski A, Conneau A, Auli M (2020) Multilingual speech translation with efficient finetuning of pretrained models. arXiv preprint arXiv:201012829 find
  28. Ling S, Liu Y, Salazar J, Kirchhoff K (2020) Deep contextualized acoustic representations for semi-supervised speech recognition. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 6429–6433 find
  29. Liu AT, Yang Sw, Chi PH, Hsu Pc, Lee Hy (2020) Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 6419–6423 find
  30. Liu S, Wang D, Cao Y, Sun L, Wu X, Kang S, Wu Z, Liu X, Su D, Yu D, et al. (2020) End-to-end accent conversion without using native utterances. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 6289–6293 find
  31. Livescu K, Glass J (2000) Lexical modeling of non-native speech for automatic speech recognition. In: 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 00CH37100), IEEE, vol 3, pp 1683–1686 find
  32. Lowerre BT (1976) The harpy speech recognition system. Carnegie Mellon University find
  33. Maas A, Xie Z, Jurafsky D, Ng AY (2015) Lexicon-free conversational speech recognition with neural networks. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp 345–354 find
  34. Matassoni M, Gretter R, Falavigna D, Giuliani D (2018) Non-native children speech recognition through transfer learning. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 6229–6233 find
  35. Meister C, Vieira T, Cotterell R (2020) If beam search is the answer, what was the question? arXiv preprint arXiv:201002650 our card
  36. Miao Y, Gowayyed M, Metze F (2015) Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding. In: 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), IEEE, pp 167–174 find
  37. Oord Avd, Li Y, Vinyals O (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:180703748 our card
  38. Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22(10):1345–1359 ↗
  39. Panayotov V, Chen G, Povey D, Khudanpur S (2015) Librispeech: an asr corpus based on public domain audio books. In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 5206–5210 find
  40. Park DS, Chan W, Zhang Y, Chiu CC, Zoph B, Cubuk ED, Le QV (2019) Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:190408779 our card
  41. Ping TT (2008) Automatic speech recognition for non-native speakers. PhD thesis, Université Joseph-Fourier-Grenoble I find
  42. Radzikowski K, Wang L, Yoshie O, Nowak R (2021) Accent modification for speech recognition of non-native speakers using neural style transfer. EURASIP Journal on Audio, Speech, and Music Processing 2021(1):1–10 ↗
  43. Sak H, Senior A, Rao K, Irsoy O, Graves A, Beaufays F, Schalkwyk J (2015) Learning acoustic frame labeling for speech recognition with recurrent neural networks. In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 4280–4284 find
  44. Schneider S, Baevski A, Collobert R, Auli M (2019) wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:190405862 find
  45. Shi X, Yu F, Lu Y, Liang Y, Feng Q, Wang D, Qian Y, Xie L (2021) The accented english speech recognition challenge 2020: open datasets, tracks, baselines, results and methods. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 6918–6922 ↗ our card
  46. Shibano T, Zhang X, Li MT, Cho H, Sullivan P, Abdul-Mageed M (2021) Speech technology for everyone: Automatic speech recognition for non-native english with transfer learning. arXiv preprint arXiv:211000678 our card
  47. Sun S, Yeh CF, Hwang MY, Ostendorf M, Xie L (2018) Domain adversarial training for accented speech recognition. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 4854– 4858 find
  48. Synnaeve G, Xu Q, Kahn J, Likhomanenko T, Grave E, Pratap V, Sriram A, Liptchinsky V, Collobert R (2019) End-to-end asr: from supervised to semi-supervised learning with modern architectures. arXiv preprint arXiv:191108460 find
  49. Turan MAT, Vincent E, Jouvet D (2020) Achieving multi-accent asr via unsupervised acoustic model adaptation. In: INTERSPEECH 2020 find
  50. Viglino T, Motlicek P, Cernak M (2019) End-to-end accented speech recognition. In: Interspeech, pp 2140–2144 find
  51. Wang D, Zheng TF (2015) Transfer learning for speech and language processing. In: 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), IEEE, pp 1225–1237 find
  52. Wang Y, Luan H, Yuan J, Wang B, Lin H (2020) Laix corpus of chinese learner english: Towards a benchmark for l2 english asr. In: INTERSPEECH, pp 414–418 find
  53. Wang Y, Mohamed A, Le D, Liu C, Xiao A, Mahadeokar J, Huang H, Tjandra A, Zhang X, Zhang F, et al. (2020) Transformer-based acoustic modeling for hybrid speech recognition. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 6874–6878 find
  54. Wang Z, Schultz T, Waibel A (2003) Comparison of acoustic model adaptation techniques on non-native speech. In: 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., IEEE, vol 1, pp I–I find
  55. Watanabe S, Hori T, Kim S, Hershey JR, Hayashi T (2017) Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing 11(8):1240–1253 ↗
  56. Xu Q, Baevski A, Likhomanenko T, Tomasello P, Conneau A, Collobert R, Synnaeve G, Auli M (2021) Self-training and pre-training are complementary for speech recognition. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 3030–3034 ↗
  57. Yu W, Freiwald J, Tewes S, Huennemeyer F, Kolossa D (2021) Federated learning in asr: Not as easy as you think. In: Speech Communication; 14th ITG Conference, VDE, pp 1–5 ↗
  58. Zenkel T, Sanabria R, Metze F, Niehues J, Sperber M, Stüker S, Waibel A (2017) Comparison of decoding strategies for ctc acoustic models. arXiv preprint arXiv:170804469 find
  59. Zhao G, Sonsaat S, Silpachai AO, Lucic I, Chukharev-Hudilainen E, Levis J, Gutierrez-Osuna R (2018) L2-arctic: A non-native english speech corpus. Perception Sensing Instrumentation Lab find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sullivan-2022-improving-automatic-speech-recognition-non/