Elamur Research
☰
home / papers / 2019 / SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition

SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1904.08779

Summary

End-to-end speech recognizers overfit easily and need large amounts of labelled audio. Earlier augmentation methods mostly changed the raw waveform, for example through speed perturbation, added noise or room simulation. The authors propose SpecAugment, which instead alters the log mel spectrogram directly, treating it like an image. It needs no extra data and is cheap enough to run online during training. It combines three operations: time warping, masking runs of consecutive mel frequency channels, and masking runs of consecutive time steps. The masking is inspired by Cutout from computer vision. They apply it to Listen, Attend and Spell (LAS) models on LibriSpeech 960h and Switchboard 300h. They pair it with longer learning-rate schedules, since augmentation benefits more from extended training. The augmented LAS models beat the previous best hybrid systems even without a language model. Shallow fusion with an RNN LM improves results further. The work shows that a very simple, domain-agnostic input perturbation can let plain end-to-end models outperform heavily engineered hybrid pipelines.

Key points

  • Results on LibriSpeech 960h:
    • Without an LM, WER is 2.8% on test-clean and 6.8% on test-other.
    • With shallow fusion, WER is 2.5% and 5.8%.
    • The previous best hybrid result on test-other was 7.5%, so this is roughly a 22% relative gain.
  • Results on Switchboard 300h (Hub5'00, Switchboard/CallHome portions):
    • Without an LM, WER is 7.2%/14.6%.
    • With fusion, WER is 6.8%/14.1%.
    • The prior hybrid state of the art was 8.3%/17.3%.
  • Augmentation policies: four hand-designed policies, LB, LD, SM and SS, each setting the warp parameter W, mask widths F and T, mask counts, and a cap p on time-mask width.
  • Models and tokenization: LASd-w models (e.g., LAS-4-1024, LAS-6-1280) use a CNN front end, bi-LSTM encoder and 2-layer RNN decoder, with Word Piece Models of 16k (LibriSpeech) and 1k (Switchboard).
  • Training schedules: three learning-rate schedules, B, D and the longest L (up to 320k steps). Longer schedules help more when augmentation is used. LibriSpeech runs took 7 days on 32 Cloud TPU chips.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

302 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 293 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.21437/interspeech.2019-2680 · arXiv 1904.08779

@misc{park2019specaugment,
  title = {{SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition}},
  author = {Park, Daniel S. and Chan, William and Zhang, Yu and Chiu, Chung-Cheng and Zoph, Barret and Cubuk, Ekin D. and Le, Quoc V.},
  year = {2019},
  doi = {10.21437/interspeech.2019-2680},
  eprint = {1904.08779},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.21437/interspeech.2019-2680}
}

References

50 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G. Hinton, L. Deng, D. Yu, G. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, B. Kingsbury et al., “Deep neural networks for acoustic modeling in speech recognition,” IEEE Signal processing magazine, vol. 29, 2012. find
  2. G. Dahl, D. Yu, L. Deng, and A. Acero, “Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 20, Jan 2012. find
  3. T. Sainath, A. rahman Mohamed, B. Kingsbury, and B. Ramabhadran, “Deep Convolutional Neural Networks for LVCSR,” in ICASSP, 2013. find
  4. A. Graves, A. rahman Mohamed, and G. Hinton, “Speech Recognition with Deep Recurrent Neural Networks,” in ICASSP, 2013. find
  5. A. Graves and N. Jaitly, “Towards End-to-End Speech Recognition with Recurrent Neural Networks,” in ICML, 2014. find
  6. W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition,” in ICASSP, 2016. find
  7. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-End Attention-based Large Vocabulary Speech Recognition,” in ICASSP, 2016. find
  8. C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, E. Gonina, N. Jaitly, B. Li, J. Chorowski, and M. Bacchiani, “State-of-the-art Speech Recognition With Sequence-to-Sequence Models,” in ICASSP, 2018. find
  9. N. Kanda, R. Takeda, and Y. Obuchi, “Elastic spectral distortion for low resource speech recognition with deep neural networks,” in ASRU, 2013. find
  10. A. Ragni, K. M. Knill, S. P. Rath, and M. J. F. Gales, “Data augmentation for low resource languages,” in INTERSPEECH, 2014. find
  11. N. Jaitly and G. Hinton, “Vocal Tract Length Perturbation (VTLP) improves speech recognition,” in ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013. find
  12. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Ng, “Deep Speech: Scaling up end-to-end speech recognition,” in arXiv, 2014. find our card
  13. T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio Augmentation for Speech Recognition,” in INTERSPEECH, 2015. find
  14. C. Kim, A. Misra, K. Chin, T. Hughes, A. Narayanan, T. Sainath, and M. Bacchiani, “Generation of large-scale simulated utterances in virtual rooms to train deep-neural networks for far-field speech recognition in Google Home,” in INTERSPEECH, 2017. find
  15. R. Prabhavalkar, R. Alvarez, C. Parada, P. Nakkiran, and T. N. Sainath, “Automatic gain control and multi-style training for robust small-footprint keyword spotting with deep neural networks,” in ICASSP, 2015. find
  16. A. Raju, S. Panchapagesan, X. Liu, A. Mandal, and N. Strom, “Data Augmentation for Robust Keyword Spotting under Playback Interference,” in arXiv, 2018. find
  17. S. H. Mallidi and H. Hermansky, “Novel neural network based fusion for Multistream ASR,” in ICASSP, 2016. find
  18. E. D. Cubuk, B. Zoph, D. Mané, V. Vasudevan, and Q. V. Le, “Autoaugment: Learning augmentation policies from data,” in CVPR, 2019. find
  19. T. DeVries and G. Taylor, “Improved Regularization of Convolutional Neural Networks with Cutout,” in arXiv, 2017. find
  20. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in ICASSP, 2015. find
  21. Ç. Gülçehre, O. Firat, K. Xu, K. Cho, L. Barrault, H. Lin, F. Bougares, H. Schwenk, and Y. Bengio, “On using monolingual corpora in neural machine translation,” in arxiv, 2015. find our card
  22. J. Godfrey, E. Holliman, and J. McDaniel, “SWITCHBOARD: telephone speech corpus for research and development,” in ICASSP, 1992. find
  23. C. Cieri, D. Miller, and K. Walker, “The fisher corpus: a resource for the next generations of speech-to-text,” in LREC, 2004. find
  24. A. Zeyer, K. Irie, R. Schlüter, and H. Ney, “Improved training of end-to-end attention models for speech recognition,” in INTER-SPEECH, 2018. find
  25. K. Irie, R. Prabhavalkar, A. Kannan, A. Bruguier, D. Rybach, and P. Nguyen, “Model Unit Exploration for Sequence-to-Sequence Speech Recognition,” in arXiv, 2019. find
  26. M. Schuster and K. Nakajima, “Japanese and korean voice search,” in ICASSP, 2012. find
  27. A. Graves, “Practical Variational Inference for Neural Networks,” in NIPS, 2011. find
  28. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in CVPR, 2016. find
  29. J. Chorowski and N. Jaitly, “Towards better decoding and language model integration in sequence to sequence models,” in IN-TERSPEECH, 2017. find
  30. D. Povey, V. Peddinti, D. Galvez, P. Ghahrmani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur, “Purely sequence-trained neural networks for ASR based on lattice-free MMI,” in INTER-SPEECH, 2016. find
  31. K. J. Han, A. Chandrashekaran, J. Kim, and I. Lane, “The CA-PIO 2017 Conversational Speech Recognition System,” in arXiv, 2017. find
  32. X. Yang, J. Li, and X. Zhou, “A novel pyramidal-FSMN architecture with lattice-free MMI for speech recognition,” in arXiv, 2018. find
  33. R. Collobert, C. Puhrsch, and G. Synnaeve, “Wav2Letter: an End-to-End ConvNet-based Speech Recognition System,” in arXiv, 2016. find
  34. V. Liptchinsky, G. Synnaeve, and R. Collobert, “Letter-Based Speech Recognition with Gated ConvNets,” in arXiv, 2017. find
  35. Y. Zhou, C. Xiong, and R. Socher, “Improving End-to-End Speech Recognition with Policy Learning,” in ICASSP, 2018. find
  36. N. Zeghidour, Q. Xu, V. Liptchinsky, N. Usunier, G. Synnaeve, and R. Collobert, “Fully Convolutional Speech Recognition,” in arXiv, 2018. find
  37. J. Li, V. Lavrukhin, B. Ginsburg, R. Leary, O. Kuchaiev, J. M. Cohen, H. Nguyen, and R. T. Gadde, “Jasper: An End-to-End Convolutional Neural Acoustic Model,” in arXiv, 2019. find
  38. A. Zeyer, A. Merboldt, R. Schlüter, and H. Ney, “A comprehensive analysis on attention models,” in NIPS: Workshop IRASL, 2018. find
  39. S. Sabour, W. Chan, and M. Norouzi, “Optimal Completion Distillation for Sequence Learning,” in ICLR, 2019. find
  40. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, “The Kaldi Speech Recognition Toolkit,” in ASRU, 2011. find
  41. K. Vesely, A. Ghoshal, L. Burger, and D. Povey, “Sequence-discriminative training of deep neural networks,” in INTER-SPEECH, 2013. find
  42. H. Hadian, H. Sameti, D. Povey, and S. Khudanpur, “End-to-end speech recognition using lattice-free MMI,” in INTERSPEECH, 2018. find
  43. G. Zweig, C. Yu, J. Droppo, and A. Stolcke, “Advances in All-Neural Speech Recognition,” in ICASSP, 2017. find
  44. K. Audhkhasi, B. Ramabhadran, G. Saon, M. Picheny, and D. Nahamoo, “Direct Acoustics-to-Word Models for English Conversational Speech Recognition,” in INTERSPEECH, 2018. find
  45. K. Audhkhasi, B. Kingsbury, B. Ramabhadran, G. Saon, and M. Picheny, “Building competitive direct acoustics-to-word models for english conversational speech recognition,” in ICASSP, 2018. find
  46. L. Lu, X. Zhang, and S. Renals, “On training the recurrent neural network encoder-decoder for large vocabulary end-to-end speech recognition,” in ICASSP, 2016. find
  47. S. Toshniwal, H. Tang, L. Lu, and K. Livescu, “Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition,” in INTERSPEECH, 2017. find
  48. C. Weng, J. Cui, G. Wang, J. Wang, C. Yu, D. Su, and D. Yu, “Improving Attention Based Sequence-to-Sequence Models for End-to-End English Conversational Speech Recognition,” in INTER-SPEECH, 2018. find
  49. G. Kovács, L. Tóth, D. Van Compernolle, and S. Ganapathy, “Increasing the robustness of cnn acoustic models using autoregressive moving average spectrogram features and channel dropout,” Pattern Recognition Letters, vol. 100, pp. 44–50, 2017. find
  50. L. Tóth, G. Kovács, and D. Van Compernolle, “A perceptually inspired data augmentation method for noise robust cnn acoustic models,” in SPECOM, 2018. find