Elamur Research
☰
home / papers / 2020 / wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2006.11477

Summary

Modern speech recognisers usually need thousands of hours of transcribed audio. Most of the world's roughly 7,000 languages don't have that much. The authors introduce wav2vec 2.0, which first learns from raw audio with no transcripts and is then adapted to speech recognition using a small labelled set.

A convolutional encoder turns the waveform into latent frames. Spans of these frames are masked, and a Transformer builds context-aware representations from what remains. During pre-training, the model must pick the true target for each masked frame from a set of distractors. The targets are discretised by product quantization with a Gumbel softmax, learned jointly with the rest of the network. For fine-tuning, a linear output layer is added and trained with CTC.

The model is pre-trained on Librispeech audio (960 h) or LibriVox (about 53k h). It sets new records on TIMIT and the Librispeech 100-hour subset, and it stays accurate with as little as ten minutes of labelled speech. This matters for low-resource languages such as Ukrainian: plentiful raw audio plus a tiny transcribed sample can yield usable ASR.

Key points

  • With all 960 hours of Librispeech labels, the LARGE model pre-trained on LV-60k reaches 1.8/3.3 WER on test-clean/other. Its own architecture trained from scratch gets only 2.1/4.6.
  • With 10 minutes of labels (48 short recordings) and 53.2k hours of unlabelled LibriVox audio, it reaches 4.8/8.2 WER. With 1 hour of labels it beats the previous best self-training result on the 100-hour subset, which used a hundredfold more transcribed data.
  • On TIMIT phoneme recognition it gets 7.4 dev / 8.3 test PER, a relative improvement of 23%/29% over the previous best system (vq-wav2vec).
  • An ablation shows the best setup feeds continuous latents to the Transformer and quantizes only the targets (7.97 average dev WER). Quantizing both inputs and targets gives 12.18.
  • BASE has 12 Transformer blocks with dimension 768 and was trained on 64 V100 GPUs for 1.6 days. LARGE has 24 blocks with dimension 1,024 and was trained on 128 V100 GPUs. Code and models are released in fairseq.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

971 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 951 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2006.11477 · arXiv 2006.11477

@misc{baevski2020wav2vec,
  title = {{wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations}},
  author = {Baevski, Alexei and Zhou, Henry and Mohamed, Abdelrahman and Auli, Michael},
  year = {2020},
  doi = {10.48550/arxiv.2006.11477},
  eprint = {2006.11477},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.11477}
}

References

60 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv, 2016. find our card
  2. P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. In Proc. of NeurIPS, 2019. find
  3. A. Baevski and M. Auli. Adaptive input representations for neural language modeling. In Proc. of ICLR, 2018. find
  4. A. Baevski, M. Auli, and A. Mohamed. Effectiveness of self-supervised pre-training for speech recognition. arXiv, abs/1911.03912, 2019. — [arXiv:1911.03912](https://arxiv.org/abs/1911.03912) find
  5. A. Baevski, S. Schneider, and M. Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In Proc. of ICLR, 2020. find
  6. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. arXiv, abs/2002.05709, 2020. find our card
  7. J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord. Unsupervised speech representation learning using wavenet autoencoders. arXiv, abs/1901.08810, 2019. — [arXiv:1901.08810](https://arxiv.org/abs/1901.08810) find
  8. Y. Chung, W. Hsu, H. Tang, and J. R. Glass. An unsupervised autoregressive model for speech representation learning. arXiv, abs/1904.03240, 2019. — [arXiv:1904.03240](https://arxiv.org/abs/1904.03240) find
  9. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv, abs/1810.04805, 2018. find
  10. S. Dieleman, A. van den Oord, and K. Simonyan. The challenge of realistic music generation: modelling raw audio at scale. arXiv, 2018. find
  11. R. Eloff, A. Nortje, B. van Niekerk, A. Govender, L. Nortje, A. Pretorius, E. Van Biljon, E. van der Westhuizen, L. van Staden, and H. Kamper. Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks. arXiv, abs/1904.07556, 2019. — [arXiv:1904.07556](https://arxiv.org/abs/1904.07556) find
  12. A. Fan, E. Grave, and A. Joulin. Reducing transformer depth on demand with structured dropout. In Proc. of ICLR, 2020. find our card
  13. J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren. The DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus CDROM. Linguistic Data Consortium, 1993. find
  14. A. Graves, S. Fernández, and F. Gomez. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proc. of ICML, 2006. find
  15. A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang. Conformer: Convolution-augmented transformer for speech recognition. arXiv, 2020. find our card
  16. E. J. Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures, volume 33. US Government Printing Office, 1954. find
  17. W. Han, Z. Zhang, Y. Zhang, J. Yu, C.-C. Chiu, J. Qin, A. Gulati, R. Pang, and Y. Wu. Contextnet: Improving convolutional neural networks for automatic speech recognition with global context. arXiv, 2020. find
  18. D. Harwath, W.-N. Hsu, and J. Glass. Learning hierarchical discrete linguistic units from visually-grounded speech. In Proc. of ICLR, 2020. find
  19. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. arXiv, abs/1911.05722, 2019. find our card
  20. O. J. Hénaff, A. Razavi, C. Doersch, S. M. A. Eslami, and A. van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv, abs/1905.09272, 2019. — [arXiv:1905.09272](https://arxiv.org/abs/1905.09272) find
  21. D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv, 2016. find
  22. G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger. Deep networks with stochastic depth. arXiv, 2016. find
  23. M. G. A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. of AISTATS, 2010. find
  24. E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. arXiv, abs/1611.01144, 2016. find our card
  25. H. Jegou, M. Douze, and C. Schmid. Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell., 33(1):117–128, Jan. 2011. find
  26. D. Jiang, X. Lei, W. Li, N. Luo, Y. Hu, W. Zou, and X. Li. Improving transformer-based speech recognition using unsupervised pre-training. arXiv, abs/1910.09932, 2019. — [arXiv:1910.09932](https://arxiv.org/abs/1910.09932) find
  27. J. Kahn et al. Libri-light: A benchmark for asr with limited or no supervision. In Proc. of ICASSP, 2020. find
  28. K. Kawakami, L. Wang, C. Dyer, P. Blunsom, and A. van den Oord. Learning robust and multilingual speech representations. arXiv, 2020. find
  29. D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In Proc. of ICLR, 2015. find
  30. A. Laptev, R. Korostik, A. Svischev, A. Andrusenko, I. Medennikov, and S. Rybin. You do not need more data: Improving end-to-end speech recognition by text-to-speech data augmentation. arXiv, abs/2005.07157, 2020. — [arXiv:2005.07157](https://arxiv.org/abs/2005.07157) find
  31. M. P. Lewis, G. F. Simon, and C. D. Fennig. Ethnologue: Languages of the world, nineteenth edition. Online version: http://www.ethnologue.com, 2016. ↗
  32. A. H. Liu, T. Tu, H. yi Lee, and L. shan Lee. Towards unsupervised speech recognition and synthesis with quantized speech representation learning. arXiv, 2019. find
  33. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. find
  34. C. Lüscher, E. Beck, K. Irie, M. Kitza, W. Michel, A. Zeyer, R. Schlüter, and H. Ney. Rwth asr systems for librispeech: Hybrid vs attention. In Interspeech 2019, 2019. find
  35. C. J. Maddison, D. Tarlow, and T. Minka. A* sampling. In Advances in Neural Information Processing Systems, pages 3086–3094, 2014. find
  36. I. Misra and L. van der Maaten. Self-supervised learning of pretext-invariant representations. arXiv, 2019. find
  37. A. Mohamed, D. Okhonko, and L. Zettlemoyer. Transformers with convolutional context for ASR. arXiv, abs/1904.11660, 2019. — [arXiv:1904.11660](https://arxiv.org/abs/1904.11660) find
  38. M. Ott, S. Edunov, D. Grangier, and M. Auli. Scaling neural machine translation. In Proc. of WMT, 2018. find our card
  39. M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL System Demonstrations, 2019. find our card
  40. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an asr corpus based on public domain audio books. In Proc. of ICASSP, pages 5206–5210. IEEE, 2015. find
  41. D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le. Specaugment: A simple data augmentation method for automatic speech recognition. In Proc. of Interspeech, 2019. find our card
  42. D. S. Park, Y. Zhang, Y. Jia, W. Han, C.-C. Chiu, B. Li, Y. Wu, and Q. V. Le. Improved noisy student training for automatic speech recognition. arXiv, abs/2005.09629, 2020. — [arXiv:2005.09629](https://arxiv.org/abs/2005.09629) find
  43. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In Proc. of ACL, 2018. find
  44. V. Pratap, A. Hannun, Q. Xu, J. Cai, J. Kahn, G. Synnaeve, V. Liptchinsky, and R. Collobert. Wav2letter++: A fast open-source speech recognition system. In Proc. of ICASSP, 2019. find
  45. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language\_understanding\_paper.pdf, 2018. ↗
  46. M. Ravanelli, P. Brakel, M. Omologo, and Y. Bengio. Light gated recurrent units for speech recognition. IEEE Transactions on Emerging Topics in Computational Intelligence, 2(2):92–102, 2018. find
  47. M. Ravanelli, J. Zhong, S. Pascual, P. Swietojanski, J. Monteiro, J. Trmal, and Y. Bengio. Multi-task self-supervised learning for robust speech recognition. arXiv, 2020. find
  48. M. Rivière, A. Joulin, P.-E. Mazaré, and E. Dupoux. Unsupervised pretraining transfers well across languages. arXiv, abs/2002.02848, 2020. — [arXiv:2002.02848](https://arxiv.org/abs/2002.02848) find
  49. S. Schneider, A. Baevski, R. Collobert, and M. Auli. wav2vec: Unsupervised pre-training for speech recognition. In Proc. of Interspeech, 2019. find
  50. M. Schuster and K. Nakajima. Japanese and korean voice search. In Proc. of ICASSP, 2012. find
  51. G. Synnaeve, Q. Xu, J. Kahn, T. Likhomanenko, E. Grave, V. Pratap, A. Sriram, V. Liptchinsky, and R. Collobert. End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures. arXiv, abs/1911.08460, 2020. — [arXiv:1911.08460](https://arxiv.org/abs/1911.08460) find
  52. A. Tjandra, B. Sisman, M. Zhang, S. Sakti, H. Li, and S. Nakamura. Vqvae unsupervised unit discovery and multi-scale code2spec inverter for zerospeech challenge 2019. arXiv, 1905.11449, 2019. find
  53. A. van den Oord, O. Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pages 6306–6315, 2017. find
  54. A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv, abs/1807.03748, 2018. find our card
  55. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Proc. of NIPS, 2017. find
  56. W. Wang, Q. Tang, and K. Livescu. Unsupervised pre-training of bidirectional speech encoders via masked reconstruction. arXiv, 2020. find
  57. F. Wu, A. Fan, A. Baevski, Y. N. Dauphin, and M. Auli. Pay less attention with lightweight and dynamic convolutions. In Proc. of ICLR, 2019. find our card
  58. Q. Xu, T. Likhomanenko, J. Kahn, A. Hannun, G. Synnaeve, and R. Collobert. Iterative pseudo-labeling for speech recognition. arXiv, 2020. find
  59. N. Zeghidour, N. Usunier, I. Kokkinos, T. Schaiz, G. Synnaeve, and E. Dupoux. Learning filterbanks from raw speech for phone recognition. In Proc. of ICASSP, 2018. find
  60. Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar. Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss. arXiv, 2020. find