Elamur Research
☰
home / papers / 2021 / HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2106.07447

Summary

Unsupervised speech representation learning is harder than in text or vision for three reasons. Each utterance contains many sounds, there is no ready-made inventory of discrete units, and unit boundaries are unknown. HuBERT (Hidden-Unit BERT) handles this by first clustering acoustic frames offline, starting with k-means over MFCC features, and treating the cluster IDs as frame-level pseudo-targets. A Transformer encoder based on the wav2vec 2.0 design receives audio with spans masked out. It must infer the cluster IDs of the hidden frames from context, so it learns acoustic and long-range sequential structure together. The clusters are then rebuilt from the model's own learned features, and pre-training repeats. The authors stress that consistent targets matter more than accurate ones. The models are pre-trained on Librispeech 960h or Libri-Light 60k hours and fine-tuned on labeled subsets from 10 minutes up to 960 hours. Across all of these settings, HuBERT equals or beats wav2vec 2.0. This is useful for languages and dialects with little transcribed audio, and the released checkpoints make reuse straightforward.

Key points

  • Training recipe: the initial teacher is k-means with 100 clusters, followed by two clustering iterations. The cross-entropy loss is computed only on masked frames (α = 1), which makes training less sensitive to noisy cluster labels. Mask spans follow the SpanBERT / wav2vec 2.0 scheme.
  • Model sizes: BASE (~95M parameters, 12 layers), LARGE (~317M, 24 layers) and X-LARGE (~964M, 48 layers). All three share a CNN waveform encoder made of seven 512-channel convolution layers.
  • Benchmarks: pre-training uses Librispeech (960h) and Libri-Light (60,000h). Fine-tuning splits are 10min, 1h, 10h, 100h and 960h. Results match or improve on wav2vec 2.0 in every split.
  • Scaling result: pre-trained on Libri-Light 60k hours, X-LARGE cuts WER relative to LARGE by up to 19% on dev-other and 13% on test-other.
  • Availability: code plus pre-trained and fine-tuned models are released in fairseq (examples/hubert). The method draws on DeepCluster from computer vision.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

520 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 506 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2106.07447 · arXiv 2106.07447

@misc{hsu2021hubert,
  title = {{HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units}},
  author = {Hsu, Wei-Ning and Bolte, Benjamin and Tsai, Yao-Hung Hubert and Lakhotia, Kushal and Salakhutdinov, Ruslan and Mohamed, Abdelrahman},
  year = {2021},
  doi = {10.48550/arxiv.2106.07447},
  eprint = {2106.07447},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2106.07447}
}

References

64 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018. find our card
  2. S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” arXiv preprint arXiv:1904.05862, 2019. — [arXiv:1904.05862](https://arxiv.org/abs/1904.05862) find
  3. E. Kharitonov, M. Rivière, G. Synnaeve, L. Wolf, P.-E. Mazaré, M. Douze, and E. Dupoux, “Data augmenting contrastive learning of speech representations in the time domain,” arXiv preprint arXiv:2007.00991, 2020. — [arXiv:2007.00991](https://arxiv.org/abs/2007.00991) find
  4. Y.-A. Chung, W.-N. Hsu, H. Tang, and J. Glass, “An unsupervised autoregressive model for speech representation learning,” arXiv preprint arXiv:1904.03240, 2019. — [arXiv:1904.03240](https://arxiv.org/abs/1904.03240) find
  5. A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” arXiv preprint arXiv:1910.05453, 2019. — [arXiv:1910.05453](https://arxiv.org/abs/1910.05453) find
  6. A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” arXiv preprint arXiv:2006.11477, 2020. find our card
  7. G. Zavaliagkos and T. Colthurst, “Utilizing untranscribed training data to improve performance,” in DARPA Broadcast News Transcription and Understanding Workshop, 1998. find
  8. J. Ma, S. Matsoukas, O. Kimball, and R. Schwartz, “Unsupervised training on large amounts of broadcast news data,” in ICASSP, 2006. find
  9. J. Kahn, A. Lee, and A. Hannun, “Self-training for end-to-end speech recognition,” in ICASSP, 2020. find
  10. W.-N. Hsu, A. Lee, G. Synnaeve, and A. Hannun, “Semi-supervised speech recognition via local prior matching,” arXiv preprint arXiv:2002.10336, 2020. — [arXiv:2002.10336](https://arxiv.org/abs/2002.10336) find
  11. A. Xiao, C. Fuegen, and A. Mohamed, “Contrastive semi-supervised learning for asr,” arXiv preprint arXiv:2103.05149, 2021. — [arXiv:2103.05149](https://arxiv.org/abs/2103.05149) find
  12. Q. Xu, T. Likhomanenko, J. Kahn, A. Hannun, G. Synnaeve, and R. Collobert, “Iterative pseudo-labeling for speech recognition,” arXiv preprint arXiv:2005.09267, 2020. — [arXiv:2005.09267](https://arxiv.org/abs/2005.09267) find
  13. M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” CoRR, vol. abs/2006.09882, 2020. — [arXiv:2006.09882](https://arxiv.org/abs/2006.09882) find
  14. X. Chen and K. He, “Exploring simple siamese representation learning,” CoRR, vol. abs/2011.10566, 2020. — [arXiv:2011.10566](https://arxiv.org/abs/2011.10566) find
  15. J. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. Á. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko, “Bootstrap your own latent: A new approach to self-supervised learning,” CoRR, vol. abs/2006.07733, 2020. — [arXiv:2006.07733](https://arxiv.org/abs/2006.07733) find
  16. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… find
  17. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019. find
  18. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” arXiv preprint arXiv:1910.13461, 2019. find our card
  19. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018. find
  20. K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, “Electra: Pre-training text encoders as discriminators rather than generators,” arXiv preprint arXiv:2003.10555, 2020. find
  21. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in NAACL, 2018. find
  22. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in CVPR, 2020. find our card
  23. M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in ECCV, 2018. find
  24. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in ICASSP, 2015. find
  25. J. Kahn et al., “Libri-light: A benchmark for asr with limited or no supervision,” in ICASSP, 2020. find
  26. C.-y. Lee and J. Glass, “A nonparametric bayesian approach to acoustic model discovery,” in ACL, 2012. find
  27. L. Ondel, L. Burget, and J. Černockỳ, “Variational inference for acoustic unit discovery,” Procedia Computer Science, vol. 81, pp. 80–86, 2016. find
  28. J. Ebbers, J. Heymann, L. Drude, T. Glarner, R. Haeb-Umbach, and B. Raj, “Hidden markov model variational autoencoder for acoustic unit discovery.” in INTERSPEECH, 2017. find
  29. W.-N. Hsu, Y. Zhang, and J. Glass, “Learning latent representations for speech generation and transformation,” in INTERSPEECH, 2017. find
  30. ——, “Unsupervised learning of disentangled and interpretable representations from sequential data,” in NeurIPS, 2017. find
  31. J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord, “Unsupervised speech representation learning using wavenet autoencoders,” IEEE/ACM transactions on audio, speech, and language processing, vol. 27, no. 12, pp. 2041–2053, 2019. find
  32. S. Khurana, S. R. Joty, A. Ali, and J. Glass, “A factorial deep markov model for unsupervised disentangled representation learning from speech,” in ICASSP, 2019. find
  33. S. Khurana, A. Laurent, W.-N. Hsu, J. Chorowski, A. Lancucki, R. Marxer, and J. Glass, “A convolutional deep markov model for unsupervised speech representation learning,” arXiv preprint arXiv:2006.02547, 2020. — [arXiv:2006.02547](https://arxiv.org/abs/2006.02547) find
  34. M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy, “Spanbert: Improving pre-training by representing and predicting spans,” Transactions of the Association for Computational Linguistics, 2020. find our card
  35. S. Young, “Large vocabulary continuous speech recognition: A review,” IEEE Signal Processing Magazine, vol. 13, no. 5, pp. 45–57, 1996. find
  36. O. Abdel-Hamid, A.-r. Mohamed, H. Jiang, and G. Penn, “Applying convolutional neural networks concepts to hybrid nn-hmm model for speech recognition,” in 2012 IEEE international conference on Acoustics, speech and signal processing (ICASSP). IEEE, 2012, pp. 4277– 4280. find
  37. D. Povey, “Discriminative training for large vocabulary speech recognition,” Ph.D. dissertation, University of Cambridge, 2005. find
  38. H. A. Bourlard and N. Morgan, Connectionist speech recognition: a hybrid approach. Springer Science & Business Media, 2012, vol. 247. find
  39. R. M. Gray and D. L. Neuhoff, “Quantization,” IEEE transactions on information theory, vol. 44, no. 6, pp. 2325–2383, 1998. find
  40. Y. Zhang, J. Qin, D. S. Park, W. Han, C.-C. Chiu, R. Pang, Q. V. Le, and Y. Wu, “Pushing the limits of semi-supervised learning for automatic speech recognition,” arXiv preprint arXiv:2010.10504, 2020. — [arXiv:2010.10504](https://arxiv.org/abs/2010.10504) find
  41. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in ICML, 2006. find
  42. A. van den Oord, O. Vinyals et al., “Neural discrete representation learning,” in NeurIPS, 2017. find
  43. T. Glarner, P. Hanebrink, J. Ebbers, and R. Haeb-Umbach, “Full bayesian hidden markov model variational autoencoder for acoustic unit discovery.” in INTERSPEECH, 2018. find
  44. Y.-A. Chung and J. Glass, “Generative pre-training for speech with autoregressive predictive coding,” in ICASSP, 2020. find
  45. ——, “Improved speech representations with multi-target autoregressive predictive coding,” arXiv preprint arXiv:2004.05274, 2020. find
  46. S. Ling, Y. Liu, J. Salazar, and K. Kirchhoff, “Deep contextualized acoustic representations for semi-supervised speech recognition,” in ICASSP, 2020. find
  47. W. Wang, Q. Tang, and K. Livescu, “Unsupervised pre-training of bidirectional speech encoders via masked reconstruction,” in ICASSP, 2020. find
  48. A. T. Liu, S.-w. Yang, P.-H. Chi, P.-c. Hsu, and H.-y. Lee, “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,” in ICASSP, 2020. find
  49. P.-H. Chi, P.-H. Chung, T.-H. Wu, C.-C. Hsieh, S.-W. Li, and H.-y. Lee, “Audio albert: A lite bert for self-supervised learning of audio representation,” arXiv preprint arXiv:2005.08575, 2020. — [arXiv:2005.08575](https://arxiv.org/abs/2005.08575) find
  50. S. Ling and Y. Liu, “Decoar 2.0: Deep contextualized acoustic representations with vector quantization,” arXiv preprint arXiv:2012.06659, 2020. — [arXiv:2012.06659](https://arxiv.org/abs/2012.06659) find
  51. A. Baevski, M. Auli, and A. Mohamed, “Effectiveness of self-supervised pre-training for speech recognition,” arXiv preprint arXiv:1911.03912, 2019. — [arXiv:1911.03912](https://arxiv.org/abs/1911.03912) find
  52. Y.-H. H. Tsai, Y. Wu, R. Salakhutdinov, and L.-P. Morency, “Self-supervised learning from a multi-view perspective,” arXiv preprint arXiv:2006.05576, 2020. — [arXiv:2006.05576](https://arxiv.org/abs/2006.05576) find
  53. S. Pascual, M. Ravanelli, J. Serrà, A. Bonafonte, and Y. Bengio, “Learning problem-agnostic speech representations from multiple self-supervised tasks,” in INTERSPEECH, 2019. find
  54. T. Likhomanenko, Q. Xu, J. Kahn, G. Synnaeve, and R. Collobert, “slimipl: Language-model-free iterative pseudo-labeling,” arXiv preprint arXiv:2010.11524, 2020. — [arXiv:2010.11524](https://arxiv.org/abs/2010.11524) find
  55. S. Lloyd, “Least squares quantization in pcm,” IEEE transactions on information theory, vol. 28, no. 2, pp. 129–137, 1982. find
  56. F. Pedregosa et al., “Scikit-learn: Machine learning in python,” the Journal of machine Learning research, 2011. find
  57. D. Arthur and S. Vassilvitskii, “k-means++: The advantages of careful seeding,” Stanford, Tech. Rep., 2006. find
  58. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. find
  59. V. Pratap et al., “wav2letter++: The fastest open-source speech recognition system,” arXiv preprint arXiv:1812.07625, 2018. find
  60. M. Ott et al., “fairseq: A fast, extensible toolkit for sequence modeling,” in NAACL, 2019. find our card
  61. D. S. Park, Y. Zhang, Y. Jia, W. Han, C.-C. Chiu, B. Li, Y. Wu, and Q. V. Le, “Improved noisy student training for automatic speech recognition,” arXiv preprint arXiv:2005.09629, 2020. — [arXiv:2005.09629](https://arxiv.org/abs/2005.09629) find
  62. A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu et al., “Conformer: Convolution-augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100, 2020. find our card
  63. Q. Xu, A. Baevski, T. Likhomanenko, P. Tomasello, A. Conneau, R. Collobert, G. Synnaeve, and M. Auli, “Self-training and pre-training are complementary for speech recognition,” arXiv preprint arXiv:2010.11430, 2020. — [arXiv:2010.11430](https://arxiv.org/abs/2010.11430) find
  64. D. Le, X. Zhang, W. Zheng, C. Fügen, G. Zweig, and M. L. Seltzer, “From senones to chenones: Tied context-dependent graphemes for hybrid speech recognition,” in ASRU, 2019. find