Elamur Research
☰
home / papers / 2022 / AudioLM: a Language Modeling Approach to Audio Generation

AudioLM: a Language Modeling Approach to Audio Generation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2209.03143

Summary

AudioLM tackles a long-standing tension in audio synthesis: models that produce realistic-sounding waveforms tend to drift into incoherent output (such as babbling speech) without strong conditioning, while earlier "textless" spoken language models stay coherent but offer limited quality and only one clean-speech speaker. The Google Research authors frame audio generation as language modeling over discrete tokens and combine two kinds of tokenizers. Coarse semantic tokens come from k-means clustering of intermediate w2v-BERT activations and carry long-range structure. Fine acoustic tokens come from the residual quantizer of the SoundStream codec and preserve waveform detail. A multi-stage, decoder-only Transformer predicts these tokens hierarchically, and SoundStream decodes them back into audio. Trained only on raw audio, with no transcripts or symbolic scores, the system continues short prompts. Speech continuations are plausible in syntax and meaning and keep the voice, prosody and acoustic conditions of unseen speakers. Piano continuations stay consistent in melody, harmony and rhythm. The work matters because it reaches both high fidelity and long-term coherence without supervision.

Key points

  • Hybrid tokenization: w2v-BERT semantic tokens score well on phonetic discriminability but reconstruct audio poorly, while SoundStream acoustic tokens reconstruct well. The authors show the two complement each other.
  • Speech prompting: a prompt of only 3 seconds from a speaker not seen in training is enough for AudioLM to continue in the same voice, intonation and recording conditions, including reverberation and background noise.
  • Compact sequences: token sequences are 2–3 orders of magnitude shorter than the raw waveform, which makes long contexts feasible despite the quadratic cost of self-attention.
  • Comparison with prior systems: AudioLM improves on GSLM, which is limited to a single speaker, and on Jukebox and Perceiver AR, which trade off either audio quality or temporal structure.
  • Misuse mitigation: the authors trained a classifier that detects AudioLM-generated speech with very high accuracy. Audio samples are published at google-research.github.io/seanet/audiolm/examples.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

146 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 139 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2209.03143 · arXiv 2209.03143

@misc{borsos2022audiolm,
  title = {{AudioLM: a Language Modeling Approach to Audio Generation}},
  author = {Borsos, Zalan and Marinier, Raphael and Vincent, Damien and Kharitonov, Eugene and Pietquin, Olivier and Sharifi, Matt and Roblek, Dominik and Teboul, Olivier and Grangier, David and Tagliasacchi, Marco and Zeghidour, Neil},
  year = {2022},
  doi = {10.48550/arxiv.2209.03143},
  eprint = {2209.03143},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2209.03143}
}

References

63 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K. Kavukcuoglu, “Efficient neural audio synthesis,” in International Conference on Machine Learning (ICML), vol. 80. PMLR, 2018, pp. 2415–2424. find
  2. [3] K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. de Brebisson, Y. Bengio, and A. Courville, “MelGAN: Generative adversarial networks for conditional waveform synthesis,” in Advances in Neural Information Processing Systems (NeurIPS), 2019. find
  3. [4] J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” in Advances in Neural Information Processing Systems (NeurIPS), 2020. find
  4. [5] M. Tagliasacchi, Y. Li, K. Misiunas, and D. Roblek, “SEANet: A multimodal speech enhancement network,” in Interspeech, 2020. find
  5. [6] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “DiffWave: A versatile diffusion model for audio synthesis,” in International Conference on Learning Representations (ICLR), 2021. find
  6. [7] N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, and W. Chan, “WaveGrad: Estimating gradients for waveform generation,” in International Conference on Learning Representations (ICLR), 2021. find
  7. [8] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C.… find
  8. [10] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. B. Rao, P. Barnes, Y. Tay, N. M. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. C. Hutchinson, R. Pope, J. Bradbury, J. Austin, M.… find
  9. [11] J. Yu, X. Li, J. Y. Koh, H. Zhang, R. Pang, J. Qin, A. Ku, Y. Xu, J. Baldridge, and Y. Wu, “Vector-quantized image modeling with improved VQGAN,” in International Conference on Learning Representations (ICLR), 2022. find
  10. [12] J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan, B. Hutchinson, W. Han, Z. Parekh, X. Li, H. Zhang, J. Baldridge, and Y. Wu, “Scaling autoregressive models for content-rich text-to-image generation,” Transactions on Machine Learning Research (TMLR),… find
  11. [13] E. Dunbar, M. Bernard, N. Hamilakis, T. A. Nguyen, M. de Seyssel, P. Rozé, M. Rivière, E. Kharitonov, and E. Dupoux, “The zero resource speech challenge 2021: Spoken language modelling,” in Interspeech. find
  12. [14] K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed et al., “On generative spoken language modeling from raw audio,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 1336–1354, 2021. find
  13. [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 5998–6008. find
  14. [16] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE ACM Trans. Audio Speech Lang. Process., vol. 30, pp. 495–507, 2022. find our card
  15. [17] Y. Chung, Y. Zhang, W. Han, C. Chiu, J. Qin, R. Pang, and Y. Wu, “w2vbert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” in IEEE Automatic Speech Recognition and Understanding Workshop, ASRU. IEEE, 2021, pp. 244–250. find
  16. [18] A. Roberts, H. W. Chung, A. Levskaya, G. Mishra, J. Bradbury, D. Andor, S. Narang, B. Lester, C. Gaffney, A. Mohiuddin, C. Hawthorne, A. Lewkowycz, A. Salcianu, M. van Zee, J. Austin, S. Goodman, L. B. Soares, H. Hu, S. Tsvyashchenko, A. Chowdhery, J. Bastings, J. Bulian, X. Garcia, J. Ni, A. C… find
  17. [19] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT). Association for Computational… find
  18. [20] A. van den Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. van den Driessche, E. Lockhart, L. C. Cobo, F. Stimberg, N. Casagrande, D. Grewe, S. Noury, S. Dieleman, E. Elsen, N. Kalchbrenner, H. Zen, A. Graves, H. King, T. Walters, D. Belov, and D. Hassabis, “Parallel Wav… find
  19. [21] M. Binkowski, J. Donahue, S. Dieleman, A. Clark, E. Elsen, N. Casagrande, L. C. Cobo, and K. Simonyan, “High fidelity speech synthesis with adversarial networks,” in International Conference on Learning Representations (ICLR), 2020. find
  20. [22] A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” in Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 6306–6315. find
  21. [23] A. Razavi, A. van den Oord, and O. Vinyals, “Generating diverse high-fidelity images with VQ-VAE-2,” in Advances in Neural Information Processing Systems (NeurIPS), 2019, pp. 14 837–14 847. find
  22. [24] E. Agustsson, F. Mentzer, M. Tschannen, L. Cavigelli, R. Timofte, L. Benini, and L. V. Gool, “Soft-to-hard vector quantization for end-to-end learned compression of images and neural networks,” Advances in Neural Information Processing Systems (NeurIPS), 2017. find
  23. [25] S. Kankanahalli, “End-to-end optimized speech coding with deep neural networks,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 2521–2525. find
  24. [26] K. Zhen, J. Sung, M. S. Lee, S. Beack, and M. Kim, “Cascaded cross-module residual learning towards lightweight end-to-end speech coding,” in Interspeech. ISCA, 2019, pp. 3396–3400. find
  25. [27] D. Petermann, S. Beack, and M. Kim, “HARP-Net: Hyper-autoencoded reconstruction propagation for scalable neural audio coding,” arXiv:2107.10843, 2021. — [arXiv:2107.10843](https://arxiv.org/abs/2107.10843) find
  26. [28] A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv:1807.03748, 2018. find our card
  27. [29] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” arXiv:2006.11477, 2020. find our card
  28. [30] A. Saeed, D. Grangier, and N. Zeghidour, “Contrastive learning of general-purpose audio representations,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 3875–3879. find
  29. [31] E. Kharitonov, M. Rivière, G. Synnaeve, L. Wolf, P. Mazaré, M. Douze, and E. Dupoux, “Data augmenting contrastive learning of speech representations in the time domain,” in IEEE Spoken Language Technology Workshop, SLT 2021, Shenzhen, China, January 19-22, 2021. find
  30. [32] A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” in International Conference on Learning Representations (ICLR), 2020. find
  31. [33] W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE ACM Trans. Audio Speech Lang. Process., vol. 29, pp. 3451–3460, 2021. find our card
  32. [34] A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self-supervised speech representation model,” in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021, pp. 914–921. find
  33. [35] R. Thoppilan, D. D. Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, Y. Zhou, C. Chang, I. Krivokon, W. Rusch,… find our card
  34. [36] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. S… find our card
  35. [37] G. Lample and F. Charton, “Deep learning for symbolic mathematics,” in International Conference on Learning Representations (ICLR), 2020. find
  36. [38] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” JMLR, vol. 21, pp. 140:1–140:67, 2020. find
  37. [39] A. Roy, M. Saffar, A. Vaswani, and D. Grangier, “Efficient content-based sparse attention with routing transformers,” Trans. Assoc. Comput. Linguistics, vol. 9, pp. 53–68, 2021. find our card
  38. [40] K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlós, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser, D. B. Belanger, L. J. Colwell, and A. Weller, “Rethinking attention with performers,” in International Conference on Learning Representations (ICLR), 2021. find
  39. [41] C. Hawthorne, A. Jaegle, C. Cangea, S. Borgeaud, C. Nash, M. Malinowski, S. Dieleman, O. Vinyals, M. M. Botvinick, I. Simon, H. Sheahan, N. Zeghidour, J. Alayrac, J. Carreira, and J. H. Engel, “General-purpose, long-context autoregressive modeling with Perceiver AR,” in International Conference… find
  40. [42] P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Computer Vision Foundation / IEEE, 2021, pp. 12 873–12 883. find
  41. [43] S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang, D. Jacobs, J. Huang, and D. Parikh, “Long video generation with time-agnostic VQGAN and time-sensitive transformer,” arXiv:2204.03638, 2022. — [arXiv:2204.03638](https://arxiv.org/abs/2204.03638) find
  42. [44] P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A generative model for music,” arXiv:2005.00341, 2020. — [arXiv:2005.00341](https://arxiv.org/abs/2005.00341) find
  43. [45] E. Kharitonov, A. Lee, A. Polyak, Y. Adi, J. Copet, K. Lakhotia, T. A. Nguyen, M. Rivière, A. Mohamed, E. Dupoux, and W. Hsu, “Text-free prosody-aware generative spoken language modeling,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL). Associa… find
  44. [46] E. Kharitonov, J. Copet, K. Lakhotia, T. A. Nguyen, P. Tomasello, A. Lee, A. Elkahky, W. Hsu, A. Mohamed, E. Dupoux, and Y. Adi, “textless-lib: a library for textless spoken language processing,” arXiv:2202.07359, 2022. find
  45. [47] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Interspeech. ISCA, 2020, pp. 5036–5040. find our card
  46. [48] A. Conneau, A. Bapna, Y. Zhang, M. Ma, P. von Platen, A. Lozhkov, C. Cherry, Y. Jia, C. Rivera, M. Kale, D. van Esch, V. Axelrod, S. Khanuja, J. H. Clark, O. Firat, M. Auli, S. Ruder, J. Riesa, and M. Johnson, “XTREME-S: evaluating cross-lingual speech representations,” arXiv:2203.10752, 2022. — [arXiv:2203.10752](https://arxiv.org/abs/2203.10752) find
  47. [49] A. Hines, J. Skoglund, A. C. Kokaram, and N. Harte, “ViSQOL: an objective speech quality model,” EURASIP Journal on Audio, Speech, and Music Processing, vol. 2015, no. 1, pp. 1–18, 2015. find
  48. [50] M. Chinen, F. S. C. Lim, J. Skoglund, N. Gureev, F. O’Gorman, and A. Hines, “ViSQOL v3: an open source production ready objective speech and audio metric,” in Twelfth International Conference on Quality of Multimedia Experience (QoMEX), 2020, pp. 1–6. find
  49. [51] T. Schatz, V. Peddinti, F. R. Bach, A. Jansen, H. Hermansky, and E. Dupoux, “Evaluating speech features with the minimal-pair ABX task: analysis of the classical MFC/PLP pipeline,” in Interspeech. ISCA, 2013, pp. 1781–1785. find
  50. [52] T. Schatz, “Abx-discriminability measures and applications. (mesures de discriminabilité ABX et applications),” Ph.D. dissertation, Pierre and Marie Curie University, Paris, France, 2016. find
  51. [53] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux, “Libri-light: A benchmark for ASR with limited or no supervision,” in IEEE International Conference on Acou… find
  52. [54] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, pp. 5206–5210. find
  53. [55] M. Rivière and E. Dupoux, “Towards unsupervised learning of speech features in the wild,” in IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 156–163. find
  54. [56] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Interspeech. ISCA, 2020, pp. 5036–5040. find our card
  55. [57] W. Hsu, Y. H. Tsai, B. Bolte, R. Salakhutdinov, and A. Mohamed, “Hubert: How much can a bad teacher benefit ASR pre-training?” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6533–6537. find
  56. [58] M. Tagliasacchi, B. Gfeller, F. de Chaumont Quitry, and D. Roblek, “Self-supervised audio representation learning for mobile devices,” arXiv:1905.11796, 2019. — [arXiv:1905.11796](https://arxiv.org/abs/1905.11796) find
  57. [59] T. A. Nguyen, B. Sagot, and E. Dupoux, “Are discrete units necessary for spoken language modeling?” arXiv:2203.05936, 2022. — [arXiv:2203.05936](https://arxiv.org/abs/2203.05936) find
  58. [60] P. Peng and D. Harwath, “Self-supervised representation learning for speech using visual grounding and masked language modeling,” arXiv:2202.03543, 2022. — [arXiv:2202.03543](https://arxiv.org/abs/2202.03543) find
  59. [61] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” arXiv:1907.11692, 2019. find
  60. [62] B. van Niekerk, L. Nortje, M. Baas, and H. Kamper, “Analyzing speaker information in self-supervised models to improve zero-resource speech processing,” in Interspeech. ISCA, 2021, pp. 1554–1558. find
  61. [63] C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C. A. Huang, S. Dieleman, E. Elsen, J. H. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” in International Conference on Learning Representations (ICLR), 2019. find
  62. [64] H. Delgado, N. W. D. Evans, T. Kinnunen, K. A. Lee, X. Liu, A. Nautsch, J. Patino, M. Sahidullah, M. Todisco, X. Wang, and J. Yamagishi, “Asvspoof 2021: Automatic speaker verification spoofing and countermeasures challenge evaluation plan,” arXiv:2109.00535, 2021. — [arXiv:2109.00535](https://arxiv.org/abs/2109.00535) find
  63. [65] E. Casanova, J. Weber, C. D. Shulby, A. C. Júnior, E. Gölge, and M. A. Ponti, “Yourtts: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone,” in International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 2022… find