Elamur Research
☰
home / papers / 2021 / SoundStream: An End-to-End Neural Audio Codec

SoundStream: An End-to-End Neural Audio Codec

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2107.03312

Summary

Classic audio codecs involve a trade-off. Waveform codecs handle any content but produce artifacts at low bitrates. Parametric codecs stay efficient by assuming the input is speech. Earlier neural codecs such as LPCNet and Lyra also focused only on low-bitrate speech. Zeghidour et al. (Google) present SoundStream, a learned codec for speech, music and general audio at 24 kHz. A fully convolutional encoder turns the waveform into lower-rate embeddings. A residual vector quantizer (RVQ), extending the VQ-VAE quantizer, discretizes them. A convolutional decoder then rebuilds the signal. All three parts are trained jointly using reconstruction losses plus adversarial losses from discriminators, following MelGAN/HiFiGAN-style generators. Randomly dropping quantizer stages during training lets one model serve many bitrates. Causal convolutions keep latency low enough for streaming. In listening tests, SoundStream at 3 kbps beat Opus at 12 kbps and came close to EVS at 9.6 kbps. This shows that end-to-end learned compression can beat strong standard codecs across content types while running on a phone.

Key points

  • One model covers bitrates from 3 to 18 kbps thanks to "quantizer dropout" on RVQ layers. Its quality is nearly the same as models trained for a single fixed bitrate.
  • At 24 kHz, SoundStream at 3 kbps scores above Opus at 12 kbps in subjective tests and nears EVS at 9.6 kbps.
  • The architecture is streamable. Its latency depends only on the encoder's downsampling ratio, and it runs in real time on a single smartphone CPU thread.
  • Learning the encoder, instead of using fixed mel-spectrogram features as Lyra does, gives a large gain in coding efficiency.
  • An optional conditioning signal lets the codec also suppress background noise in speech, at either the encoder or the decoder, with no extra latency.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

117 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 107 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2107.03312 · arXiv 2107.03312

@misc{zeghidour2021soundstream,
  title = {{SoundStream: An End-to-End Neural Audio Codec}},
  author = {Zeghidour, Neil and Luebs, Alejandro and Omran, Ahmed and Skoglund, Jan and Tagliasacchi, Marco},
  year = {2021},
  doi = {10.48550/arxiv.2107.03312},
  eprint = {2107.03312},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2107.03312}
}

References

62 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Li, M. Tagliasacchi, O. Rybakov, V. Ungureanu, and D. Roblek, “Real-time speech frequency bandwidth extension,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 691–695. find
  2. A. Biswas and D. Jia, “Audio codec enhancement with generative adversarial networks,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 356–360. find
  3. F. Stimberg, A. Narest, A. Bazzica, L. Kolmodin, P. Barrera González, O. Sharonova, H. Lundin, and T. C. Walters, “WaveNetEQ — Packet loss concealment with WaveRNN,” in 54th Asilomar Conference on Signals, Systems, and Computers, 2020, pp. 672–676. find
  4. A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A generative model for raw audio,” arXiv:1609.03499, 2016. find our card
  5. W. B. Kleijn, F. S. Lim, A. Luebs, J. Skoglund, F. Stimberg, Q. Wang, and T. C. Walters, “Wavenet based low rate speech coding,” in IEEE international conference on acoustics, speech and signal processing (ICASSP), 2018, pp. 676–680. find
  6. C. Gârbacea, A. van den Oord, Y. Li, F. S. C. Lim, A. Luebs, O. Vinyals, and T. C. Walters, “Low bit-rate speech coding with VQ-VAE and a WaveNet decoder,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 735–739. find
  7. J.-M. Valin and J. Skoglund, “LPCNet: improving neural speech synthesis through linear prediction,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 5891–5895. find
  8. W. B. Kleijn, A. Storus, M. Chinen, T. Denton, F. S. C. Lim, A. Luebs, J. Skoglund, and H. Yeh, “Generative speech coding with predictive variance regularization,” arXiv:2102.09660, 2021. — [arXiv:2102.09660](https://arxiv.org/abs/2102.09660) find
  9. J.-M. Valin, K. Vos, and T. B. Terriberry, “Definition of the Opus Audio Codec,” IETF RFC 6716, 2012, https://tools.ietf.org/html/rfc6716. ↗
  10. M. Dietz, M. Multrus, V. Eksler, V. Malenovsky, E. Norvell, H. Pobloth, L. Miao, Z. Wang, L. Laaksonen, A. Vasilache, Y. Kamamoto, K. Kikuiri, S. Ragot, J. Faure, H. Ehara, V. Rajendran, V. Atti, H. Sung, E. Oh, H. Yuan, and C. Zhu, “Overview of the EVS codec architecture,” in IEEE International Con… find
  11. S. Mehri, K. Kumar, I. Gulrajani, R. Kumar, S. Jain, J. Sotelo, A. Courville, and Y. Bengio, “SampleRNN: An unconditional end-to-end neural audio generation model,” arXiv:1612.07837, 2017. — [arXiv:1612.07837](https://arxiv.org/abs/1612.07837) find
  12. A. van den Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. van den Driessche, E. Lockhart, L. Cobo, F. Stimberg, N. Casagrande, D. Grewe, S. Noury, S. Dieleman, E. Elsen, N. Kalchbrenner, H. Zen, A. Graves, H. King, T. Walters, D. Belov, and D. Hassabis, “Parallel WaveNet: Fa… find
  13. N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. van den Oord, S. Dieleman, and K. Kavukcuoglu, “Efficient neural audio synthesis,” arXiv:1802.08435, 2018. find
  14. Z. Jin, A. Finkelstein, G. J. Mysore, and J. Lu, “FFTNet: a real-time speaker-dependent neural vocoder,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 2251–2255. find
  15. K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. de Brebisson, Y. Bengio, and A. Courville, “MelGAN: Generative adversarial networks for conditional waveform synthesis,” in Advances in Neural Information Processing Systems, 2019. find
  16. J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative Adversarial Networks for efficient and high fidelity speech synthesis,” arXiv:2010.05646, 2020. find
  17. X. Feng, Y. Zhang, and J. Glass, “Speech feature denoising and dereverberation via deep autoencoders for noisy reverberant speech recognition,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 1759–1763. find
  18. S. Pascual, A. Bonafonte, and J. Serra, “SEGAN: Speech enhancement generative adversarial network,” arXiv:1703.09452, 2017. — [arXiv:1703.09452](https://arxiv.org/abs/1703.09452) find
  19. F. G. Germain, Q. Chen, and V. Koltun, “Speech denoising with deep feature losses,” arXiv:1806.10522, 2018. — [arXiv:1806.10522](https://arxiv.org/abs/1806.10522) find
  20. D. Rethage, J. Pons, and X. Serra, “A WaveNet for speech denoising,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 5069–5073. find
  21. C. Donahue, B. Li, and R. Prabhavalkar, “Exploring speech enhancement with generative adversarial networks for robust speech recognition,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 5024–5028. find
  22. T. Ishii, H. Komiyama, T. Shinozaki, Y. Horiuchi, and S. Kuroiwa, “Reverberant speech recognition based on denoising autoencoder.” in Interspeech, 2013, pp. 3512–3516. find
  23. D. S. Williamson and D. Wang, “Time-frequency masking in the complex domain for speech dereverberation and denoising,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, pp. 1492–1501, 2017. find
  24. T. Y. Lim, R. A. Yeh, Y. Xu, M. N. Do, and M. Hasegawa-Johnson, “Time-frequency networks for audio super-resolution,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 646–650. find
  25. S. Lloyd, “Least squares quantization in PCM,” IEEE transactions on information theory, vol. 28, pp. 129–137, 1982. find
  26. Y. Linde, A. Buzo, and R. Gray, “An algorithm for vector quantizer design,” IEEE Transactions on Communications, vol. 28, pp. 84–95, 1980. find
  27. J. MacQueen, “Some methods for classification and analysis of multivariate observations,” Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, pp. 281–297, 1967. find
  28. R. Gray, “Vector quantization,” IEEE ASSP Magazine, vol. 1, pp. 4–29, 1984. find
  29. J. Makhoul, S. Roucos, and H. Gish, “Vector quantization in speech coding,” Proceedings of the IEEE, vol. 73, pp. 1551–1588, 1985. find
  30. M. Schroeder and B. Atal, “Code-excited linear prediction (CELP): High-quality speech at very low bit rates,” in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 1985, pp. 937–940. find
  31. A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” arXiv:1711.00937, 2017. — [arXiv:1711.00937](https://arxiv.org/abs/1711.00937) find
  32. A. Razavi, A. van den Oord, and O. Vinyals, “Generating diverse high-fidelity images with VQ-VAE-2,” arXiv:1906.00446, 2019. — [arXiv:1906.00446](https://arxiv.org/abs/1906.00446) find
  33. S. Dieleman, A. van den Oord, and K. Simonyan, “The challenge of realistic music generation: Modelling raw audio at scale,” arXiv:1806.10474, 2018. — [arXiv:1806.10474](https://arxiv.org/abs/1806.10474) find
  34. P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A generative model for music,” arXiv:2005.00341, 2020. — [arXiv:2005.00341](https://arxiv.org/abs/2005.00341) find
  35. B.-H. Juang and A. Gray, “Multiple stage vector quantization for speech coding,” in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 1982, pp. 597–600. find
  36. A. Vasuki and P. Vanathi, “A review of vector quantization techniques,” IEEE Potentials, vol. 25, pp. 39–47, 2006. find
  37. S. Morishima, H. Harashima, and Y. Katayama, “Speech coding based on a multi-layer neural network,” in IEEE International Conference on Communications, Including Supercomm Technical Sessions, 1990, pp. 429–433. find
  38. S. Kankanahalli, “End-to-end optimized speech coding with deep neural networks,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 2521–2525. find
  39. A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux, “Speech resynthesis from discrete disentangled self-supervised representations,” arXiv:2104.00355, 2021. — [arXiv:2104.00355](https://arxiv.org/abs/2104.00355) find
  40. K. Zhen, J. Sung, M. S. Lee, S. Beack, and M. Kim, “Cascaded cross-module residual learning towards lightweight end-to-end speech coding,” arXiv:1906.07769, 2019. — [arXiv:1906.07769](https://arxiv.org/abs/1906.07769) find
  41. J. Casebeer, V. Vale, U. Isik, J.-M. Valin, R. Giri, and A. Krishnaswamy, “Enhancing into the codec: Noise robust speech coding with vector-quantized autoencoders,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 711–715. find
  42. D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (elus),” arXiv preprint arXiv:1511.07289, 2015. find our card
  43. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” The journal of machine learning research, vol. 15, pp. 1929–1958, 2014. find
  44. A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” arXiv:2006.11477, 2020. find our card
  45. M. Tagliasacchi, Y. Li, K. Misiunas, and D. Roblek, “SEANet: A multimodal speech enhancement network,” in Interspeech, 2020, pp. 1126– 1130. find
  46. Y. Blau and T. Michaeli, “The perception-distortion tradeoff,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 6228–6237. find
  47. J. Engel, L. H. Hantrakul, C. Gu, and A. Roberts, “DDSP: Differentiable digital signal processing,” arXiv:2001.04643, 2020. — [arXiv:2001.04643](https://arxiv.org/abs/2001.04643) find
  48. A. A. Gritsenko, T. Salimans, R. van den Berg, J. Snoek, and N. Kalchbrenner, “A spectral energy distance for parallel speech synthesis,” arXiv:2008.01160, 2020. — [arXiv:2008.01160](https://arxiv.org/abs/2008.01160) find
  49. E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville, “FiLM: Visual reasoning with a general conditioning layer,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, pp. 3942–3951, 2018. find our card
  50. H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu, “LibriTTS: a corpus derived from LibriSpeech for text-to-speech,” arXiv:1904.02882, 2019. find our card
  51. E. Fonseca, J. Pons Puig, X. Favory, F. Font Corbera, D. Bogdanov, A. Ferraro, S. Oramas, A. Porter, and X. Serra, “Freesound datasets: a platform for the creation of open audio datasets,” in Proceedings of the 18th ISMIR Conference, 2017, pp. 486–493. find
  52. E. Law, K. West, M. I. Mandel, M. Bay, and J. S. Downie, “Evaluation of algorithms using games: The case of music tagging.” in ISMIR, 2009, pp. 387–392. find
  53. ITU-R, Recommendation BS.1534-1: Method for the subjective assessment of intermediate quality level of coding systems, International Telecommunications Union, 2001. find
  54. ITU, “Perceptual evaluation of speech quality (PESQ): an objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs,” Int. Telecomm. Union, Geneva, Switzerland, ITU-T Rec. P.862, 2001. find
  55. ——, “Perceptual objective listening quality assessment,” Int. Telecomm. Union, Geneva, Switzerland, ITU-T Rec. P.863, 2018. find
  56. A. Hines, J. Skoglund, A. Kokaram, and N. Harte, “ViSQOL: The virtual speech quality objective listener,” in International Workshop on Acoustic Signal Enhancement (IWAENC), 2012, pp. 1–4. find
  57. M. Chinen, F. S. C. Lim, J. Skoglund, N. Gureev, F. O’Gorman, and A. Hines, “ViSQOL v3: an open source production ready objective speech and audio metric,” in Twelfth International Conference on Quality of Multimedia Experience (QoMEX), 2020, pp. 1–6. find
  58. W3C, “WebRTC 1.0: Real-time communication between browsers,” 2019, https://www.w3.org/TR/webrtc/. ↗
  59. C. Holmberg, S. Håkansson, and G. Eriksson, “Web real-time communication use cases and requirements,” IETF RFC 7478, Mar. 2015, https://tools.ietf.org/html/rfc7478. ↗
  60. B. Bessette, R. Salami, R. Lefebvre, M. Jelinek, J. Rotola-Pukkila, J. Vainio, H. Mikkola, and K. Jarvinen, “The adaptive multirate wideband speech codec (AMR-WB),” IEEE Transactions on Speech and Audio Processing, vol. 10, pp. 620–636, 2002. find
  61. F. Mentzer, G. Toderici, M. Tschannen, and E. Agustsson, “High-fidelity generative image compression,” arXiv:2006.09965, 2020. — [arXiv:2006.09965](https://arxiv.org/abs/2006.09965) find
  62. J. Yamagishi, C. Veaux, and K. MacDonald, “CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92),” 2019. find