Elamur Research
☰
home / papers / 2022 / High Fidelity Neural Audio Compression

High Fidelity Neural Audio Compression

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2210.13438

Summary

Lossy audio codecs must lower bitrate without audible degradation, but neural codecs risk artifacts on unfamiliar signals and can be computationally heavy. The paper introduces EnCodec, an end-to-end trained neural codec. It pairs a convolutional encoder with a mirrored decoder, both using LSTM layers, and discretizes the latent output with Residual Vector Quantization (RVQ). Training combines time- and frequency-domain reconstruction terms with one multi-scale STFT discriminator (MS-STFTD). This single adversary suppresses artifacts and trains faster than stacking several discriminators. A new gradient balancer treats each loss weight as that loss's share of the total gradient, so weights no longer depend on loss magnitude and training becomes more stable. An optional small Transformer language model then entropy-codes the discrete codes, reducing bandwidth by up to 40%. In MUSHRA listening tests on clean speech, noisy-reverberant speech and music, EnCodec beats baselines such as Opus for both 24 kHz mono and 48 kHz stereo audio. Because it runs in real time on a single CPU core and its code and models are public, it is both a practical codec and a source of discrete audio representations.

Key points

  • Supported bitrates are 1.5, 3, 6 and 12 kbps for 24 kHz mono and 6, 12 and 24 kbps for 48 kHz stereo. The encoder produces 75 latent frames per second at 24 kHz and 150 at 48 kHz.
  • On 48 kHz stereo music, EnCodec at 6 kbps scores 82.9 MUSHRA. This compares with 17.7 for Opus at 6 kbps and 82.7 for MP3 at 64 kbps. At 12 kbps EnCodec reaches 88.0, roughly matching its 24 kbps result.
  • Entropy coding with the Transformer saves about 20–30% on stereo music and up to 40% overall.
  • The streaming 24 kHz model has an initial latency of about 13 ms and runs roughly 10× faster than real time on one MacBook Pro 2019 CPU thread, though it is slower than Lyra v2. The non-streaming 48 kHz model has 1 s latency and drops below real time when entropy coding is used.
  • Ablations show that MS-STFTD alone is enough for high-quality output, with adding MPD giving only a slight gain. Code and models are released at github.com/facebookresearch/encodec.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

135 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 127 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2210.13438 · arXiv 2210.13438

@misc{defossez2022high,
  title = {{High Fidelity Neural Audio Compression}},
  author = {Defossez, Alexandre and Copet, Jade and Synnaeve, Gabriel and Adi, Yossi},
  year = {2022},
  doi = {10.48550/arxiv.2210.13438},
  eprint = {2210.13438},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2210.13438}
}

References

77 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pavel Andreev, Aibek Alanov, Oleg Ivanov, and Dmitry Vetrov. Hifi++: a unified framework for neural vocoding, bandwidth extension and speech enhancement. arXiv preprint arXiv:2203.13086, 2022. — [arXiv:2203.13086](https://arxiv.org/abs/2203.13086) find
  2. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019. find our card
  3. Bishnu S Atal and Suzanne L Hanauer. Speech analysis and synthesis by linear prediction of the speech wave. The journal of the acoustical society of America, 50(2B):637–655, 1971. find
  4. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. arXiv:1607.06450, 2016. find our card
  5. Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. In ICLR, 2017. find
  6. Johannes Ballé, Nick Johnston, and David Minnen. Integer networks for data compression with latent-variable models. In International Conference on Learning Representations, 2018. find
  7. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. find
  8. Dipesh Bhagat, Ninad Bhatt, and Yogeshwar Kosta. Adaptive multi-rate wideband speech codec based on celp algorithm: architectural study, implementation & performance analysis. In 2012 International Conference on Communication Systems and Network Technologies, pp. 547–551. find
  9. Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mtg-jamendo dataset for automatic music tagging. In Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML 2019), Long Beach, CA, United States, 2019. URL http: //hdl.handle… find
  10. Shlomo E Chazan, Lior Wolf, Eliya Nachmani, and Yossi Adi. Single channel voice separation for unknown number of speakers under reverberant and noisy settings. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3730–3734. find
  11. Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O’Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In 2020 twelfth international conference on quality of multimedia experience (QoMEX), pp. 1–6. find
  12. Cisco. Global - 2021 forecast highlights - cisco. https://www.cisco.com/c/dam/m/en\_us/solutions/service-provider/vni-forecast-highlights/pdf/Global\_2021\_Forecast\_Highlights.pdf, 2021. ↗
  13. Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015. find our card
  14. Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach. Music source separation in the waveform domain. arXiv preprint arXiv:1911.13254, 2019. — [arXiv:1911.13254](https://arxiv.org/abs/1911.13254) find
  15. Alexandre Defossez, Gabriel Synnaeve, and Yossi Adi. Real time speech enhancement in the waveform domain. arXiv preprint arXiv:2006.12847, 2020. — [arXiv:2006.12847](https://arxiv.org/abs/2006.12847) find
  16. Alexandre Défossez, Yossi Adi, and Gabriel Synnaeve. Differentiable model compression via pseudo quantization noise. arXiv preprint arXiv:2104.09987, 2021. — [arXiv:2104.09987](https://arxiv.org/abs/2104.09987) find
  17. Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020. — [arXiv:2005.00341](https://arxiv.org/abs/2005.00341) find
  18. Sander Dieleman, Aaron van den Oord, and Karen Simonyan. The challenge of realistic music generation: modelling raw audio at scale. Advances in Neural Information Processing Systems, 31, 2018. find
  19. Layer normalization. arXiv preprint Martin Dietz, Markus Multrus, Vaclav Eksler, Vladimir Malenovsky, Erik Norvell, Harald Pobloth, Lei Miao, Zhe Wang, Lasse Laaksonen, Adriana Vasilache, et al. Overview of the evs codec architecture. In 2015 IEEE International Conference on Acoustics, Speech and Si… find
  20. Harishchandra Dubey, Vishak Gopal, Ross Cutler, Sergiy Matusevych, Sebastian Braun, Emre Sefik Eskimez, Manthan Thakker, Takuya Yoshioka, Hannes Gamper, and Robert Aichner. Icassp 2022 deep noise suppression challenge. In ICASSP, 2022. find
  21. Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 829–852, 2021. find
  22. Cristina Gârbacea, Aäron van den Oord, Yazhe Li, Felicia SC Lim, Alejandro Luebs, Oriol Vinyals, and Thomas C Walters. Low bit-rate speech coding with vq-vae and a wavenet decoder. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 735–739. find
  23. Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776–780. find
  24. Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It’s raw! audio generation with state-space models. arXiv preprint arXiv:2202.09729, 2022. — [arXiv:2202.09729](https://arxiv.org/abs/2202.09729) find
  25. Robert Gray. Vector quantization. IEEE Assp Magazine, 1(2):4–29, 1984. find
  26. D Griffin and Jae Lim. A new model-based speech analysis/synthesis system. In ICASSP, 1985. find
  27. Alexey Gritsenko, Tim Salimans, Rianne van den Berg, Jasper Snoek, and Nal Kalchbrenner. A spectral energy distance for parallel speech synthesis. Advances in Neural Information Processing Systems, 33: 13062–13072, 2020. find
  28. Andrew Hines, Jan Skoglund, Anil Kokaram, and Naomi Harte. Visqol: The virtual speech quality objective listener. In IWAENC 2012; International Workshop on Acoustic Signal Enhancement, pp. 1–4. find
  29. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021. find our card
  30. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Technical Report 1502.03167, arXiv, 2015. find
  31. Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2017. find our card
  32. Tejas Jayashankar, Thilo Koehler, Kaustubh Kalgaonkar, Zhiping Xiu, Jilong Wu, Ju Lin, Prabhav Agrawal, and Qing He. Architecture for variable bitrate neural speech codec with configurable computation complexity. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Proce… find
  33. Xue Jiang, Xiulian Peng, Chengyu Zheng, Huaying Xue, Yuan Zhang, and Yan Lu. End-to-end neural speech coding for real-time communications. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 866–870. find
  34. Biing-Hwang Juang and A Gray. Multiple stage vector quantization for speech coding. In ICASSP’82. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 7, pp. 597–600. IEEE, 1982. find
  35. Nal Kalchbrenner et al. Efficient Neural Audio Synthesis. In ICML, 2018. find
  36. Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, et al. Text-free prosody-aware generative spoken language modeling. arXiv preprint arXiv:2109.03264, 2021. find
  37. W Bastiaan Kleijn, Felicia SC Lim, Alejandro Luebs, Jan Skoglund, Florian Stimberg, Quan Wang, and Thomas C Walters. Wavenet based low rate speech coding. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 676–680. find
  38. W Bastiaan Kleijn, Andrew Storus, Michael Chinen, Tom Denton, Felicia SC Lim, Alejandro Luebs, Jan Skoglund, and Hengchin Yeh. Generative speech coding with predictive variance regularization. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.… find
  39. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022–17033, 2020. find
  40. Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. Textless speech emotion conversion using decomposed and discrete representations. arXiv preprint arXiv:2111.07402, 2021. — [arXiv:2111.07402](https://arxiv.org/abs/2111.07402) find
  41. Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019. find
  42. Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336–1354, 2021. find
  43. Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, et al. Direct speech-to-speech translation with discrete units. arXiv preprint arXiv:2107.05604, 2021a. find
  44. Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. Textless speech-to-speech translation on real data. arXiv preprint arXiv:2112.08352, 2021b. find
  45. Shuyang Li, Huanru Henry Mao, and Julian McAuley. Variable bitrate discrete neural representations via causal self-attention. In 2nd Pre-registration workshop (NeurIPS 2021), Remote. find
  46. Yunpeng Li, Marco Tagliasacchi, Oleg Rybakov, Victor Ungureanu, and Dominik Roblek. Real-time speech frequency bandwidth extension. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 691–695. find
  47. Felicia SC Lim, W Bastiaan Kleijn, Michael Chinen, and Jan Skoglund. Robust low rate speech coding based on cloned networks and wavenet. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6769–6773. find
  48. Ju Lin, Kaustubh Kalgaonkar, Qing He, and Xin Lei. Speech enhancement for low bit rate speech codec. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7777–7781. find
  49. Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation. IEEE/ACM transactions on audio, speech, and language processing, 27(8):1256–1266, 2019. find
  50. Alan McCree, Kwan Truong, E Bryan George, Thomas P Barnwell, and Vishu Viswanathan. A 2.4 kbit/s melp coder candidate for the new us federal standard. In ICASSP, 1996. find
  51. Shigeo Morishima, H Harashima, and Y Katayama. Speech coding based on a multi-layer neural network. In IEEE International Conference on Communications, Including Supercomm Technical Sessions, pp. 429–433. IEEE, 1990. find
  52. Eliya Nachmani, Yossi Adi, and Lior Wolf. Voice separation with an unknown number of multiple speakers. In International Conference on Machine Learning, pp. 7164–7175. PMLR, 2020. find
  53. Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. Generative spoken dialogue language modeling. arXiv preprint arXiv:2203.16502, 2022. find
  54. Ahmed Omran, Neil Zeghidour, Zalán Borsos, Félix de Chaumont Quitry, Malcolm Slaney, and Marco Tagliasacchi. Disentangling speech from surroundings in a neural audio codec. arXiv preprint arXiv:2203.15578, 2022. — [arXiv:2203.15578](https://arxiv.org/abs/2203.15578) find
  55. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016. find our card
  56. Richard Clark Pasco. Source coding algorithms for fast data compression. PhD thesis, Stanford University CA, 1976. find
  57. Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00355, 2021. — [arXiv:2104.00355](https://arxiv.org/abs/2104.00355) find
  58. Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. Enhanced direct speech-to-speech translation using self-supervised pre-training and data augmentation. arXiv preprint arXiv:2204.02967, 2022. — [arXiv:2204.02967](https://arxiv.org/abs/2204.02967) find
  59. Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019. find
  60. Oren Rippel, Sanjay Nair, Carissa Lew, Steve Branson, Alexander G Anderson, and Lubomir Bourdev. Learned video compression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3454–3463, 2019. find
  61. Jorma Rissanen and Glen Langdon. Universal modeling and coding. IEEE Transactions on Information Theory, 27(1):12–23, 1981. find
  62. Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016. find
  63. B Series. Method for the subjective assessment of intermediate quality level of audio systems. International Telecommunication Union Radiocommunication Assembly, 2014. find
  64. Jan Skoglund and Jean-Marc Valin. Improving opus low bit rate quality with neural speech synthesis. arXiv preprint arXiv:1905.04628, 2019. — [arXiv:1905.04628](https://arxiv.org/abs/1905.04628) find
  65. Marco Tagliasacchi, Yunpeng Li, Karolis Misiunas, and Dominik Roblek. Seanet: A multi-modal speech enhancement network. arXiv preprint arXiv:2009.02095, 2020. — [arXiv:2009.02095](https://arxiv.org/abs/2009.02095) find
  66. Jean-Marc Valin and Jan Skoglund. Lpcnet: Improving neural speech synthesis through linear prediction. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5891–5895. find
  67. Jean-Marc Valin and Jan Skoglund. A real-time wideband neural vocoder at 1.6 kb/s using lpcnet. arXiv preprint arXiv:1903.12087, 2019b. — [arXiv:1903.12087](https://arxiv.org/abs/1903.12087) find
  68. Jean-Marc Valin, Koen Vos, and Timothy Terriberry. Definition of the opus audio codec. IETF, September, 2, 2012. find
  69. Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. find
  70. A Vasuki and PT Vanathi. A review of vector quantization techniques. IEEE Potentials, 25(4):39–47, 2006. find
  71. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of Neural Information Processing Systems, 2017. find
  72. Bernard Widrow, Istvan Kollar, and Ming-Chang Liu. Statistical theory of quantization. IEEE Transactions on instrumentation and measurement, 45(2):353–361, 1996. find
  73. Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6199–6203. find
  74. Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6199–6203. find
  75. Jaeseong You, Dalhyun Kim, Gyuhyeon Nam, Geumbyeol Hwang, and Gyeongsu Chae. Gan vocoder: Multi-resolution discriminator is all you need. arXiv preprint arXiv:2103.05236, 2021. — [arXiv:2103.05236](https://arxiv.org/abs/2103.05236) find
  76. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021. find our card
  77. Table A.1: Datasets description. License with asterisk annotation * imply that the specific license varies across the dataset and is specific to each sample. Dataset Audio domain Common Voice 7.0 DNS Challenge 4 (speech) AudioSet FSD50K Jamendo Speech Speech General audio General audio Music find