Elamur Research⌕
☰
home / papers / 2026 / BSCodec: A Band-Split Neural Codec for High-Quality Universal Audio Reconstruction

BSCodec: A Band-Split Neural Codec for High-Quality Universal Audio Reconstruction

Paper · 2026 · Haoran Wang, Jiatong Shi, Jinchuan Tian et al.

Summary

Neural audio codecs tuned for speech tend to degrade on music and environmental sound, whose energy is spread differently across the spectrum. Residual-quantization codecs such as DAC generalize better, but their layers do not map onto any physical structure of the signal. The authors propose BSCodec, which splits the input audio into non-overlapping frequency bands. They do this by masking an STFT spectrogram at 24 kHz and converting each band back to a waveform. Each band then passes through its own encoder, quantizer and decoder, and the band outputs are summed to form the reconstruction. Training follows a DAC-like adversarial recipe on a mixed corpus of speech, music and sound. The authors study band counts, frequency boundaries and codebook sizes systematically. Their 2.55 and 3.83 kbps models roughly match DAC at 4.5 and 6 kbps, which amounts to about half the bitrate. Codec-SUPERB and ARCH evaluations show that the tokens and encoder features are also useful for downstream understanding tasks. The work suggests that a simple, physically motivated band decomposition can replace complex domain-specific mechanisms in universal codecs.

Key points

  • Architecture:
    • Encoders are unshared SEANet networks with strides [2, 4, 5, 8], producing 512-dimensional latents at 75 Hz.
    • Each band uses a single-layer SimVQ, which supports large codebooks.
    • Code is released in an ESPnet fork (github.com/whr-a/espnet, branch bscodec).
  • Codec-SUPERB results: The 3-band model at 3.83 kbps was compared with DAC at 6 kbps.
    • It beats DAC on speaker verification (EER 2.67 vs 3.94).
    • It beats DAC on audio event classification (mAP 87.95 vs 78.15).
    • It also beats DAC on emotion recognition (ACC 73.26 vs 72.36).
    • ASR is slightly worse (WER 4.40 vs 4.10).
  • Band count: Dropping the high-frequency band and its 17-bit codebook (3 → 2 bands, 2.55 kbps) barely affects speech but clearly hurts music and sound. Five bands harm speech tasks, so three bands is the best trade-off.
  • ARCH benchmark (frozen encoder plus linear probe): All band-split variants beat the single-encoder baseline. Five bands work best for speech and audio, while music gains nothing beyond a moderate number of bands.
  • Ablation: Swapping in SimVQ or Residual SimVQ gives no significant gains, so the improvements come from band splitting rather than the quantizer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 480 works

Cite

Cite as Haoran Wang, Jiatong Shi, Jinchuan Tian, Bohan Li, Kai Yu and Shinji Watanabe. 2026. BSCodec: A Band-Split Neural Codec for High-Quality Universal Audio Reconstruction. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.245

@inproceedings{wang2026bscodec,
  title = {{BSCodec: A Band-Split Neural Codec for High-Quality Universal Audio Reconstruction}},
  author = {Haoran Wang, Jiatong Shi, Jinchuan Tian, Bohan Li, Kai Yu and Shinji Watanabe},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.245},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.245}
}

References

58 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670.
  2. Sören Becker, Johanna Vielhaben, Marcel Ackermann, Klaus-Robert Müller, Sebastian Lapuschkin, and Wojciech Samek. 2024. Audiomnist: Exploring explainable artificial intelligence for audio analysis on a simple benchmark. Journal of the Franklin Institute, 361(1):418–428. find
  3. Rachel M Bittner, Justin Salamon, Mike Tierney, Matthias Mauch, Chris Cannam, and Juan Pablo Bello. 2014. Medleydb: A multitrack dataset for annotation-intensive mir research. In Ismir, volume 14, pages 155–160. find
  4. Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. 2019. The mtg-jamendo dataset for automatic music tagging. ICML. find
  5. Karlheinz Brandenburg. 1999. Mp3 and aac explained. In Audio Engineering Society Conference: 17th International Conference: High-Quality Audio Coding. Audio Engineering Society. find
  6. Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438.
  7. Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 77… find
  8. Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems, 27. find
  9. Yiwei Guo, Zhihan Li, Hankun Wang, Bohan Li, Chongtian Shao, Hanglei Zhang, Chenpeng Du, Xie Chen, Shujie Liu, and Kai Yu. 2025. Recent advances in discrete speech tokens: A review. arXiv preprint arXiv:2502.06490. — [arXiv:2502.06490](https://arxiv.org/abs/2502.06490) find
  10. Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck. 2019. Enabling factorized piano music modeling and generation with the MAESTRO dataset. In International Conference on Learning Representations. find
  11. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15979–15988.
  12. Andrew Hines, Jan Skoglund, Anil C Kokaram, and Naomi Harte. 2015. Visqol: an objective speech quality model. EURASIP Journal on Audio, Speech, and Music Processing, 2015(1):13. find
  13. Natalie Holz, Pauline Larrouy-Maestri, and David Poeppel. 2022. The variably intense vocalizations of affect and emotion (vivae) corpus prompts new perspective on nonspeech perception. Emotion, 22(1):213. find
  14. Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. 2022. Masked autoencoders that listen. Advances in Neural Information Processing Systems, 35:28708–28720. find
  15. Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, and 1 others. 2024. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint arXiv:2408.16532. — [arXiv:2408.16532](https://arxiv.org/abs/2408.16532) find
  16. Yidi Jiang, Qian Chen, Shengpeng Ji, Yu Xi, Wen Wang, Chong Zhang, Xianghu Yue, ShiLiang Zhang, and Haizhou Li. 2025. Unicodec: Unified audio codec with single domain-adaptive codebook. arXiv preprint arXiv:2502.20067. find
  17. Jee-weon Jung, Wangyou Zhang, Jiatong Shi, Zakaria Aldeneh, Takuya Higuchi, Barry-John Theobald, Ahmed Hussen Abdelaziz, and Shinji Watanabe. 2024. Espnet-spk: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models. arXiv preprint arXi… — [arXiv:2401.17230](https://arxiv.org/abs/2401.17230) find
  18. Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. 2023. High-fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Systems, 36:27980–27993. find
  19. Moreno La Quatra, Alkis Koudounas, Lorenzo Vaiani, Elena Baralis, Luca Cagliero, Paolo Garza, and Sabato Marco Siniscalchi. 2024. Benchmarking representations for speech, music, and acoustic events. In 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICAS-SPW… find
  20. Edith Law, Kris West, M Mandel, M Bay, and JS Downie. 2010. Evaluation of algorithms using games: the case of music annotation. In Proceedings of the 11th International Society for Music Information Retrieval Conference (ISMIR). Utrecht, the Netherlands. find
  21. Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. 2024. Semanticodec: An ultra low bitrate semantic audio codec for general sound. IEEE Journal of Selected Topics in Signal Processing. find
  22. Steven R Livingstone and Frank A Russo. 2018. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5):e0196391. find
  23. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. find
  24. Yi Luo and Jianwei Yu. 2023. Music source separation with band-split rnn. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:1893– 1901. find
  25. Yi Luo, Jianwei Yu, Hangting Chen, Rongzhi Gu, and Chao Weng. 2024. Gull: A generative multifunctional audio codec. arXiv preprint arXiv:2404.04947. — [arXiv:2404.04947](https://arxiv.org/abs/2404.04947) find
  26. Henrique S Malvar. 1990. Modulated qmf filter banks with perfect reconstruction. Electronics letters, 26(13):906–907. find
  27. Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. 2023. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505. — [arXiv:2309.15505](https://arxiv.org/abs/2309.15505) find
  28. Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. 2001. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE international conference on acoustics, speech, and signal processing. Proceedings (C… find
  29. Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152. find
  30. Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. 2014. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pages 1041–1044. find
  31. Tapio Saramaki and Robert Bregovic. 2002. Multirate systems and filterbanks. In Multirate systems: design and applications, pages 27–85. IGI Global Scientific Publishing. find
  32. Bertram Scharf. 1970. Critical bands. Foundations of modern auditory theory, 1:157–202. find
  33. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538. find
  34. Jiatong Shi, Hye-jin Shim, Jinchuan Tian, Siddhant Arora, Haibin Wu, Darius Petermann, Jia Qi Yip, You Zhang, Yuxun Tang, Wangyou Zhang, and 1 others. 2024a. Versa: A versatile evaluation toolkit for speech, audio, and music. arXiv preprint arXiv:2412.17667. find
  35. Pooneh Mousavi, Gallil Maimon, Adel Moumen, Darius Petermann, Jiatong Shi, Haibin Wu, Haici Yang, Anastasia Kuznetsova, Artem Ploujnikov, Ricard Marxer, and 1 others. 2025. Discrete audio tokens: More than a survey! arXiv preprint arXiv:2506.10274. — [arXiv:2506.10274](https://arxiv.org/abs/2506.10274) find
  36. Jiatong Shi, Jinchuan Tian, Yihan Wu, Jee-weon Jung, Jia Qi Yip, Yoshiki Masuyama, William Chen, Yuning Wu, Yuxun Tang, Massa Baali, and 1 others. 2024b. Espnet-codec: Comprehensive training and evaluation of neural codecs for audio, music, and speech. In 2024 IEEE Spoken language technology worksho… find
  37. Dianwen Ng, Kun Zhou, Yi-Wen Chao, Zhiwei Xiong, Bin Ma, and EngSiong Chng. 2025. Multi-band frequency reconstruction for neural psychoacoustic coding. In Forty-second International Conference on Machine Learning. find
  38. Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. 2010. A short-time objective intelligibility measure for time-frequency weighted noisy speech. In 2010 IEEE international conference on acoustics, speech and signal processing, pages 4214– 4217. IEEE. find
  39. Karol J Piczak. 2015. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015– 1018. find
  40. Marco Tagliasacchi, Yunpeng Li, Karolis Misiunas, and Dominik Roblek. 2020. Seanet: A multimodal speech enhancement network. arXiv preprint arXiv:2009.02095. — [arXiv:2009.02095](https://arxiv.org/abs/2009.02095) find
  41. Mathias Prokop. 2003. General principles of mdct. European journal of radiology, 45:S4–S10. find
  42. Jinchuan Tian, Jiatong Shi, William Chen, Siddhant Arora, Yoshiki Masuyama, Takashi Maekaku, Yihan Wu, Junyi Peng, Shikhar Bharadwaj, Yiwen Zhao, and 1 others. 2025. Espnet-speechlm: An open speech language model toolkit. arXiv preprint arXiv:2502.15218. find
  43. Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. 2017. The MUSDB18 corpus for music separation. find
  44. Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, and 1 others. 2023. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111.
  45. Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, and 1 others. 2018. Espnet: End-to-end speech processing toolkit. arXiv preprint arXiv:1804.00015. — [arXiv:1804.00015](https://arxiv.org/abs/1804.00015) find
  46. Haibin Wu, Xuanjun Chen, Yi-Cheng Lin, Kaiwei Chang, Jiawei Du, Ke-Han Lu, Alexander H Liu, Ho-Lam Chung, Yuan-Kuei Wu, Dongchao Yang, and 1 others. 2024. Codec-superb@ slt 2024: A lightweight benchmark for neural audio codec models. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 570–… find
  47. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882.
  48. Dengsheng Zhang. 2019. Wavelet transform. In Fundamentals of image data mining: Analysis, Features, Classification and Retrieval, pages 35–44. Springer. find
  49. Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2023. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308.16692.
  50. Yongxin Zhu, Bocheng Li, Yifei Xin, Zhihua Xia, and Linli Xu. 2024. Addressing representation collapse in vector quantized models with one linear layer. arXiv preprint arXiv:2411.02038. — [arXiv:2411.02038](https://arxiv.org/abs/2411.02038) find
  51. Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2024. Bigcodec: Pushing the limits of low-bitrate neural speech codec. arXiv preprint arXiv:2409.05377. — [arXiv:2409.05377](https://arxiv.org/abs/2409.05377) find
  52. Junichi Yamagishi, Christophe Veaux, and Kirsten Mac-Donald. 2019. CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92). find
  53. Dongchao Yang, Songxiang Liu, Haohan Guo, Jiankun Zhao, Yuanyuan Wang, Helin Wang, Zeqian Ju, Xubo Liu, Xueyuan Chen, Xu Tan, and 1 others. 2025. Almtokenizer: A low-bitrate and semantic-rich audio codec tokenizer for audio language modeling. arXiv preprint arXiv:2504.10344. — [arXiv:2504.10344](https://arxiv.org/abs/2504.10344) find
  54. Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. 2023. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765. find
  55. Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, and Lei Xie. 2021. Multi-band melgan: Faster waveform generation for high-quality text-to-speech. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 492–498. IEEE. find
  56. Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, and 1 others. 2025. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 3… find
  57. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2021. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507.
  58. Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-bscodec-band-split-neural-codec/