Elamur Research⌕
☰
home / papers / 2026 / VocalRep: Structure-Aware Vocal Representations for Multimodal Generation

VocalRep: Structure-Aware Vocal Representations for Multimodal Generation

Paper · 2026 · Da Shen, Zhenqiang Weng, Tianyu Liu et al.

Summary

Generative pipelines such as singing voice conversion (SVC) and lip-sync driven by audio usually get their vocal input from music source separation (MSS). Common separators, however, merge the lead singer and backing harmonies into one vocal stem and are tuned for SDR-style reconstruction. The authors argue that this mixing causes pitch jitter and blurred phoneme cues downstream, even when SDR looks good.

They introduce VocalRep, which splits a mix into three stems: lead, harmony and instrument. Its backbone is a Band-Roformer modulated by a song-level singer identity vector. Training adds two ranking-based discriminative critics. One judges how well the lead fits its instrumental context, and the other models lead–harmony dominance. Neither critic runs at inference, and no pitch or symbolic labels are required.

The system was tested on separation benchmarks and through several SVC and lip-sync systems. VocalRep is competitive on separation scores, but its main gain is in intelligibility, pitch stability and sync quality. The work suggests that downstream generation quality is a better yardstick for separators than reconstruction metrics.

Key points

  • Identity conditioning: the Band-Roformer transformer is conditioned via FiLM on a CAM++ speaker embedding.
    • During training, 20 embeddings from random ground-truth lead segments are averaged, and the vector is dropped with probability 0.5.
    • At inference, sliding-window embeddings from the mixture are spectrally clustered, and the centroid of the dominant cluster serves as the song-level anchor.
  • Baselines: MelBand-Roformer, Band-Roformer, MDX23C and SCNet Large, each also in a "stem-reconfigured" variant.
  • Downstream evaluation:
    • Lip-sync was tested with Wav2Lip, EchoMimic and Hallo2 on the Multisong and PHV-40 datasets, measured by Sync-D and Sync-C.
    • VocalRep reports the best relative lip-sync results.
  • Ablation (PHV-40):
    • Lead/harmony SDR rises from 6.94/3.49 dB (base) to 7.29/4.05 dB (full model with ID correction).
    • RVC F0PCC rises from 88.15% to 90.81%.
    • The dual-scorer critics alone cut CER from 46.44% to 41.95%.
  • Limitations and release:
    • VocalRep scores slightly below specialized models on 2-stem benchmarks because errors accumulate when stems are separated and then summed.
    • The single-singer assumption breaks down for duets.
    • Code is on GitHub (Zhenqiang-Weng/music_source_separation).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓East China University of Science and TechnologyShanghai · China10 works · 87 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Da Shen, Zhenqiang Weng, Tianyu Liu, Gongyu Chen, Runhua Shi, Jiahui Chen, Chaofan Ding, Wei-Qiang Zhang and Zihao Chen. 2026. VocalRep: Structure-Aware Vocal Representations for Multimodal Generation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1785

@inproceedings{shen2026vocalrep,
  title = {{VocalRep: Structure-Aware Vocal Representations for Multimodal Generation}},
  author = {Da Shen, Zhenqiang Weng, Tianyu Liu, Gongyu Chen, Runhua Shi, Jiahui Chen, Chaofan Ding, Wei-Qiang Zhang and Zihao Chen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1785},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1785}
}

References

43 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ye Bai, Chenxing Li, Hao Li, Yuanyuan Zhao, and Xiaorui Wang. 2024. Jointly recognizing speech and singing voices based on multi-task audio source separation. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE. find
  2. Rachel M Bittner, Justin Salamon, Mike Tierney, Matthias Mauch, Chris Cannam, and Juan Pablo Bello. 2014. Medleydb: A multitrack dataset for annotation-intensive mir research. In Ismir, volume 14, pages 155–160. find
  3. Estefanía Cano, Derry FitzGerald, and Karlheinz Brandenburg. 2016. Evaluation of quality of sound source separation algorithms: Human perception vs quantitative metrics. In 2016 24th European Signal Processing Conference (EUSIPCO), pages 1758–1762. IEEE. find
  4. Estefania Cano, Derry FitzGerald, Antoine Liutkus, Mark D Plumbley, and Fabian-Robert Stöter. 2018. Musical source separation: An introduction. IEEE Signal Processing Magazine, 36(1):31–40. find
  5. Junyu Chen, Susmitha Vekkot, and Pancham Shukla. 2024. Music source separation based on a lightweight deep learning framework (dttnet: Dual-path tfc-tdf unet). In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 656–660. IEEE. find
  6. Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. 2025. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2403–2410. find
  7. Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, and Jingdong Wang. 2024. Hallo2: Long-duration and high-resolution audio-driven portrait image animation. arXiv preprint arXiv:2410.07718. — [arXiv:2410.07718](https://arxiv.org/abs/2410.07718) find
  8. Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach. 2019. Music source separation in the waveform domain. arXiv preprint arXiv:1911.13254. — [arXiv:1911.13254](https://arxiv.org/abs/1911.13254) find
  9. Romain Hennequin, Anis Khlif, Felix Voituret, and Manuel Moussallam. 2020. Spleeter: a fast and efficient music source separation tool with pre-trained models. Journal of Open Source Software, 5(50):2154. find
  10. Wen-Chin Huang, Lester Phillip Violeta, Songxiang Liu, Jiatong Shi, and Tomoki Toda. 2023. The singing voice conversion challenge 2023. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. IEEE. find
  11. Noah Jaffe and John Ashley Burgoyne. 2025. Musical source separation bake-off: Comparing objective metrics with human perception. arXiv preprint arXiv:2507.06917. — [arXiv:2507.06917](https://arxiv.org/abs/2507.06917) find
  12. Won Jang, Dan Lim, Jaesam Yoon, Bongwan Kim, and Juntae Kim. 2021. Univnet: A neural vocoder with multi-resolution spectrogram discriminators for high-fidelity waveform generation. arXiv preprint arXiv:2106.07889. find
  13. Audio, Speech and Language Processing, 33:458– 471. find
  14. Wei-Tsung Lu, Ju-Chiang Wang, Qiuqiang Kong, and Yun-Ning Hung. 2024. Music source separation with band-split rope transformer. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 481–485. IEEE. find
  15. Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pages 498–502. find
  16. Andreas Jansson, Eric Humphrey, Nicola Montecchio, Rachel Bittner, Aparna Kumar, and Tillman Weyde. 2017. Singing voice separation with deep u-net convolutional networks. find
  17. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
  18. Hideki Kawahara, Jo Estill, and Osamu Fujimura. 2001. Aperiodicity extraction and control using mixed mode excitation and group delay manipulation for a high quality speech analysis, modification and synthesis system straight. In MAVEBA, pages 59–64. Firenze. find
  19. KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. 2020. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pages 484–492. find
  20. Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pages 5530–5540. PMLR. find
  21. Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello. 2018. Crepe: A convolutional representation for pitch estimation. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 161–165. IEEE. find
  22. Minseok Kim, Jun Hyung Lee, and Soonyoung Jung. 2023. Sound demixing challenge 2023 music demixing track technical report: Tfc-tdf-unet v3. arXiv preprint arXiv:2306.09382. — [arXiv:2306.09382](https://arxiv.org/abs/2306.09382) find
  23. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022– 17033. find
  24. Songting Liu. 2024. Zero-shot voice conversion with diffusion transformers. arXiv preprint arXiv:2411.09943. — [arXiv:2411.09943](https://arxiv.org/abs/2411.09943) find
  25. Songxiang Liu, Yuewen Cao, Dan Su, and Helen Meng. 2021. Diffsvc: A diffusion probabilistic model for singing voice conversion. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 741–748. IEEE. find
  26. Xubo Liu, Qiuqiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark D. Plumbley, and Wenwu Wang. 2025. Separate anything you describe. IEEE Transactions on Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. 2017. The musdb18… find
  27. Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. 2019. Musdb18-hq-an uncompressed version of musdb18. find
  28. Lekshmi Chandrika Reghunath, Rajeev Rajan, Christian Napoli, and Cristian Randieri. 2025. Cross-attentive cnns for joint specral and pitch feature learning in predominant instrument recognition from polyphonic music. Technologies, 14(1):3. find
  29. Simon Rouard, Francisco Massa, and Alexandre Défossez. 2023. Hybrid transformers for music source separation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  30. RVC-Project. 2023. Retrieval-based voice conversion webui. https://github.com/RVC-Project/ Retrieval-based-Voice-Conversion-WebUI. GitHub repository, accessed 2025-01-06. ↗
  31. Justin Salamon and Emilia Gómez. 2012. Melody extraction from polyphonic music signals using pitch contour characteristics. IEEE transactions on audio, speech, and language processing, 20(6):1759–1770. find
  32. Bowen Shi, Andros Tjandra, John Hoffman, Helin Wang, Yi-Chiao Wu, Luya Gao, Julius Richter, Matt Le, Apoorv Vyas, Sanyuan Chen, Christoph Feichtenhofer, Piotr Dollár, Wei-Ning Hsu, and Ann Lee. 2025. Sam audio: Segment anything in audio. Preprint, arXiv:2512.18099. — [arXiv:2512.18099](https://arxiv.org/abs/2512.18099) find
  33. Roman Solovyev, Alexander Stempkovskiy, and Tatiana Habruseva. 2023. Benchmarks and leaderboards for sound demixing tasks. arXiv preprint arXiv:2305.07489. — [arXiv:2305.07489](https://arxiv.org/abs/2305.07489) find
  34. Fabian-Robert Stöter, Stefan Uhlich, Antoine Liutkus, and Yuki Mitsufuji. 2019. Open-unmix-a reference implementation for music source separation. Journal of Open Source Software, 4(41):1667. find
  35. Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. 2017. Synthesizing obama: learning lip sync from audio. ACM Transactions on Graphics (ToG), 36(4):1–13. svc-develop-team. 2023. so-vits-svc: Softvc + vits singing voice conversion framework. https://github.com/svc-develop-team/s… ↗
  36. Weinan Tong, Jiaxu Zhu, Jun Chen, Shiyin Kang, Tao Jiang, Yang Li, Zhiyong Wu, and Helen Meng. 2024. Scnet: sparse compression network for music source separation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1276–1280. IEEE. find
  37. Hui Wang, Siqi Zheng, Yafeng Chen, Luyao Cheng, and Qian Chen. 2023a. Cam++: A fast and efficient network for speaker verification using context-aware masking. arXiv preprint arXiv:2303.00332. — [arXiv:2303.00332](https://arxiv.org/abs/2303.00332) find
  38. Ju-Chiang Wang, Wei-Tsung Lu, and Minz Won. 2023b. Mel-band roformer for music source separation. arXiv preprint arXiv:2310.01809. — [arXiv:2310.01809](https://arxiv.org/abs/2310.01809) find
  39. Haojie Wei, Xueke Cao, Tangpeng Dan, and Yueguo Chen. 2023. Rmvpe: A robust model for vocal pitch estimation in polyphonic music. arXiv preprint arXiv:2306.15412. — [arXiv:2306.15412](https://arxiv.org/abs/2306.15412) find
  40. Ya-Qi Yu, Siqi Zheng, Hongbin Suo, Yun Lei, and Wu-Jun Li. 2021. Cam: Context-aware masking for robust speaker verification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6703–6707. IEEE. find
  41. Chongbin Zhang, Jiaxiang Zheng, and Moxi Cao. 2025. A music source separation method integrating time– frequency decoupling and mamba-based state space modeling. Scientific Reports, 15(1):36280. find
  42. Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. 2023. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition… find
  43. Junjie Zheng, Gongyu Chen, Chaofan Ding, and Zihao Chen. 2025. R2-svc: Towards real-world robust and expressive zero-shot singing voice conversion. arXiv preprint arXiv:2510.20677. — [arXiv:2510.20677](https://arxiv.org/abs/2510.20677) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2026-vocalrep-structure-aware-vocal-representations/