Elamur Research⌕
☰
home / papers / 2026 / SegTune: Structured and Fine-Grained Control for Song Generation

SegTune: Structured and Fine-Grained Control for Song Generation

Paper · 2026 · Yuejiao Wang, Zihao Ji, Pengfei Cai et al.

Summary

Most open-source song generators, whether autoregressive or diffusion-based, take only a single global text prompt. Such a prompt cannot describe how instrumentation, emotion or energy shift between verses and choruses. Some diffusion systems also depend on lyric timings written by hand or produced zero-shot by an LLM. SegTune addresses both problems. It is a non-autoregressive framework built on a flow-matching Diffusion Transformer that takes three inputs: a global style prompt, separate prompts for individual segments, and lyrics aligned to time. Each segment prompt is encoded and broadcast only over its own time window. The global prompt covers the whole song. A fine-tuned LLM predicts sentence-level lyric timestamps in LRC format, so manual alignment is no longer needed. The authors also built a pipeline that collects songs and annotates them with aligned lyrics and multi-level descriptions. They introduced new metrics for checking segment alignment and singer attributes. Experiments focus on Mandarin pop, where SegTune beats existing baselines in both musicality and controllability. This makes song generation more usable for structured, interactive creative work.

Key points

  • Architecture: LLaMA-style DiT backbone trained with conditional flow-matching. A 1D VAE compresses 44 kHz audio into latents at 21.5 Hz. Global, segment and lyric embeddings are concatenated channel-wise before entering the DiT.
  • Encoders: Qwen3-Embedding serves as the global encoder. For the segment encoder, Qwen3-Embedding gave better instruction following than Muq-MuLan, with similar musicality. Mixing global and segment embeddings linearly (weights 0.2/0.8) hurt results compared with concatenation.
  • Duration predictor: A fine-tuned Qwen3-4B-Base reached an MAE of 0.99 s on per-line start times. This was clearly better than a zero-shot GPT-4o predictor in the style of JAM. The test set was 15 held-out Mandarin pop songs.
  • New metrics: The paper proposes a segment-level MuLan score and singer attribute scoring (e.g., gender, age) to measure fine-grained instruction following.
  • Limitations:
    • Quality drops when the user's segment structure is ambiguous.
    • The model cannot control changes within a segment, such as crescendos or vocal ornamentation.
    • Experiments cover Mandarin only, although the method is described as language-agnostic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓University of Science and Technology of ChinaHefei · China108 works · 112,698 citations🏢KuaishouBeijing · China15 works · 95 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksSignal Processing · 452 works

Cite

Cite as Yuejiao Wang, Zihao Ji, Pengfei Cai, Xu Li, Haorui Zheng, Zewen Song, Zhongliang Liu, Chen Zhang and Pengfei Wan. 2026. SegTune: Structured and Fine-Grained Control for Song Generation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.586

@inproceedings{wang2026segtune,
  title = {{SegTune: Structured and Fine-Grained Control for Song Generation}},
  author = {Yuejiao Wang, Zihao Ji, Pengfei Cai, Xu Li, Haorui Zheng, Zewen Song, Zhongliang Liu, Chen Zhang and Pengfei Wan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.586},
  url = {https://doi.org/10.18653/v1/2026.acl-long.586}
}

References

41 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. 2023. Musiclm: Generating music from text. Preprint, arXiv:2301.11325. — [arXiv:2301.11325](https://arxiv.org/abs/2301.11325) find
  2. Yuanhao Ban, Ruochen Wang, Tianyi Zhou, Minhao Cheng, Boqing Gong, and Cho-Jui Hsieh. 2024. Understanding the impact of negative prompts: When and how do they take effect? In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LXXXIX,… find
  3. Huakang Chen, Yuepeng Jiang, Guobin Ma, Chunbo Hao, Shuai Wang, Jixun Yao, Ziqian Ning, Meng Meng, Jian Luan, and Lei Xie. 2025. Diffrhythm+: Controllable and flexible full-length song generation with preference optimization. Preprint, arXiv:2507.12890. — [arXiv:2507.12890](https://arxiv.org/abs/2507.12890) find
  4. Yuxuan Jiang, Zehua Chen, Zeqian Ju, Chang Li, Weibei Dou, and Jun Zhu. 2025. Freeaudio: Training-free timing planning for controllable long-form text-to-audio generation. arXiv preprint arXiv:2507.08557. — [arXiv:2507.08557](https://arxiv.org/abs/2507.08557) find
  5. Taejun Kim and Juhan Nam. 2023. All-in-one metrical and functional structure analysis with neighborhood attentions on demixed audio. In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). find
  6. Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2024. Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP… find
  7. Max W. Y. Lam, Qiao Tian, Tang Li, Zongyu Yin, Siyuan Feng, Ming Tu, Yuliang Ji, Rui Xia, Mingbo Ma, Xuchen Song, Jitong Chen, Yuping Wang, and Yuxuan Wang. 2023. Efficient neural music generation. In Thirty-seventh Conference on Neural Information Processing Systems. find
  8. Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. 2023. Simple and controllable music generation. In Thirty-seventh Conference on Neural Information Processing Systems. find
  9. Max W. Y. Lam, Yijin Xing, Weiya You, Jingcheng Wu, Zongyu Yin, Fuqiang Jiang, Hangyu Liu, Feng Liu, Xingda Li, Wei-Tsung Lu, Hanyu Chen, Tong Feng, Tianwei Zhao, Chien-Hung Liu, Xuchen Song, Yang Li, and Yahui Zhou. 2025. Analyzable chain-of-musical-thought prompting for high-fidelity music generat… — [arXiv:2503.19611](https://arxiv.org/abs/2503.19611) find
  10. Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. 2020. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341. find
  11. Zach Evans, Julian D. Parker, CJ Carr, Zachary Zukowski, Josiah Taylor, and Jordi Pons. 2024a. Long-form music generation with latent diffusion. In Proceedings of the 25th International Society for Music Information Retrieval Conference, ISMIR 2024, San Francisco, California, USA and Online, Novembe… find
  12. Zach Evans, Julian D. Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. 2024b. Stable audio open. Preprint, arXiv:2407.14358. — [arXiv:2407.14358](https://arxiv.org/abs/2407.14358) find
  13. Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. 2025. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. Preprint, arXiv:2507.08128. — [arXiv:2507.08128](https://arxiv.org/abs/2507.08128) find
  14. Junmin Gong, Sean Zhao, Sen Wang, Shengyuan Xu, and Joe Guo. 2025. Ace-step: A step towards music generation foundation model. Preprint, arXiv:2506.00045. find
  15. Shun Lei, Yaoxun Xu, Zhiwei Lin, Huaicheng Zhang, Wei Tan, Hangting Chen, Jianwei Yu, Yixuan Zhang, Chenyu Yang, Haina Zhu, Shuai Wang, Zhiyong Wu, and Dong Yu. 2025. Levo: High-quality song generation with multi-preference alignment. arXiv preprint arXiv:2506.07520. — [arXiv:2506.07520](https://arxiv.org/abs/2506.07520) find
  16. Shun Lei, Yixuan Zhou, Boshi Tang, Max W. Y. Lam, Feng liu, Hangyu Liu, Jingcheng Wu, Shiyin Kang, Zhiyong Wu, and Helen M. Meng. 2024. Songcreator: Lyrics-based universal song generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  17. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. 2023. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. Open-Review.net.
  18. Renhang Liu, Chia-Yu Hung, Navonil Majumder, Taylor Gautreaux, Amir Ali Bagherzadeh, Chuan Li, Dorien Herremans, and Soujanya Poria. 2025a. Jam: A tiny flow-based song generator with fine-grained controllability and aesthetic alignment. Preprint, arXiv:2507.20880. — [arXiv:2507.20880](https://arxiv.org/abs/2507.20880) find
  19. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc. find
  20. Zihan Liu, Shuangrui Ding, Zhixiong Zhang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025b. Songgen: A single stage auto-regressive transformer for text-to-song generation. In Forty-second International Conference on Machine Learning. find
  21. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  22. William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 4172–4182. IEEE.
  23. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. Preprint, arXiv:2212.04356.
  24. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems.
  25. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10674–10685. IEEE. find
  26. Simon Rouard, Francisco Massa, and Alexandre Défossez. 2023. Hybrid transformers for music source separation. In ICASSP 23. find
  27. Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, and 1 others. 2025. Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. arXiv preprint arXiv:2502.05139. — [arXiv:2502.05139](https://arxiv.org/abs/2502.05139) find
  28. Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Guy Wolf, and Yoshua Bengio. 2024. Improving and generalizing flow-based generative models with mini-batch optimal transport. find
  29. Trans. Mach. Learn. Res., 2024. find
  30. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint…
  31. Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. 2024. Maskgct: Zero-shot text-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.00750. — [arXiv:2409.00750](https://arxiv.org/abs/2409.00750) find
  32. Shih-Lun Wu, Chris Donahue, Shinji Watanabe, and Nicholas J. Bryan. 2024. Music controlnet: Multiple time-varying controls for music generation. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 32:2692–2703. find
  33. Jin Xu, Zhifang Guo, Hangrui Hu, and Yunfei Chu et al. 2025a. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765.
  34. Kai-Tuo Xu, Feng-Long Xie, Xu Tang, and Yao Hu. 2025b. Fireredasr: Open-source industrial-grade mandarin speech recognition models from encoder-decoder to llm integration. arXiv preprint arXiv:2501.14350. — [arXiv:2501.14350](https://arxiv.org/abs/2501.14350) find
  35. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, and Bowen Yu et al. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388.
  36. Chenyu Yang, Hangting Chen, Shuai Wang, Haina Zhu, and Haizhou Li. 2025b. Tvc-musicgen: Time-varying structure control for background music generation via self-supervised training. In Annual Conference of the International Speech Communication Association, pages 1238–1242. find
  37. Jixun Yao, Guobin Ma, Huixin Xue, Huakang Chen, Chunbo Hao, Yuepeng Jiang, Haohe Liu, Ruibin Yuan, Jin Xu, Wei Xue, and 1 others. 2025. Songeval: A benchmark dataset for song aesthetics evaluation. arXiv preprint arXiv:2505.10793. — [arXiv:2505.10793](https://arxiv.org/abs/2505.10793) find
  38. Ruibin Yuan, Hanfeng Lin, Shuyue Guo, Ge Zhang, Jiahao Pan, Yongyi Zang, and Haohe Liu et al. 2025. Yue: Scaling open foundation models for long-form music generation. Preprint, arXiv:2503.08638. — [arXiv:2503.08638](https://arxiv.org/abs/2503.08638) find
  39. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. Preprint, arXiv:2506.05176.
  40. Haina Zhu, Yizhi Zhou, Hangting Chen, Jianwei Yu, Ziyang Ma, Rongzhi Gu, Yi Luo, Wei Tan, and Xie Chen. 2025. Muq: Self-supervised music representation learning with mel residual vector quantization. Preprint, arXiv:2501.01108. — [arXiv:2501.01108](https://arxiv.org/abs/2501.01108) find
  41. Ning Ziqian, Chen Huakang, Jiang Yuepeng, Hao Chunbo, Ma Guobin, Wang Shuai, Yao Jixun, and Xie Lei. 2025. DiffRhythm: Blazingly fast and embarrassingly simple end-to-end full-length song generation with latent diffusion. arXiv preprint arXiv:2503.01183. — [arXiv:2503.01183](https://arxiv.org/abs/2503.01183) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-segtune-structured-fine-grained-control/