Elamur Research⌕
☰
home / papers / 2026 / ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment

ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment

Paper · 2026 · Jun-Hak Yun, Seung-Bin Kim and Seong-Whan Lee

Summary

ImmersiveTTS targets environment-aware speech synthesis: producing intelligible speech together with a fitting acoustic backdrop (e.g., birdsong) from two text inputs, a transcript and a scene description. Earlier systems such as VoiceLDM and VoiceDiT often yield speech and background that do not blend convincingly. The authors adapt the multimodal diffusion transformer (MM-DiT) from SD3 to audio. One stream carries transcript-aligned speech latents and a parallel stream carries environment conditioning from text encoders (CLAP, FLAN-T5). Joint attention links the streams, and later single-stream blocks merge them. Training uses a flow matching objective in the latent space of the pretrained AudioLDM2 VAE. They also add a domain-specific variant of REPA that aligns intermediate hidden states with two self-supervised teachers: WavLM for speech and ATST-Frame for environmental audio. Objective metrics and listening tests show gains in naturalness, intelligibility and fidelity over prior methods, with far fewer sampling steps. This matters for immersive media, games and data augmentation, where speech must sound as if recorded in a specific place.

Key points

  • Architecture: MM-DiT with DoubleStream blocks (speech and environment streams, joint attention) followed by SingleStream blocks, trained with rectified flow / flow matching on AudioLDM2 VAE latents (4× time-frequency downsampling, 8 channels).
  • Domain-specific REPA uses WavLM as the speech teacher and ATST-Frame as the environment teacher. Of six teacher setups tested on the AudioCaps test set, this pair scored best on WER, FAD and CLAP, beating setups with the unified encoder USAD.
  • Teacher trade-off: with a single teacher, WavLM mainly lowers WER, while ATST-Frame raises the CLAP score but hurts intelligibility. The dual-teacher setup removes this trade-off.
  • Efficiency: with only 9 sampling steps (NFEs), ImmersiveTTS outperforms VoiceLDM and VoiceDiT, both run at 200 NFEs, on WER, FAD and CLAP.
  • Limitations: training relies mostly on synthetic speech-environment mixtures, robustness across SNR levels is not well studied, and there is no explicit control of prosody, style or emotion. Samples and code are available at jjunak-yun.github.io/ImmersiveTTS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Korea UniversitySeoul · South Korea29 works · 9,490 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 811 worksSpeech & spoken language · 519 works

Cite

Cite as Jun-Hak Yun, Seung-Bin Kim and Seong-Whan Lee. 2026. ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1774

@inproceedings{yun2026immersivetts,
  title = {{ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment}},
  author = {Jun-Hak Yun, Seung-Bin Kim and Seong-Whan Lee},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1774},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1774}
}

References

71 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, and 1 others. 2022. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. In Proc. Annu. Meet. Assoc. Comput. Linguist. (ACL), pages 5723–5738.
  2. Marcely Zanon Boito, Vivek Iyer, Nikolaos Lagos, Laurent Besacier, and Ioan Calapodescu. 2024. mhubert-147: A compact multilingual hubert model. In Ann. Conf. Int. Speech Commun. Assoc. (INTERSPEECH). find
  3. Heng-Jui Chang, Saurabhchand Bhati, James Glass, and Alexander H Liu. 2025. Usad: Universal speech and audio representation via distillation. In IEEE Autom. Speech Recognit. Underst. Workshop (ASRU). find
  4. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, and 1 others. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, pages 1505–1518. find
  5. Sanyuan Chen, Chengyi Wang, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, and 1 others. 2025a. Neural codec language models are zero-shot text to speech synthesizers. IEEE/ACM Trans. Audio, Speech, Lang. Process. (TASLP).
  6. Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. 2025b. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. In Proc. Annu. Meet. Assoc. Comput. Linguist. (ACL).
  7. Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. 2025. Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR). find
  8. Ha-Yeong Choi, Sang-Hoon Lee, and Seong-Whan Lee. 2024. Dddm-vc: Decoupled denoising diffusion models with disentangled representation and prior mixup for verified robust voice conversion. In Proc. AAAI Conf. Artificial Intelligence (AAAI), volume 38, pages 17862–17870. find
  9. Jeongsoo Choi, Zhikang Niu, Ji-Hoon Kim, Chunhui Wang, Joon Son Chung, and Xie Chen. 2025. Accelerating diffusion-based text-to-speech model training with dual modality alignment. In Ann. Conf. Int. Speech Commun. Assoc. (INTERSPEECH). find
  10. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and 1 others. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. find
  11. Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In IEEE Autom. Speech Recognit. Underst. Workshop (ASRU), pages 244–250. find
  12. Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, and 1 others. 2025. Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training. arXiv preprint arXiv:2505.17589. — [arXiv:2505.17589](https://arxiv.org/abs/2505.17589) find
  13. Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, and 1 others. 2024. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117.
  14. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, and 1 others. 2024. Scaling rectified flow transformers for high-resolution image synthesis, 2024. In Proc. Int. Conf. Mach. Learn. (ICML). find
  15. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), volume 33, pages 6840–6851. find
  16. Jonathan Ho and Tim Salimans. 2022. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598.
  17. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio, Speech, Lang. Process. (TASLP), 29:3451–3460.
  18. Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. 2023. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In Proc. Int. Conf. Mach. Learn. (ICML). find
  19. Chia-Yu Hung, Navonil Majumder, Zhifeng Kong, Ambuj Mehrish, Amir Ali Bagherzadeh, Chuan Li, Rafael Valle, Bryan Catanzaro, and Soujanya Poria. 2024. Tangoflux: Super fast and faithful text to audio generation with flow matching and clap-ranked preference optimization. arXiv preprint arXiv:2412.2103… find
  20. Jaemin Jung, Junseok Ahn, Chaeyoung Jung, Tan Dat Nguyen, Youngjoon Jang, and Joon Son Chung. 2025. Voicedit: Dual-condition diffusion transformer for environment-aware speech synthesis. In IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP). find
  21. Zhengcong Fei, Mingyuan Fan, Changqian Yu, and Junshi Huang. 2024. Flux that plays music. arXiv preprint arXiv:2409.00587. — [arXiv:2409.00587](https://arxiv.org/abs/2409.00587) find
  22. Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. 2020. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), volume 33, pages 8067–8077. find
  23. Neta Glazer, Aviv Navon, Yael Segal, Aviv Shamsian, Hilit Segev, Asaf Buchnick, Menachem Pirchi, Gil Hetz, and Joseph Keshet. 2025. Umbratts: Adapting text-to-speech to environmental contexts with flow matching. In arXiv preprint arXiv:2506.09874. — [arXiv:2506.09874](https://arxiv.org/abs/2506.09874) find
  24. Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021a. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In Proc. Int. Conf. Mach. Learn. (ICML), pages 5530–5540. find
  25. Junmin Gong, Sean Zhao, Sen Wang, Shengyuan Xu, and Joe Guo. 2025. Ace-step: A step towards music generation foundation model. arXiv preprint arXiv:2506.00045. find
  26. Ji-Hoon Kim, Sang-Hoon Lee, Ji-Hyun Lee, and Seong-Whan Lee. 2021b. Fre-gan: Adversarial frequency-consistent audio synthesis. find
  27. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proc. Int. Conf. Mach. Learn. (ICML), pages 369–376. find
  28. Seung-Bin Kim, Jun-Hyeok Cha, Hyung-Seok Oh, Hee-jin Choi, and Seong-Whan Lee. 2025. Fillerspeech: Towards human-like text-to-speech synthesis with filler insertion and filler style control. In Proc. Conf. Empir. Methods Nat. Lang. Process. (EMNLP), pages 34096–34113. find
  29. Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, and 1 others. 2017. Cnn architectures for large-scale audio classification. In IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP). find
  30. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), pages 17022– 17033.
  31. Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. 2023. Audiogen: Textually guided audio generation. In Proc. Int. Conf. Learn. Represent. (ICLR). find
  32. Sang-Hoon Lee, Ha-Yeong Choi, Seung-Bin Kim, and Seong-Whan Lee. 2025. Hierspeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis. IEEE Transactions on Neural Networks and Learning Systems. find
  33. Sang-Hoon Lee, Seung-Bin Kim, Ji-Hyun Lee, Eunwoo Song, Min-Jae Hwang, and Seong-Whan Lee. 2022. Hierspeech: Bridging the gap between text and speech by hierarchical variational inference using self-supervised representations for speech synthesis. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS),… find
  34. Yeonghyeon Lee, Inmo Yeon, Juhan Nam, and Joon Son Chung. 2024. Voiceldm: Text-to-speech with environmental context. In IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP). find
  35. Xian Li, Nian Shao, and Xiaofei Li. 2024. Self-supervised audio teacher-student transformer for both clip-level and frame-level tasks. IEEE/ACM Trans. Audio, Speech, Lang. Process. (TASLP), pages 1336– 1351. find
  36. Xiquan Li, Junxi Liu, Yuzhe Liang, Zhikang Niu, Wenxi Chen, and Xie Chen. 2025. Meanaudio: Fast and faithful text-to-audio generation with mean flows. arXiv preprint arXiv:2508.06098. find
  37. Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, and 1 others. 2023. Mert: Acoustic music understanding model with large-scale self-supervised training. arXiv preprint arXiv:2306.00107. — [arXiv:2306.00107](https://arxiv.org/abs/2306.00107) find
  38. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. 2023. Flow matching for generative modeling. In Proc. Int. Conf. Learn. Represent. (ICLR).
  39. Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. 2023a. AudioLDM: Text-to-audio generation with latent diffusion models. In Proc. Int. Conf. Mach. Learn. (ICML).
  40. Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. 2024. AudioLDM 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Trans. Audio, Speech, Lang. Process. (TASLP), pages 2871–2883. find
  41. Huadai Liu, Rongjie Huang, Xuan Lin, Wenqiang Xu, Maozong Zheng, Hong Chen, Jinzheng He, and Zhou Zhao. 2023b. Vit-tts: visual text-to-speech with scalable diffusion transformer. In Proc. Conf. Empir. find
  42. Methods Nat. Lang. Process. (EMNLP), pages 15957– 15969. find
  43. Huadai Liu, Kaicheng Luo, Jialei Wang, Wen Wang, Qian Chen, Zhou Zhao, and Wei Xue. 2025a. Thinksound: Chain-of-thought reasoning in multimodal large language models for audio generation and editing. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS). find
  44. Rui Liu, Shuwei He, Yifan Hu, and Haizhou Li. 2025b. Multi-modal and multi-scale spatial environment understanding for immersive visual text-to-speech. In Proc. AAAI Conf. Artificial Intelligence (AAAI), volume 39, pages 24632–24640. find
  45. Xingchao Liu, Chengyue Gong, and Qiang Liu. 2023c. Flow straight and fast: Learning to generate and transfer data with rectified flow. In Proc. Int. Conf. Learn. Represent. (ICLR).
  46. Ye-Xin Lu, Hui-Peng Du, Zheng-Yan Sheng, Yang Ai, and Zhen-Hua Ling. 2025a. Incremental disentanglement for environment-aware zero-shot text-to-speech synthesis. In IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP). find
  47. Ye-Xin Lu, Yu Gu, Kun Wei, Hui-Peng Du, Yang Ai, and Zhen-Hua Ling. 2025b. Daien-tts: Disentangled audio infilling for environment-aware text-to-speech synthesis. arXiv preprint arXiv:2509.14684. — [arXiv:2509.14684](https://arxiv.org/abs/2509.14684) find
  48. Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. 2024. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Trans. Audio, Speech, Lang. Process. (TASLP). find
  49. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, and Alaaeldin El-Nouby. 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193.
  50. William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 4195–4205.
  51. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In Proc. Int. Conf. Mach. Learn. (ICML).
  52. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proc. Int. Conf. Mach. Learn. (ICML), pages 28492–28518.
  53. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, pages 1–67.
  54. Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP). find
  55. Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2021. Fastspeech 2: Fast and high-quality end-to-end text to speech. In Proc. Int. Conf. Learn. Represent. (ICLR). find
  56. Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and 1 others. 2025. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. In Proc. Int. Conf. Learn. Represent. (ICLR). find
  57. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (CVPR), pages 10684–10695.
  58. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241.
  59. Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152.
  60. Sizhe Shan, Qiulin Li, Yutao Cui, Miles Yang, Yuehai Wang, Qun Yang, Jin Zhou, and Zhao Zhong. 2025. Hunyuanvideo-foley: Multimodal diffusion with representation alignment for high-fidelity foley audio generation. arXiv preprint arXiv:2508.16930. — [arXiv:2508.16930](https://arxiv.org/abs/2508.16930) find
  61. Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, and 1 others. 2018. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP). find
  62. Yang Song and 1 others. 2021. Score-based generative modeling through stochastic differential equations. In Proc. Int. Conf. Learn. Represent. (ICLR).
  63. Daxin Tan, Guangyan Zhang, and Tan Lee. 2022. Environment aware text-to-speech synthesis. In Ann. Conf. Int. Speech Commun. Assoc. (INTERSPEECH). find
  64. Jinlong Xue, Yayue Deng, Yingming Gao, and Ya Li. 2024. Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation. IEEE/ACM Trans. Audio, Speech, Lang. Process. (TASLP). find
  65. Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, and 1 others. 2024. Uniaudio: An audio foundation model toward universal audio generation. In Proc. Int. Conf. Mach. Learn. (ICML). find
  66. Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. 2025. Representation alignment for generation: Training diffusion transformers is easier than you think. In Proc. Int. Conf. Learn. Represent. (ICLR). find
  67. Jun-Hak Yun, Seung-Bin Kim, and Seong-Whan Lee. 2025. Flowhigh: Towards efficient and high-quality audio super-resolution with single-step flow matching. In IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP). find
  68. Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882.
  69. Xueyao Zhang, Junan Zhang, Yuancheng Wang, Chaoren Wang, Yuanzhe Chen, Dongya Jia, Zhuo Chen, and Zhizheng Wu. 2025. Vevo2: Bridging controllable speech and singing voice generation via unified prosody learning. arXiv preprint arXiv:2508.16332. — [arXiv:2508.16332](https://arxiv.org/abs/2508.16332) find
  70. Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, and 1 others. 2023. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312.15821. — [arXiv:2312.15821](https://arxiv.org/abs/2312.15821) find
  71. Jun Wang, Xijuan Zeng, Chunyu Qiang, Ruilong Chen, Shiyao Wang, Le Wang, Wangjing Zhou, Pengfei Cai, Jiahui Zhao, Nan Li, and 1 others. 2025. Kling-foley: Multimodal diffusion transformer for high-quality video-to-audio generation. arXiv preprint arXiv:2506.19774. — [arXiv:2506.19774](https://arxiv.org/abs/2506.19774) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yun-2026-immersivetts-environment-aware-text-speech/