Elamur Research⌕
☰
home / papers / 2025 / F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

Paper · 2025 · Yushen Chen, Zhikang Niu, Ziyang Ma et al.

Summary

F5-TTS is a fully non-autoregressive zero-shot text-to-speech system. It drops the usual alignment machinery: no duration model, no text encoder, and no phoneme alignment. Like E2 TTS, it turns the transcript into a character sequence, pads it with filler tokens to the length of the mel spectrogram, and trains a Diffusion Transformer (DiT) with conditional flow matching on a text-guided speech-infilling task. The authors found that E2 TTS converges slowly and aligns text to speech unreliably. They attribute this to semantic and acoustic features being tangled together. To fix it, F5-TTS first refines the padded text with ConvNeXt V2 blocks and only then joins it with the speech input. They also add Sway Sampling, an inference-time rule that concentrates flow steps early in the trajectory. It can be applied to existing flow-matching models without retraining. The model was trained on about 100K hours of public multilingual speech. It produces natural zero-shot speech, handles code-switching and speed control, and trains and runs faster than comparable diffusion systems. This matters because a simple, fully released pipeline makes strong TTS reproducible for the community.

Key points

  • Inference reaches a real-time factor (RTF) of 0.15, which the authors describe as a large improvement over state-of-the-art diffusion-based TTS models.
  • Input text is represented as raw characters for English and full pinyin for Chinese. It is padded with filler tokens ⟨F⟩ to match the number of mel frames.
  • In ablations, adding a ConvNeXt branch for audio as well (F5-TTS+Conv2Audio) cost 1.61 WER in exchange for only +0.01 SIM. Long skip connections did not transfer to DiT, and ConvNeXt text refinement did not transfer to E2 TTS's flat U-Net Transformer.
  • Sway Sampling with a negative coefficient s improves results on Seed-TTS test-zh. A "leak and override" test (t′ = 0.1) indicates that early flow steps determine how faithfully the output follows the prompt.
  • All code and checkpoints are released. Stated limitations are long mel sequences and no fine-grained control of paralinguistic features such as emotion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 worksSignal Processing · 213 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 233 works

Cite

Cite as Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu and Xie Chen. 2025. F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.313

@inproceedings{chen2025f5,
  title = {{F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching}},
  author = {Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu and Xie Chen},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.313},
  url = {https://doi.org/10.18653/v1/2025.acl-long.313}
}

References

72 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, et al. 2024. Seed-TTS: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430.
  2. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670.
  3. Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. 2024. VALL-E 2: Neural codec language models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.05370. — [arXiv:2406.05370](https://arxiv.org/abs/2406.05370) find
  4. Zhengyang Chen, Sanyuan Chen, Yu Wu, Yao Qian, Chengyi Wang, Shujie Liu, Yanmin Qian, and Michael Zeng. 2022. Large-scale self-supervised speech representation learning for automatic speaker verification. In Proc. ICASSP, pages 6147–6151. IEEE. find
  5. Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438.
  6. Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780– 8794. find
  7. Chenpeng Du, Yiwei Guo, Hankun Wang, Yifan Yang, Zhikang Niu, Shuai Wang, Hui Zhang, Xie Chen, and Kai Yu. 2024a. VALL-T: Decoder-only generative transducer for robust and decoding-controllable text-to-speech. arXiv preprint arXiv:2401.14321. — [arXiv:2401.14321](https://arxiv.org/abs/2401.14321) find
  8. Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, et al. 2024b. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407. find
  9. Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, et al. 2024. E2 TTS: Embarrassingly easy fully non-autoregressive zero-shot TTS. arXiv preprint arXiv:2406.18009. — [arXiv:2406.18009](https://arxiv.org/abs/2406.18009) find
  10. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, et al. 2024. Scaling rectified flow transformers for high-resolution image synthesis. In Proc. ICML. find
  11. Zhengcong Fei, Mingyuan Fan, Changqian Yu, and Junshi Huang. 2024. Flux that plays music. arXiv preprint arXiv:2409.00587. — [arXiv:2409.00587](https://arxiv.org/abs/2409.00587) find
  12. Yuan Gao, Nobuyuki Morioka, Yu Zhang, and Nanxin Chen. 2023a. E3 TTS: Easy end-to-end diffusion-based text to speech. In Proc. ASRU, pages 1–8. IEEE. find
  13. Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, et al. 2023b. FunASR: A fundamental end-to-end speech recognition toolkit. arXiv preprint arXiv:2305.11013. — [arXiv:2305.11013](https://arxiv.org/abs/2305.11013) find
  14. Wei Kang, Xiaoyu Yang, Zengwei Yao, Fangjun Kuang, Yifan Yang, Liyong Guo, Long Lin, and Daniel Povey. 2024. Libriheavy: a 50,000 hours asr corpus with punctuation casing and context. In Proc. ICASSP, pages 10991–10995. IEEE. find
  15. Hao-Han Guo, Kun Liu, Fei-Yu Shen, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. 2024a. FireRedTTS: A foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint arXiv:2409.03283. — [arXiv:2409.03283](https://arxiv.org/abs/2409.03283) find
  16. Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. 2024. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24174–24184. find
  17. Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, et al. 2021. Didispeech: A large scale Mandarin speech corpus. In Proc. ICASSP, pages 6968–6972. IEEE. find
  18. Yiwei Guo, Chenpeng Du, Ziyang Ma, Xie Chen, and Kai Yu. 2024b. Voiceflow: Efficient text-to-speech with rectified flow matching. In Proc. ICASSP, pages 11121–11125. IEEE. find
  19. Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, and Furu Wei. 2024. VALL-E R: Robust and efficient zero-shot text-to-speech synthesis via monotonic alignment. arXiv preprint arXiv:2406.07855. — [arXiv:2406.07855](https://arxiv.org/abs/2406.07855) find
  20. Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, et al. 2024. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. arXiv preprint arXiv:2407.05361. — [arXiv:2407.05361](https://arxiv.org/abs/2407.05361) find
  21. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840– 6851. find
  22. Jonathan Ho and Tim Salimans. 2022. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598.
  23. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460.
  24. Keith Ito and Linda Johnson. 2017. The LJ speech dataset. find
  25. Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. 2024. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100. — [arXiv:2403.03100](https://arxiv.org/abs/2403.03100) find
  26. Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, et al. 2020. Libri-light: A benchmark for ASR with limited or no supervision. In Proc. ICASSP, pages 7669–7673. IEEE. find
  27. Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. 2020. Glow-TTS: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33:8067–8077. find
  28. Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pages 5530–5540. PMLR. find
  29. Diederik Kingma and Ruiqi Gao. 2024. Understanding diffusion objectives as the ELBO with simple data augmentation. Advances in Neural Information Processing Systems, 36. find
  30. Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, et al. 2024. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36. find
  31. Keon Lee, Dong Won Kim, Jaehyeon Kim, and Jaewoong Cho. 2024. DiTTo-TTS: Efficient and scalable zero-shot text-to-speech with diffusion transformer. arXiv preprint arXiv:2406.11427. — [arXiv:2406.11427](https://arxiv.org/abs/2406.11427) find
  32. Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. 2022. Bigvgan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206.04658. — [arXiv:2206.04658](https://arxiv.org/abs/2206.04658) find
  33. Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. 2019. Neural speech synthesis with transformer network. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 6706–6713. find
  34. Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. 2024. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838. — [arXiv:2406.11838](https://arxiv.org/abs/2406.11838) find
  35. Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2022. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747.
  36. Zhijun Liu, Shuai Wang, Sho Inoue, Qibing Bai, and Haizhou Li. 2024a. Autoregressive diffusion transformer for text-to-speech synthesis. arXiv preprint arXiv:2406.05551. — [arXiv:2406.05551](https://arxiv.org/abs/2406.05551) find
  37. Zhijun Liu, Shuai Wang, Pengcheng Zhu, Mengxiao Bi, and Haizhou Li. 2024b. E1 TTS: Simple and fast non-autoregressive TTS. arXiv preprint arXiv:2409.09351. — [arXiv:2409.09351](https://arxiv.org/abs/2409.09351) find
  38. Ofir Press, Noah A Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409. find
  39. Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. 2022. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986. find
  40. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  41. I Loshchilov. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. find
  42. Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558. find
  43. Linhan Ma, Dake Guo, Kun Song, Yuepeng Jiang, Shuai Wang, Liumeng Xue, Weiming Xu, Huan Zhao, Binbin Zhang, and Lei Xie. 2024. Wenet-Speech4TTS: A 12,800-hour Mandarin TTS corpus for large speech generation model benchmark. arXiv preprint arXiv:2406.05763. — [arXiv:2406.05763](https://arxiv.org/abs/2406.05763) find
  44. Shivam Mehta, Ruibo Tu, Jonas Beskow, Éva Székely, and Gustav Eje Henter. 2024. Matcha-TTS: A fast TTS architecture with conditional flow matching. In Proc. ICASSP, pages 11341–11345. IEEE. find
  45. Aleksandr Meister, Matvei Novikov, Nikolay Karpov, Evelina Bakhturina, Vitaly Lavrukhin, and Boris Ginsburg. 2023. LibriSpeech-PC: Benchmark for evaluation of punctuation and capitalization capabilities of end-to-end ASR models. In Proc. ASRU, pages 1–7. IEEE. find
  46. Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, et al. 2024. Autoregressive speech synthesis without vector quantization. arXiv preprint arXiv:2407.08551. find
  47. Zhikang Niu, Sanyuan Chen, Long Zhou, Ziyang Ma, Xie Chen, and Shujie Liu. 2024. NDVQ: Robust neural audio codec with normal distribution-based vector quantization. arXiv preprint arXiv:2409.12717. — [arXiv:2409.12717](https://arxiv.org/abs/2409.12717) find
  48. Takuma Okamoto, Yamato Ohtani, Tomoki Toda, and Hisashi Kawai. 2024. Convnext-TTS and Convnext-VC: Convnext-based fast end-to-end sequence-to-sequence text-to-speech and voice conversion. In Proc. ICASSP, pages 12456–12460. IEEE. find
  49. William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205.
  50. Puyuan Peng, Po-Yao Huang, Daniel Li, Abdelrahman Mohamed, and David Harwath. 2024. Voicecraft: Zero-shot speech editing and text-to-speech in the wild. arXiv preprint arXiv:2403.16973. find
  51. Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. 2021. Grad-tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pages 8599–8608. PMLR. find
  52. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Proc. MICCAI, pages 234– 241. Springer.
  53. Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152. find
  54. Tim Salimans and Jonathan Ho. 2022. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512. — [arXiv:2202.00512](https://arxiv.org/abs/2202.00512) find
  55. Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, et al. 2018. Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions. In Proc. ICASSP, pages 4779–4783. IEEE. find
  56. Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, et al. 2023. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09116. — [arXiv:2304.09116](https://arxiv.org/abs/2304.09116) find
  57. Hubert Siuzdak. 2023. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306.00814. — [arXiv:2306.00814](https://arxiv.org/abs/2306.00814) find
  58. Yakun Song, Zhuo Chen, Xiaofei Wang, Ziyang Ma, and Xie Chen. 2024. ELLA-V: Stable neural codec language modeling with alignment-guided sequence reordering. arXiv preprint arXiv:2401.07333. — [arXiv:2401.07333](https://arxiv.org/abs/2401.07333) find
  59. Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2020. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456.
  60. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.
  61. Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, et al. 2024. Naturalspeech: End-to-end text-to-speech synthesis with human-level quality. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  62. Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, et al. 2023a. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111.
  63. Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. 2023b. VioLA: Unified codec language models for speech recognition, synthesis, and translation. arXiv preprint arXiv:2305.16107. — [arXiv:2305.16107](https://arxiv.org/abs/2305.16107) find
  64. Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Shunsi Zhang, and Zhizheng Wu. 2024. MaskGCT: Zero-shot text-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.00750. — [arXiv:2409.00750](https://arxiv.org/abs/2409.00750) find
  65. Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. 2023. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16133–16142. find
  66. Yi-Chiao Wu, Israel D Gebru, Dejan Marković, and Alexander Richard. 2023. Audiodec: An open-source streaming high-fidelity neural audio codec. In Proc. ICASSP, pages 1–5. IEEE. find
  67. Detai Xin, Xu Tan, Kai Shen, Zeqian Ju, et al. 2024. Rall-e: Robust codec language modeling with chain-of-thought prompting for text-to-speech synthesis. arXiv preprint arXiv:2404.03204. — [arXiv:2404.03204](https://arxiv.org/abs/2404.03204) find
  68. Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. 2023. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765. find
  69. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2021. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507.
  70. Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882.
  71. Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2023a. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308.16692.
  72. Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, et al. 2023b. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv preprint arXiv:2303.03926. — [arXiv:2303.03926](https://arxiv.org/abs/2303.03926) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2025-f5-tts-fairytaler-that-fakes/