Elamur Research⌕
☰
home / papers / 2025 / Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction

Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction

Paper · 2025 · Ailin Huang, Boyong Wu, Bruce Wang et al.

Summary

Open-source spoken dialogue systems often keep speech understanding and speech generation in separate components. They also rely on expensive hand-collected voice data and offer weak control over emotion, dialect, speaking style and tool use. The StepFun team presents Step-Audio as a production-ready open framework to address these gaps. Its core is a 130B-parameter speech-text model built on Step-1 and pretrained on 3.3T multimodal tokens. A dual-codebook tokenizer turns audio into discrete units. For real-time dialogue, the system pairs audio-in/text-out (AQTA) generation with a controllable TTS stage. The large model also acts as a data engine: it synthesizes training speech, which is then distilled into the compact Step-Audio-TTS-3B. Post-training combines SFT with RLHF, and the system supports instruction-based control of emotions, dialects, singing and RAP, plus asynchronous tool calling and role-play. Step-Audio-Chat leads open-source baselines on spoken QA benchmarks and in human evaluation on the new StepEval-Audio-360 benchmark. This makes it a strong open foundation for voice assistants.

Key points

  • Tokenizer design: speech is encoded by two tokenizers whose streams are interleaved in a 2:3 ratio, in the style of SpiritLM.
    • Linguistic tokens come from a Paraformer encoder at 16.7 Hz with a 1024-entry codebook.
    • Semantic tokens come from a CosyVoice tokenizer at 25 Hz with a 4096-entry codebook.
  • Open releases: the 130B Step-Audio-Chat model and the distilled Step-Audio-TTS-3B are published at github.com/stepfun-ai/Step-Audio.
  • Spoken QA results: on Llama Question, Web Questions, TriviaQA, ComplexBench and HSK-6, the average gain over the best open-source results is about 9.3 points.
    • Llama Question: 81.0, compared with 78.9 for MinMo.
    • HSK-6: 86.0, compared with 74.0 for GLM4-Voice.
  • StepEval-Audio-360 human evaluation: the benchmark spans 9 dimensions, with comparisons against GLM-4-Voice and Qwen2-Audio.
    • Response quality improved by 19.2%, relevance by 23.7% and factual accuracy by 43.2%.
    • Instruction following (IF) rose by 29.8% and MOS by 27.1%.
  • Asynchronous tool calls: tool calls run in a text thread separate from audio synthesis, so external lookups don't stall the speech output.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 519 works

Cite

Cite as Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, Peng Liu, Ruihang Miao, Wang You, Xi Chen, Xuerui Yang, Yechang Huang, Yuxiang Zhang, Zheng Gong, Zixin Zhang, Hongyu Zhou, Jianjian Sun, Brian Li, Chengting Feng, Changyi Wan, Hanpeng Hu, Jianchang Wu, Jiangjie Zhen, Ranchen Ming, Song Yuan, Xuelin Zhang, Yu Zhou, Bingxin Li, Buyun Ma, Hongyuan Wang, Kang An, Wei Ji, Wen Li, Xuan Wen, Xiangwen Kong, Yuankai Ma, Yuanwei Liang, Yun Mou, Bahtiyar Ahmidi, Bin Wang, Bo Li, Changxin Miao, Chen Xu, Chenrun Wang, Dapeng Shi, Deshan Sun, Dingyuan Hu, Dula Sai, Enle Liu, Guanzhe Huang, Gulin Yan, Heng Wang, Haonan Jia, Haoyang Zhang, Jiahao Gong, Junjing Guo, Jiashuai Liu, Jiahong Liu, Jie Feng, Jie Wu, Jiaoren Wu, Jie Yang, Jinguo Wang, Jingyang Zhang, Junzhe Lin, Kaixiang Li, Lei Xia, Li Zhou, Liang Zhao, Longlong Gu, Mei Chen, Menglin Wu, Ming Li, Mingxiao Li, Mingliang Li, Mingyao Liang, Na Wang, Nie Hao, Qiling Wu, Qinyuan Tan, Ran Sun, Shuai Shuai, Shaoliang Pang, Shiliang Yang, Shuli Gao, Shanshan Yuan, Siqi Liu, Shihong Deng, Shilei Jiang, Sitong Liu, Tiancheng Cao, Tianyu Wang, Wenjin Deng, Wuxun Xie, Weipeng Ming, Wenqing He, Wen Sun, Xin Han, Xin Huang, Xiaomin Deng, Xiaojia Liu, Xin Wu, Xu Zhao, Yanan Wei, Yanbo Yu, Yang Cao, Yangguang Li, Yangzhen Ma, Yanming Xu, Yaoyu Wang, Yaqiang Shi, Yilei Wang, Yizhuang Zhou, Yinmin Zhong, Yang Zhang, Yaoben Wei, Yu Luo, Yuanwei Lu, Yuhe Yin, Yuchu Luo, Yuanhao Ding, Yuting Yan, Yaqi Dai, Yuxiang Yang, Zhe Xie, Zheng Ge, Zheng Sun, Zhewei Huang, Zhichao Chang, Zhisheng Guan, Zidong Yang, Zili Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang and Yibo Zhu. 2025. Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction. arXiv:2502.11946.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.11946 · arXiv 2502.11946

@misc{huang2025step,
  title = {{Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction}},
  author = {Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, Peng Liu, Ruihang Miao, Wang You, Xi Chen, Xuerui Yang, Yechang Huang, Yuxiang Zhang, Zheng Gong, Zixin Zhang, Hongyu Zhou, Jianjian Sun, Brian Li, Chengting Feng, Changyi Wan, Hanpeng Hu, Jianchang Wu, Jiangjie Zhen, Ranchen Ming, Song Yuan, Xuelin Zhang, Yu Zhou, Bingxin Li, Buyun Ma, Hongyuan Wang, Kang An, Wei Ji, Wen Li, Xuan Wen, Xiangwen Kong, Yuankai Ma, Yuanwei Liang, Yun Mou, Bahtiyar Ahmidi, Bin Wang, Bo Li, Changxin Miao, Chen Xu, Chenrun Wang, Dapeng Shi, Deshan Sun, Dingyuan Hu, Dula Sai, Enle Liu, Guanzhe Huang, Gulin Yan, Heng Wang, Haonan Jia, Haoyang Zhang, Jiahao Gong, Junjing Guo, Jiashuai Liu, Jiahong Liu, Jie Feng, Jie Wu, Jiaoren Wu, Jie Yang, Jinguo Wang, Jingyang Zhang, Junzhe Lin, Kaixiang Li, Lei Xia, Li Zhou, Liang Zhao, Longlong Gu, Mei Chen, Menglin Wu, Ming Li, Mingxiao Li, Mingliang Li, Mingyao Liang, Na Wang, Nie Hao, Qiling Wu, Qinyuan Tan, Ran Sun, Shuai Shuai, Shaoliang Pang, Shiliang Yang, Shuli Gao, Shanshan Yuan, Siqi Liu, Shihong Deng, Shilei Jiang, Sitong Liu, Tiancheng Cao, Tianyu Wang, Wenjin Deng, Wuxun Xie, Weipeng Ming, Wenqing He, Wen Sun, Xin Han, Xin Huang, Xiaomin Deng, Xiaojia Liu, Xin Wu, Xu Zhao, Yanan Wei, Yanbo Yu, Yang Cao, Yangguang Li, Yangzhen Ma, Yanming Xu, Yaoyu Wang, Yaqiang Shi, Yilei Wang, Yizhuang Zhou, Yinmin Zhong, Yang Zhang, Yaoben Wei, Yu Luo, Yuanwei Lu, Yuhe Yin, Yuchu Luo, Yuanhao Ding, Yuting Yan, Yaqi Dai, Yuxiang Yang, Zhe Xie, Zheng Ge, Zheng Sun, Zhewei Huang, Zhichao Chang, Zhisheng Guan, Zidong Yang, Zili Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang and Yibo Zhu},
  year = {2025},
  doi = {10.48550/arxiv.2502.11946},
  eprint = {2502.11946},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.11946}
}

References

30 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anastassiou, P., Chen, J., Chen, J., Chen, Y., Chen, Z., Chen, Z., . . . others (2024). Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430 .
  2. Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 , 324. Retrieved from https://api.semanticscholar.org/CorpusID:125209808 bytedance. (2024). doubaovoice. https://team .doubao .com/zh/special/ realtime voice. (Access… ↗
  3. Chen, Q., Chen, Y., Chen, Y., Chen, M., Chen, Y., Deng, C., . . . others (2025). Minmo: A multimodal large language model for seamless voice interaction. arXiv preprint arXiv:2501.06282 . — [arXiv:2501.06282](https://arxiv.org/abs/2501.06282) find
  4. Chen, Y., Zheng, S., Wang, H., Cheng, L., Chen, Q., & Qi, J. (2023). An enhanced res2net with local and global feature fusion for speaker verification. arXiv preprint arXiv:2305.12838 . — [arXiv:2305.12838](https://arxiv.org/abs/2305.12838) find
  5. Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., . . . others (2024a). Qwen2-audio technical report. arXiv preprint arXiv:2407.10759 .
  6. Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., . . . others (2024b). Qwen2-audio technical report. arXiv preprint arXiv:2407.10759 .
  7. Das, N., Dingliwal, S., Ronanki, S., Paturi, R., Huang, Z., Mathur, P., . . . others (2024). Speechverse: A large-scale generalizable audio language model. arXiv preprint arXiv:2405.08295 .
  8. Défossez, A., Mazaré, L., Orsini, M., Royer, A., Pérez, P., Jégou, H., . . . Zeghidour, N. (2024). Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037 .
  9. Du, Z., Chen, Q., Zhang, S., Hu, K., Lu, H., Yang, Y., . . . others (2024). Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407 .
  10. Du, Z., Wang, Y., Chen, Q., Shi, X., Lv, X., Zhao, T., . . . others (2024). Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117 .
  11. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., . . . others (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783 .
  12. Fang, Q., Guo, S., Zhou, Y., Ma, Z., Zhang, S., & Feng, Y. (2024). Llama-omni: Seamless speech interaction with large language models. arXiv preprint arXiv:2409.06666 .
  13. Gao, H., Shao, H., Wang, X., Qiu, C., Shen, Y., Cai, S., . . . others (2025). Lucy: Linguistic understanding and control yielding early stage of her. arXiv preprint arXiv:2501.16327 . — [arXiv:2501.16327](https://arxiv.org/abs/2501.16327) find
  14. Gao, Z., Zhang, S., McLoughlin, I., & Yan, Z. (2022). Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition. arXiv preprint arXiv:2206.08317 . — [arXiv:2206.08317](https://arxiv.org/abs/2206.08317) find
  15. Hu, S., Zhou, L., Liu, S., Chen, S., Meng, L., Hao, H., . . . others (2024). Wavllm: Towards robust and adaptive speech large language model. arXiv preprint arXiv:2404.00656 . find
  16. Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z., . . . others (2024). Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the aaai conference on artificial intelligence (Vol. 38, pp. 23802–23804). find
  17. Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., . . . others (2024). Gpt-4o system card. arXiv preprint arXiv:2410.21276 .
  18. Kong, Q., Cao, Y., Iqbal, T., Wang, Y., Wang, W., & Plumbley, M. D. (2020). Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28 , 2880–2894. find
  19. Ming, R., Wu, J., Huang, Z., Ju, Z., Hu, J., Peng, L., & Zhou, S. (2024). Advancing auto-regressive continuation for video frames. arXiv preprint arXiv:2412.03758 . — [arXiv:2412.03758](https://arxiv.org/abs/2412.03758) find
  20. Nguyen, T. A., Muller, B., Yu, B., Costa-Jussa, M. R., Elbayad, M., Popuri, S., . . . others (2024). Spirit-lm: Interleaved spoken and written language model. arXiv preprint arXiv:2402.05755 .
  21. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In International conference on machine learning (pp. 28492–28518).
  22. Rouard, S., Massa, F., & Défossez, A. (2023). Hybrid transformers for music source separation. In Icassp 23. find
  23. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 . find
  24. StepFun. (2024a). Step-1: A 130b large language model. https://platform.stepfun.com/docs/llm/text. (Accessed: February 2024) ↗
  25. StepFun. (2024b). Step-2. https://platform.stepfun.com/docs/llm/text. (Accessed: February 2024) ↗
  26. Wang, X., Li, Y., Fu, C., Shen, Y., Xie, L., Li, K., . . . Ma, L. (2024). Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm. arXiv preprint arXiv:2411.00774 . find
  27. Xie, Z., & Wu, C. (2024). Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408.16725 .
  28. Zeng, A., Du, Z., Liu, M., Wang, K., Jiang, S., Zhao, L., . . . Tang, J. (2024). Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02612 .
  29. Zhang, Z., Zhong, Y., Ming, R., Hu, H., Sun, J., Ge, Z., . . . Jin, X. (2024). Disttrain: Addressing model and data heterogeneity with disaggregated training for multimodal large language models. arXiv preprint arXiv:2408.04275 . find
  30. Acknowledgment We designate core contributors as those who have been involved in the development of Step-Audio throughout its entire process, while contributors are those who worked on the early versions or contributed part-time. All contributors are listed in alphabetical order by first name. • Cor… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2025-step-audio-unified-understanding/