Elamur Research⌕
☰
home / papers / 2025 / Kimi-Audio Technical Report

Kimi-Audio Technical Report

Paper · 2025 · KimiTeam, Ding Ding, Zeqian Ju et al.

Summary

Earlier audio language models tend to target one task family, skip serious audio pre-training, or ship without code and weights. Kimi-Audio (Kimi Team, Moonshot AI) is an open-source foundation model that covers speech recognition, audio understanding, generation and spoken dialogue in one system. Its input merges discrete semantic tokens from the GLM-4-Voice tokenizer (a quantized Whisper encoder at 12.5Hz) with continuous Whisper features downsampled to the same rate. An LLM initialized from a pre-trained text model shares its lower layers, then branches into a text head and an audio head. A chunk-wise streaming flow-matching detokenizer with look-ahead renders speech at low latency. Continual pre-training uses over 13 million hours of speech, sound and music, combining text-only, audio-only, audio-to-text and interleaved tasks. Fine-tuning relies mostly on openly available data. The authors report state-of-the-art results on ASR, audio understanding, audio QA and speech conversation. They release checkpoints, code and an evaluation toolkit that makes model comparisons fairer.

Key points

  • Pre-training corpus: more than 13M hours of audio spanning speech, sound and music. It was cleaned with a pipeline covering speech enhancement, diarization, transcription and filtering.
  • Hybrid 12.5Hz input: discrete semantic tokens (GLM-4-Voice, single codebook) are added to Whisper features. An adaptor brings these features from 50Hz down to 12.5Hz.
  • Dual-head audio LLM: shared transformer layers and the text head reuse the pre-trained text LLM's weights, while the audio head starts from random initialization.
  • Detokenizer: it follows the MoonCast design, with flow matching from 12.5Hz tokens to 50Hz mel-spectrograms followed by a BigVGAN vocoder. It streams in chunks of about one second, using causal chunk masking and look-ahead to avoid boundary artifacts.
  • Evaluation and release: the model is benchmarked against Qwen2-Audio, Baichuan-Audio, Step-Audio and Qwen2.5-Omni. Code, weights and the evaluation toolkit are available at github.com/MoonshotAI/Kimi-Audio.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSignal Processing · 213 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,344 worksSpeech & spoken language · 233 works

Cite

Cite as KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, Jun Chen, Yanru Chen, Yulun Du, Weiran He, Zhenxing Hu, Guokun Lai, Qingcheng Li, Yangyang Liu, Weidong Sun, Jianzhou Wang, Yuzhi Wang, Yuefeng Wu, Yuxin Wu, Dongchao Yang, Hao Yang, Ying Yang, Zhilin Yang, Aoxiong Yin, Ruibin Yuan, Yutong Zhang and Zaida Zhou. 2025. Kimi-Audio Technical Report. arXiv:2504.18425.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2504.18425 · arXiv 2504.18425

@misc{kimiteam2025kimi,
  title = {{Kimi-Audio Technical Report}},
  author = {KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, Jun Chen, Yanru Chen, Yulun Du, Weiran He, Zhenxing Hu, Guokun Lai, Qingcheng Li, Yangyang Liu, Weidong Sun, Jianzhou Wang, Yuzhi Wang, Yuefeng Wu, Yuxin Wu, Dongchao Yang, Hao Yang, Ying Yang, Zhilin Yang, Aoxiong Yin, Ruibin Yuan, Yutong Zhang and Zaida Zhou},
  year = {2025},
  doi = {10.48550/arxiv.2504.18425},
  eprint = {2504.18425},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.18425}
}

References

89 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rosana Ardila et al. “Common voice: A massively-multilingual speech corpus”. In: arXiv preprint arXiv:1912.06670 (2019).
  2. Beijing Academy of Artificial Intelligence (BAAI). “Infinity Instruct”. In: arXiv preprint arXiv:2406.XXXX (2024). find
  3. Zalán Borsos et al. “Audiolm: a language modeling approach to audio generation”. In: IEEE/ACM transactions on audio, speech, and language processing 31 (2023), pp. 2523–2533.
  4. Hui Bu et al. “Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline”. In: 2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech I/O systems and assessment (O-COCOSDA). IEEE. 2017, pp. 1–5. find
  5. Guoguo Chen et al. “Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio”. In: arXiv preprint arXiv:2106.06909 (2021). — [arXiv:2106.06909](https://arxiv.org/abs/2106.06909) find
  6. Honglie Chen et al. “Vggsound: A large-scale audio-visual dataset”. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2020, pp. 721–725. find
  7. Qian Chen et al. “Minmo: A multimodal large language model for seamless voice interaction”. In: arXiv preprint arXiv:2501.06282 (2025). — [arXiv:2501.06282](https://arxiv.org/abs/2501.06282) find
  8. Yiming Chen et al. “VoiceBench: Benchmarking LLM-Based Voice Assistants”. In: arXiv preprint arXiv:2410.17196 (2024).
  9. Zesen Cheng et al. “Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms”. In: arXiv preprint arXiv:2406.07476 (2024). find
  10. Yunfei Chu et al. “Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models”. In: arXiv preprint arXiv:2311.07919 (2023).
  11. Yunfei Chu et al. “Qwen2-audio technical report”. In: arXiv preprint arXiv:2407.10759 (2024).
  12. Alexis Conneau et al. “Fleurs: Few-shot learning evaluation of universal representations of speech”. In: 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE. 2023, pp. 798–805.
  13. DeepSeek-AI. DeepSeek-V3 Technical Report. 2024. arXiv: 2412.19437 [cs.CL]. URL : https://arxiv.org/abs/2412.19437. ↗
  14. Alexandre Défossez et al. “Moshi: a speech-text foundation model for real-time dialogue”. In: arXiv preprint arXiv:2410.00037 (2024).
  15. Chandeepa Dissanayake et al. OpenBezoar: Small, Cost-Effective and Open Models Trained on Mixes of Instruction Data. 2024. arXiv: 2404.12195 [cs.CL]. — [arXiv:2404.12195](https://arxiv.org/abs/2404.12195) find
  16. Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. “Clotho: An audio captioning dataset”. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2020, pp. 736–740. find
  17. Jiayu Du et al. “Aishell-2: Transforming mandarin asr research into industrial scale”. In: arXiv preprint arXiv:1808.10583 (2018). find
  18. Qingkai Fang et al. “Llama-omni: Seamless speech interaction with large language models”. In: arXiv preprint arXiv:2409.06666 (2024). find
  19. Eduardo Fonseca et al. “Fsd50k: an open dataset of human-labeled sound events”. In: IEEE/ACM Transactions on Audio, Speech, and Language Processing 30 (2021), pp. 829–852. find
  20. Zhifu Gao et al. “Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition”. In: arXiv preprint arXiv:2206.08317 (2022). — [arXiv:2206.08317](https://arxiv.org/abs/2206.08317) find
  21. Xuelong Geng et al. “OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia”. In: arXiv preprint arXiv:2501.13306 (2025). — [arXiv:2501.13306](https://arxiv.org/abs/2501.13306) find
  22. Sreyan Ghosh et al. “Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities”. In: arXiv preprint arXiv:2406.11768 (2024). find
  23. Yuan Gong, Jin Yu, and James Glass. “Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition”. In: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2022, pp. 151–155. DOI : 10.1109/ICASSP43922.2022.9746828. — [doi:10.1109/icassp43922.2022.9746828](https://doi.org/10.1109/icassp43922.2022.9746828) find
  24. Aaron Grattafiori et al. “The llama 3 herd of models”. In: arXiv preprint arXiv:2407.21783 (2024). find
  25. Haorui He et al. “Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation”. In: arXiv preprint arXiv:2501.15907 (2025). — [arXiv:2501.15907](https://arxiv.org/abs/2501.15907) find
  26. Haorui He et al. “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation”. In: 2024 IEEE Spoken Language Technology Workshop (SLT). IEEE. 2024, pp. 885–890. find
  27. T. Heittola et al. TAU Urban Acoustic Scenes 2022 Mobile, Development dataset. Zenodo. Mar. 2022. DOI : 10.5281/zenodo.6337421. — [doi:10.5281/zenodo.6337421](https://doi.org/10.5281/zenodo.6337421) find
  28. Ailin Huang et al. “Step-audio: Unified understanding and generation in intelligent speech interaction”. In: arXiv preprint arXiv:2502.11946 (2025). find
  29. Aaron Hurst et al. “Gpt-4o system card”. In: arXiv preprint arXiv:2410.21276 (2024).
  30. Philip Jackson and Sana ul haq. Surrey Audio-Visual Expressed Emotion (SAVEE) database. Apr. 2011. find
  31. Il-Young Jeong and Jeongsoo Park. “Cochlscene: Acquisition of acoustic scene data using crowdsourcing”. In: 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE. 2022, pp. 17–21. find
  32. Zeqian Ju et al. “MoonCast: High-Quality Zero-Shot Podcast Generation”. In: arXiv preprint arXiv:2503.14345 (2025). — [arXiv:2503.14345](https://arxiv.org/abs/2503.14345) find
  33. Wei Kang et al. “Libriheavy: A 50,000 hours ASR corpus with punctuation casing and context”. In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2024, pp. 10991–10995. find
  34. Chris Dongjoo Kim et al. “Audiocaps: Generating captions for audios in the wild”. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019, pp. 119–132.
  35. Zhifeng Kong et al. “Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities”. In: arXiv preprint arXiv:2402.01831 (2024). — [arXiv:2402.01831](https://arxiv.org/abs/2402.01831) find
  36. Nathan Lambert et al. “T\\" ulu 3: Pushing frontiers in open language model post-training”. In: arXiv preprint arXiv:2411.15124 (2024).
  37. Matthew Le et al. “Voicebox: Text-guided multilingual universal speech generation at scale”. In: Advances in neural information processing systems 36 (2023), pp. 14005–14034. find
  38. Sang-gil Lee et al. “Bigvgan: A universal neural vocoder with large-scale training”. In: arXiv preprint arXiv:2206.04658 (2022). — [arXiv:2206.04658](https://arxiv.org/abs/2206.04658) find
  39. Guangyao Li et al. “Learning to answer questions in dynamic audio-visual scenarios”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 19108–19118. find
  40. Jia Li et al. “Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions”. In: Hugging Face repository 13 (2024), p. 9. find
  41. Tianpeng Li et al. “Baichuan-audio: A unified framework for end-to-end speech interaction”. In: arXiv preprint arXiv:2502.17239 (2025). find
  42. Wing Lian et al. OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces. https://https://huggingface.co/datasets/Open-Orca/OpenOrca. 2023. find
  43. Samuel Lipping et al. “Clotho-aqa: A crowdsourced dataset for audio question answering”. In: 2022 30th European Signal Processing Conference (EUSIPCO). IEEE. 2022, pp. 1140–1144. find
  44. Jingyuan Liu et al. Muon is Scalable for LLM Training. 2025. arXiv: 2502.16982 [cs.LG]. URL : https://arxiv.org/abs/2502.16982. ↗
  45. Rui-Bo Liu et al. “Convincing Audio Generation Based on LLM and Speech Tokenization”. In: 2024 IEEE 14th International Symposium on Chinese Spoken Language Processing (ISCSLP). IEEE. 2024, pp. 591–595. find
  46. Songting Liu. “Zero-shot Voice Conversion with Diffusion Transformers”. In: arXiv preprint arXiv:2411.09943 (2024). — [arXiv:2411.09943](https://arxiv.org/abs/2411.09943) find
  47. Steven R Livingstone and Frank A Russo. “The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English”. In: PloS one 13.5 (2018), e0196391. find
  48. Ilya Loshchilov and Frank Hutter. “Decoupled weight decay regularization”. In: arXiv preprint arXiv:1711.05101 (2017). find
  49. Yi Luo and Jianwei Yu. “Music Source Separation With Band-Split RNN”. In: IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 (2023), pp. 1893–1901. DOI : 10.1109/TASLP.2023.3271145. — [doi:10.1109/taslp.2023.3271145](https://doi.org/10.1109/taslp.2023.3271145) find
  50. Linhan Ma et al. “Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model benchmark”. In: arXiv preprint arXiv:2406.05763 (2024). — [arXiv:2406.05763](https://arxiv.org/abs/2406.05763) find
  51. Irene Martín-Morató and Annamaria Mesaros. “What is the ground truth? reliability of multi-annotator data for audio tagging”. In: 2021 29th European Signal Processing Conference (EUSIPCO). IEEE. 2021, pp. 76–80. find
  52. Xinhao Mei et al. “Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research”. In: IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024). find
  53. Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. “TUT Database for Acoustic Scene Classification and Sound Event Detection”. In: 24th European Signal Processing Conference 2016 (EUSIPCO 2016). Budapest, Hungary, 2016. find
  54. Vassil Panayotov et al. “Librispeech: an asr corpus based on public domain audio books”. In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE. 2015, pp. 5206–5210. find
  55. Karol J. Piczak. “ESC: Dataset for Environmental Sound Classification”. In: Proceedings of the 23rd Annual ACM Conference on Multimedia. Brisbane, Australia: ACM Press, Oct. 13, 2015, pp. 1015–1018. ISBN : 978-1-4503-3459-4. DOI : 10.1145/2733373.2806390. URL : http://dl.acm.org/citation.cfm?doid=27… — [doi:10.1145/2733373.2806390](https://doi.org/10.1145/2733373.2806390) ↗
  56. Soujanya Poria et al. “Meld: A multimodal multi-party dataset for emotion recognition in conversations”. In: arXiv preprint arXiv:1810.02508 (2018).
  57. Vineel Pratap et al. “MLS: A Large-Scale Multilingual Dataset for Speech Research”. In: ArXiv abs/2012.03411 (2020).
  58. Alec Radford et al. “Robust speech recognition via large-scale weak supervision”. In: International conference on machine learning. PMLR. 2023, pp. 28492–28518.
  59. Muhammad Mamunur Rashid, Guiqing Li, and Chengrui Du. “Nonspeech7k dataset: Classification and analysis of human non-speech sound”. In: IET Signal Processing 17.6 (2023), e12233. find
  60. S Sakshi et al. “Mmau: A massive multi-task audio understanding and reasoning benchmark”. In: arXiv preprint arXiv:2410.19168 (2024). find
  61. Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. “A dataset and taxonomy for urban sound research”. In: Proceedings of the 22nd ACM international conference on Multimedia. 2014, pp. 1041–1044. find
  62. Yao Shi et al. “Aishell-3: A multi-speaker mandarin tts corpus and the baselines”. In: arXiv preprint arXiv:2010.11567 (2020). — [arXiv:2010.11567](https://arxiv.org/abs/2010.11567) find
  63. Changli Tang et al. “Salmonn: Towards generic hearing abilities for large language models”. In: arXiv preprint arXiv:2310.13289 (2023).
  64. Zhiyuan Tang et al. “Kespeech: An open source speech dataset of mandarin and its eight subdialects”. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021. find
  65. Kimi Team et al. Kimi k1.5: Scaling Reinforcement Learning with LLMs. 2025. arXiv: 2501.12599 [cs.AI]. URL : https://arxiv.org/abs/2501.12599. ↗
  66. Teknium. OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants. 2023. URL : https://huggingface.co/datasets/teknium/OpenHermes-2.5. ↗
  67. Migel Tissera. Synthia-70b-v1.2: Synthetic intelligent agent. Hugging Face. 2023. URL : https://huggingface.co/migtissera/Synthia-13B. ↗
  68. Samarth Tripathi, Sarthak Tripathi, and Homayoon Beigi. “Multi-modal emotion recognition on iemocap dataset using deep learning”. In: arXiv preprint arXiv:1804.05788 (2018). — [arXiv:1804.05788](https://arxiv.org/abs/1804.05788) find
  69. Changhan Wang et al. “VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation”. In: arXiv preprint arXiv:2101.00390 (2021).
  70. Chengyi Wang et al. “Neural codec language models are zero-shot text to speech synthesizers”. In: arXiv preprint arXiv:2301.02111 (2023).
  71. Xiong Wang et al. “Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm”. In: arXiv preprint arXiv:2411.00774 (2024). find
  72. Zhifei Xie and Changqiao Wu. “Mini-omni: Language models can hear, talk while thinking in streaming”. In: arXiv preprint arXiv:2408.16725 (2024). find
  73. Jin Xu et al. “Qwen2. 5-omni technical report”. In: arXiv preprint arXiv:2503.20215 (2025).
  74. Jin Xu et al. Qwen2.5-Omni Technical Report. 2025. arXiv: 2503.20215 [cs.CL]. URL : https://arxiv.org/abs/2503.20215. ↗
  75. Zhangchen Xu et al. “Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing”. In: arXiv preprint arXiv:2406.08464 (2024).
  76. An Yang et al. “Qwen2.5 Technical Report”. In: arXiv preprint arXiv:2412.15115 (2024). find
  77. Dongchao Yang et al. “Uniaudio: An audio foundation model toward universal audio generation”. In: arXiv preprint arXiv:2310.00704 (2023). — [arXiv:2310.00704](https://arxiv.org/abs/2310.00704) find
  78. Pinci Yang et al. “Avqa: A dataset for audio-visual question answering on videos”. In: Proceedings of the 30th ACM international conference on multimedia. 2022, pp. 3480–3491. find
  79. Zehui Yang et al. “Open source magicdata-ramc: A rich annotated mandarin conversational (ramc) speech dataset”. In: arXiv preprint arXiv:2203.16844 (2022). — [arXiv:2203.16844](https://arxiv.org/abs/2203.16844) find
  80. Zhen Ye et al. “Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis”. In: arXiv preprint arXiv:2502.04128 (2025). — [arXiv:2502.04128](https://arxiv.org/abs/2502.04128) find
  81. Jianwei Yu et al. “Autoprep: An automatic preprocessing framework for in-the-wild speech data”. In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2024, pp. 1136–1140. find
  82. Jianwei Yu et al. “High fidelity speech enhancement with band-split rnn”. In: arXiv preprint arXiv:2212.00406 (2022). — [arXiv:2212.00406](https://arxiv.org/abs/2212.00406) find
  83. Heiga Zen et al. “Libritts: A corpus derived from librispeech for text-to-speech”. In: arXiv preprint arXiv:1904.02882 (2019).
  84. Aohan Zeng et al. “Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot”. In: arXiv preprint arXiv:2412.02612 (2024).
  85. Binbin Zhang et al. “Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition”. In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2022, pp. 6182–6186. find
  86. Qinglin Zhang et al. “Omniflatten: An end-to-end gpt model for seamless voice conversation”. In: arXiv preprint arXiv:2410.17799 (2024). find
  87. Yu Zhang et al. “Google usm: Scaling automatic speech recognition beyond 100 languages”. In: arXiv preprint arXiv:2303.01037 (2023). — [arXiv:2303.01037](https://arxiv.org/abs/2303.01037) find
  88. Hang Zhao et al. “MINT: Boosting Audio-Language Model via Multi-Target Pre-Training and Instruction Tuning”. In: Interspeech 2024. 2024, pp. 52–56. DOI : 10.21437/Interspeech.2024-1863. — [doi:10.21437/interspeech.2024-1863](https://doi.org/10.21437/interspeech.2024-1863) find
  89. Kun Zhou et al. “Emotional voice conversion: Theory, databases and ESD”. In: Speech Communication 137 (2022), pp. 1–18. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kimiteam-2025-kimi-audio-technical-report/