Elamur Research⌕
☰
home / papers / 2025 / Kimi k1.5: Scaling Reinforcement Learning with LLMs

Kimi k1.5: Scaling Reinforcement Learning with LLMs

Summary

Next-token pretraining is limited by how much high-quality text exists. The Kimi Team proposes reinforcement learning as another way to keep scaling: the model creates its own training signal by exploring solutions and receiving rewards. The report describes how they built Kimi k1.5, a multimodal LLM. Training runs through pretraining, ordinary SFT, a small long-CoT warm-up SFT and then RL. The RL prompts are chosen to cover many domains, to mix easy and hard problems (difficulty is estimated from how often an SFT model solves each one) and to have answers that can be checked automatically. The RL recipe is deliberately simple. It stretches the context window to 128k tokens, uses partial rollouts that reuse earlier trajectories, and optimises with a variant of online mirror descent plus curriculum sampling and a length penalty. It uses no MCTS, value functions or process reward models. The long-CoT model performs on par with OpenAI o1 on maths, coding and vision benchmarks. "long2short" techniques then carry these gains over to short-CoT models. The authors conclude that context length is a central dimension for scaling RL.

Key points

  • Long-CoT model: 77.5 on AIME 2024, 96.2 on MATH 500, 94th percentile on Codeforces and 74.9 on MathVista, which the authors report as comparable to OpenAI o1.
  • Short-CoT model (long2short, e.g. length penalty and model merging): 60.8 on AIME, 94.6 on MATH500 and 47.3 on LiveCodeBench. It beats GPT-4o and Claude 3.5 Sonnet, by as much as +550% on some benchmarks.
  • Prompt filtering:
    • Difficulty is estimated from 10 high-temperature samples per prompt.
    • Multiple-choice, true/false and proof questions are dropped.
    • A prompt counts as "easy to hack" and is removed if a model guesses its answer without CoT within N = 8 tries.
  • Ablations:
    • Curriculum sampling (the full mixed set first, then only hard problems) beats uniform sampling.
    • The method needs fewer training samples than ReST. The authors credit this to penalising wrong answers (negative gradients), which ReST does not do.
  • Infrastructure: partial rollouts make RL with a 128k context affordable, and the training data mixes text-only and image-text problems so the model reasons over both.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cite

Cite as Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Weixin Xu, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang, Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, Zonghan Yang and Zongyu Lin. 2025. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.12599 · arXiv 2501.12599

@misc{team2025kimi,
  title = {{Kimi k1.5: Scaling Reinforcement Learning with LLMs}},
  author = {Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Weixin Xu, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang, Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, Zonghan Yang and Zongyu Lin},
  year = {2025},
  doi = {10.48550/arxiv.2501.12599},
  eprint = {2501.12599},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.12599}
}

References

64 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abbasi-Yadkori, Yasin et al. “Politex: Regret bounds for policy iteration using expert prediction”. In: International Conference on Machine Learning. PMLR. 2019, pp. 3692–3702. find
  2. Ahmadian, Arash et al. “Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms”. In: arXiv preprint arXiv:2402.14740 (2024). find
  3. Ankner, Zachary et al. Critique-out-Loud Reward Models. 2024. arXiv: 2408.11791 [cs.LG]. URL : https://arxiv.org/abs/2408.11791. ↗
  4. Berner, Christopher et al. “Dota 2 with large scale deep reinforcement learning”. In: arXiv preprint arXiv:1912.06680 (2019). find
  5. AIMO2024 AIME2024 ReST Ours ReST Ours 0.40 0.30 0.35 0.25 0.20 0.30 Accuracy Accuracy 0.15 0.25 0.10 0.20 0.05 0.15 0.00 0 Step Step k12-biology GPQA 0.24 ReST Ours ReST Ours ReST Ours 0.78 0.22 0.76 0.20 0.74 0.18 Accuracy Accuracy 0.72 0.16 0.70 0.14 0.68 0.12 0.66 Step Step Total KAOYAN 0.66 ReST… find
  6. Cassano, Federico, John Gouwar, Daniel Nguyen, Sydney Nguyen, et al. “MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation”. In: IEEE Transactions on Software Engineering 49.7 (2023), pp. 3675– 3691. DOI : 10.1109/TSE.2023.3267446. — [doi:10.1109/tse.2023.3267446](https://doi.org/10.1109/tse.2023.3267446) find
  7. Chen, Jianlv et al. “Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation”. In: arXiv preprint arXiv:2402.03216 (2024). our card
  8. Chen, Xingyu et al. “Do NOT Think That Much for 2+ 3=? On the Overthinking of o1-Like LLMs”. In: arXiv preprint arXiv:2412.21187 (2024). our card
  9. Everitt, Tom et al. Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective. 2021. arXiv: 1908.04734 [cs.AI]. URL : https://arxiv.org/abs/1908.04734. — [arXiv:1908.04734](https://arxiv.org/abs/1908.04734) ↗
  10. Gadre, Samir Yitzhak et al. “Datacomp: In search of the next generation of multimodal datasets”. In: Advances in Neural Information Processing Systems 36 (2024). find
  11. Grattafiori, Aaron et al. The Llama 3 Herd of Models. 2024. arXiv: 2407.21783 [cs.AI]. URL : https://arxiv.org/abs/2407.21783. ↗
  12. Gulcehre, Caglar et al. “Reinforced self-training (rest) for language modeling”. In: arXiv preprint arXiv:2308.08998 (2023). our card
  13. Hendrycks, Dan et al. “Measuring Massive Multitask Language Understanding”. In: ArXiv abs/2009.03300 (2020). URL : https://arxiv.org/abs/2009.03300. ↗ our card
  14. Hoffmann, Jordan et al. Training Compute-Optimal Large Language Models. 2022. arXiv: 2203.15556 [cs.CL]. URL : https://arxiv.org/abs/2203.15556. ↗ our card
  15. Huang, Yuzhen et al. “C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models”. In: ArXiv abs/2305.08322 (2023). URL : https://arxiv.org/abs/2305.08322. ↗ our card
  16. Jaech, Aaron et al. “Openai o1 system card”. In: arXiv preprint arXiv:2412.16720 (2024). our card
  17. Jain, Naman et al. “LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code”. In: ArXiv abs/2403.07974 (2024). URL : https://arxiv.org/abs/2403.07974. ↗ our card
  18. Joulin, Armand et al. “Bag of tricks for efficient text classification”. In: arXiv preprint arXiv:1607.01759 (2016). find
  19. Kaplan, Jared et al. Scaling Laws for Neural Language Models. 2020. arXiv: 2001.08361 [cs.LG]. URL : https://arxiv.org/abs/2001.08361. ↗ our card
  20. Kool, Wouter, Herke van Hoof, and Max Welling. “Buy 4 reinforce samples, get a baseline for free!” In: (2019). find
  21. Kwon, Woosuk et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention”. In: Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. 2023. our card
  22. Laurençon, Hugo et al. “Obelics: An open web-scale filtered dataset of interleaved image-text documents”. In: Advances in Neural Information Processing Systems 36 (2024). our card
  23. Li, Jeffrey et al. “Datacomp-lm: In search of the next generation of training sets for language models”. In: arXiv preprint arXiv:2406.11794 (2024). our card
  24. Li, Ming et al. “From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning”. In: arXiv preprint arXiv:2308.12032 (2023). our card
  25. Li, Raymond et al. StarCoder: may the source be with you! 2023. arXiv: 2305 . 06161 [cs.CL]. URL : https : //arxiv.org/abs/2305.06161. find
  26. Lightman, Hunter et al. “Let’s Verify Step by Step”. In: arXiv preprint arXiv:2305.20050 (2023). our card
  27. Liu, Wei et al. “What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning”. In: arXiv preprint arXiv:2312.15685 (2023). our card
  28. Lozhkov, Anton et al. StarCoder 2 and The Stack v2: The Next Generation. 2024. arXiv: 2402.19173 [cs.SE]. URL : https://arxiv.org/abs/2402.19173. ↗
  29. Lu, Pan et al. “Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts”. In: arXiv preprint arXiv:2310.02255 (2023). our card
  30. McAleese, Nat et al. LLM Critics Help Catch LLM Bugs. 2024. arXiv: 2407.00215 [cs.SE]. URL : https://arxiv.org/abs/2407.00215. — [arXiv:2407.00215](https://arxiv.org/abs/2407.00215) ↗
  31. Mei, Jincheng et al. “On principled entropy exploration in policy optimization”. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence. 2019, pp. 3130–3136. find
  32. Muennighoff, Niklas et al. Scaling Data-Constrained Language Models. 2023. arXiv: 2305.16264 [cs.CL]. URL : https://arxiv.org/abs/2305.16264. ↗
  33. Nachum, Ofir et al. “Bridging the gap between value and policy based reinforcement learning”. In: Advances in neural information processing systems 30 (2017). find
  34. OpenAI. “Learning to reason with LLMs”. In: (2024). URL : https://openai.com/index/learning-to-reason-with-llms/. ↗
  35. Ouyang, Long et al. “Training language models to follow instructions with human feedback”. In: Advances in neural information processing systems 35 (2022), pp. 27730–27744. our card
  36. Pan, Alexander, Kush Bhatia, and Jacob Steinhardt. “The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models”. In: International Conference on Learning Representations. 2022. URL : https://openreview.net/forum?id=JYtwGwIL7ye. ↗
  37. Paster, Keiran et al. “Openwebmath: An open dataset of high-quality mathematical web text”. In: arXiv preprint arXiv:2310.06786 (2023). our card
  38. Penedo, Guilherme et al. “The fineweb datasets: Decanting the web for the finest text data at scale”. In: arXiv preprint arXiv:2406.17557 (2024). our card
  39. Qin, Ruoyu et al. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. 2024. arXiv: 2407. 00079 [cs.DC]. URL : https://arxiv.org/abs/2407.00079. ↗
  40. Rafailov, Rafael et al. “Direct preference optimization: Your language model is secretly a reward model”. In: Advances in Neural Information Processing Systems 36 (2024). our card
  41. Schuhmann, Christoph et al. “Laion-5b: An open large-scale dataset for training next generation image-text models”. In: Advances in Neural Information Processing Systems 35 (2022), pp. 25278–25294. our card
  42. Shoeybi, Mohammad et al. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. 2020. arXiv: 1909.08053 [cs.CL]. URL : https://arxiv.org/abs/1909.08053. ↗
  43. Silver, David et al. “Mastering the game of go without human knowledge”. In: nature 550.7676 (2017), pp. 354–359. find
  44. Snell, Charlie et al. “Scaling llm test-time compute optimally can be more effective than scaling model parameters”. In: arXiv preprint arXiv:2408.03314 (2024). our card
  45. Su, Dan et al. “Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset”. In: arXiv preprint arXiv:2412.02595 (2024). find
  46. Su, Jianlin et al. “Roformer: Enhanced transformer with rotary position embedding”. In: Neurocomputing 568 (2024), p. 127063. our card
  47. Team, Gemini et al. Gemini: A Family of Highly Capable Multimodal Models. 2024. arXiv: 2312.11805 [cs.CL]. URL : https://arxiv.org/abs/2312.11805. ↗ our card
  48. Tomar, Manan et al. “Mirror descent policy optimization”. In: arXiv preprint arXiv:2005.09814 (2020). — [arXiv:2005.09814](https://arxiv.org/abs/2005.09814) find
  49. Vaswani, Ashish et al. “Attention is All you Need”. In: Advances in Neural Information Processing Systems. Ed. by I. Guyon et al. Vol. 30. Curran Associates, Inc., 2017. URL : https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. ↗ our card
  50. Villalobos, Pablo et al. Will we run out of data? Limits of LLM scaling based on human-generated data. 2024. arXiv: 2211.04325 [cs.LG]. URL : https://arxiv.org/abs/2211.04325. ↗
  51. Vinyals, Oriol et al. “Grandmaster level in StarCraft II using multi-agent reinforcement learning”. In: nature 575.7782 (2019), pp. 350–354. find
  52. Wang, Ke et al. “Measuring multimodal mathematical reasoning with math-vision dataset”. In: arXiv preprint arXiv:2402.14804 (2024). our card
  53. Wei, Haoran et al. “General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model”. In: arXiv preprint arXiv:2409.01704 (2024). — [arXiv:2409.01704](https://arxiv.org/abs/2409.01704) find
  54. Wei, Jason et al. “Chain-of-thought prompting elicits reasoning in large language models”. In: Advances in neural information processing systems 35 (2022), pp. 24824–24837. our card
  55. Wu, Yangzhen et al. “Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models”. In: arXiv preprint arXiv:2408.00724 (2024). our card
  56. Xu, Liang et al. “CLUE: A Chinese Language Understanding Evaluation Benchmark”. In: International Conference on Computational Linguistics. 2020. URL : https://arxiv.org/abs/2004.05986. ↗ our card
  57. Yang, Enneng et al. “Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities”. In: arXiv preprint arXiv:2408.07666 (2024). find
  58. Yao, Shunyu et al. “Tree of thoughts: Deliberate problem solving with large language models”. In: Advances in Neural Information Processing Systems 36 (2024). find
  59. Yue, Xiang, Yuansheng Ni, et al. “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 9556–9567. our card
  60. Yue, Xiang, Xingwei Qu, et al. “Mammoth: Building math generalist models through hybrid instruction tuning”. In: arXiv preprint arXiv:2309.05653 (2023). our card
  61. Zhang, Lunjun et al. “Generative verifiers: Reward modeling as next-token prediction, 2024”. In: URL https://arxiv.org/abs/2408.15240 (2024). ↗ our card
  62. Zheng, Lianmin et al. SGLang: Efficient Execution of Structured Language Model Programs. 2024. arXiv: 2312.07104 [cs.AI]. URL : https://arxiv.org/abs/2312.07104. ↗ our card
  63. Zhou, Jeffrey et al. “Instruction-Following Evaluation for Large Language Models”. In: ArXiv abs/2311.07911 (2023). URL : https://arxiv.org/abs/2311.07911. ↗ our card
  64. Zhu, Wanrong et al. “Multimodal c4: An open, billion-scale corpus of images interleaved with text”. In: Advances in Neural Information Processing Systems 36 (2024). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k1-5-scaling-reinforcement/