Elamur Research
☰
home / papers / 2025 / Kimi k1.5: Scaling Reinforcement Learning with LLMs

Kimi k1.5: Scaling Reinforcement Learning with LLMs

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-ND 4.0 (arXiv)" does not allow redistributing the text itself. Original — arxiv.org/pdf/2501.12599

Summary

Next-token pretraining is limited by how much high-quality text exists. The Kimi Team proposes reinforcement learning as another way to keep scaling: the model creates its own training signal by exploring solutions and receiving rewards. The report describes how they built Kimi k1.5, a multimodal LLM. Training runs through pretraining, ordinary SFT, a small long-CoT warm-up SFT and then RL. The RL prompts are chosen to cover many domains, to mix easy and hard problems (difficulty is estimated from how often an SFT model solves each one) and to have answers that can be checked automatically. The RL recipe is deliberately simple. It stretches the context window to 128k tokens, uses partial rollouts that reuse earlier trajectories, and optimises with a variant of online mirror descent plus curriculum sampling and a length penalty. It uses no MCTS, value functions or process reward models. The long-CoT model performs on par with OpenAI o1 on maths, coding and vision benchmarks. "long2short" techniques then carry these gains over to short-CoT models. The authors conclude that context length is a central dimension for scaling RL.

Key points

  • Long-CoT model: 77.5 on AIME 2024, 96.2 on MATH 500, 94th percentile on Codeforces and 74.9 on MathVista, which the authors report as comparable to OpenAI o1.
  • Short-CoT model (long2short, e.g. length penalty and model merging): 60.8 on AIME, 94.6 on MATH500 and 47.3 on LiveCodeBench. It beats GPT-4o and Claude 3.5 Sonnet, by as much as +550% on some benchmarks.
  • Prompt filtering:
    • Difficulty is estimated from 10 high-temperature samples per prompt.
    • Multiple-choice, true/false and proof questions are dropped.
    • A prompt counts as "easy to hack" and is removed if a model guesses its answer without CoT within N = 8 tries.
  • Ablations:
    • Curriculum sampling (the full mixed set first, then only hard problems) beats uniform sampling.
    • The method needs fewer training samples than ReST. The authors credit this to penalising wrong answers (negative gradients), which ReST does not do.
  • Infrastructure: partial rollouts make RL with a 128k context affordable, and the training data mixes text-only and image-text problems so the model reasons over both.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

401 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 386 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.12599 · arXiv 2501.12599

@misc{team2025kimi,
  title = {{Kimi k1.5: Scaling Reinforcement Learning with LLMs}},
  author = {Team, Kimi and Du, Angang and Gao, Bofei and Xing, Bowei and Jiang, Changjiu and Chen, Cheng and Li, Cheng and Xiao, Chenjun and Du, Chenzhuang and Liao, Chonghua and Tang, Chuning and Wang, Congcong and Zhang, Dehao and Yuan, Enming and Lu, Enzhe and Tang, Fengxiang and Sung, Flood and Wei, Guangda and Lai, Guokun and Guo, Haiqing and Zhu, Han and Ding, Hao and Hu, Hao and Yang, Hao and Zhang, Hao and Yao, Haotian and Zhao, Haotian and Lu, Haoyu and Li, Haoze and Yu, Haozhen and Gao, Hongcheng and Zheng, Huabin and Yuan, Huan and Chen, Jia and Guo, Jianhang and Su, Jianlin and Wang, Jianzhou and Zhao, Jie and Zhang, Jin and Liu, Jingyuan and Yan, Junjie and Wu, Junyan and Shi, Lidong and Ye, Ling and Yu, Longhui and Dong, Mengnan and Zhang, Neo and Ma, Ningchen and Pan, Qiwei and Gong, Qucheng and Liu, Shaowei and Ma, Shengling and Wei, Shupeng and Cao, Sihan and Huang, Siying and Jiang, Tao and Gao, Weihao and Xiong, Weimin and He, Weiran and Huang, Weixiao and Xu, Weixin and Wu, Wenhao and He, Wenyang and Wei, Xianghui and Jia, Xianqing and Wu, Xingzhe and Xu, Xinran and Zu, Xinxing and Zhou, Xinyu and Pan, Xuehai and Charles, Y. and Li, Yang and Hu, Yangyang and Liu, Yangyang and Chen, Yanru and Wang, Yejie and Liu, Yibo and Qin, Yidao and Liu, Yifeng and Yang, Ying and Bao, Yiping and Du, Yulun and Wu, Yuxin and Wang, Yuzhi and Zhou, Zaida and Wang, Zhaoji and Li, Zhaowei and Zhu, Zhen and Zhang, Zheng and Wang, Zhexu and Yang, Zhilin and Huang, Zhiqi and Huang, Zihao and Xu, Ziyao and Yang, Zonghan and Lin, Zongyu},
  year = {2025},
  doi = {10.48550/arxiv.2501.12599},
  eprint = {2501.12599},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.12599}
}

References

64 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abbasi-Yadkori, Yasin et al. “Politex: Regret bounds for policy iteration using expert prediction”. In: International Conference on Machine Learning. PMLR. 2019, pp. 3692–3702. find
  2. Ahmadian, Arash et al. “Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms”. In: arXiv preprint arXiv:2402.14740 (2024). find
  3. Ankner, Zachary et al. Critique-out-Loud Reward Models. 2024. arXiv: 2408.11791 [cs.LG]. URL : https://arxiv. org/abs/2408.11791. find
  4. Berner, Christopher et al. “Dota 2 with large scale deep reinforcement learning”. In: arXiv preprint arXiv:1912.06680 (2019). find
  5. AIMO2024 AIME2024 ReST Ours ReST Ours 0.40 0.30 0.35 0.25 0.20 0.30 Accuracy Accuracy 0.15 0.25 0.10 0.20 0.05 0.15 0.00 0 Step Step k12-biology GPQA 0.24 ReST Ours ReST Ours ReST Ours 0.78 0.22 0.76 0.20 0.74 0.18 Accuracy Accuracy 0.72 0.16 0.70 0.14 0.68 0.12 0.66 Step Step Total KAOYAN 0.66 ReST… find
  6. Cassano, Federico, John Gouwar, Daniel Nguyen, Sydney Nguyen, et al. “MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation”. In: IEEE Transactions on Software Engineering 49.7 (2023), pp. 3675– 3691. DOI : 10.1109/TSE.2023.3267446. — [doi:10.1109/tse.2023.3267446](https://doi.org/10.1109/tse.2023.3267446) find
  7. Chen, Jianlv et al. “Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation”. In: arXiv preprint arXiv:2402.03216 (2024). find our card
  8. Chen, Xingyu et al. “Do NOT Think That Much for 2+ 3=? On the Overthinking of o1-Like LLMs”. In: arXiv preprint arXiv:2412.21187 (2024). find our card
  9. Everitt, Tom et al. Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective. 2021. arXiv: 1908.04734 [cs.AI]. URL : https://arxiv.org/abs/1908.04734. — [arXiv:1908.04734](https://arxiv.org/abs/1908.04734) ↗
  10. Gadre, Samir Yitzhak et al. “Datacomp: In search of the next generation of multimodal datasets”. In: Advances in Neural Information Processing Systems 36 (2024). find
  11. Grattafiori, Aaron et al. The Llama 3 Herd of Models. 2024. arXiv: 2407.21783 [cs.AI]. URL : https://arxiv. org/abs/2407.21783. find
  12. Gulcehre, Caglar et al. “Reinforced self-training (rest) for language modeling”. In: arXiv preprint arXiv:2308.08998 (2023). find our card
  13. Hendrycks, Dan et al. “Measuring Massive Multitask Language Understanding”. In: ArXiv abs/2009.03300 (2020). URL : https://arxiv.org/abs/2009.03300. ↗ our card
  14. Hoffmann, Jordan et al. Training Compute-Optimal Large Language Models. 2022. arXiv: 2203.15556 [cs.CL]. URL : https://arxiv.org/abs/2203.15556. ↗
  15. Huang, Yuzhen et al. “C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models”. In: ArXiv abs/2305.08322 (2023). URL : https://arxiv.org/abs/2305.08322. ↗ our card
  16. Jaech, Aaron et al. “Openai o1 system card”. In: arXiv preprint arXiv:2412.16720 (2024). find our card
  17. Jain, Naman et al. “LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code”. In: ArXiv abs/2403.07974 (2024). URL : https://arxiv.org/abs/2403.07974. ↗ our card
  18. Joulin, Armand et al. “Bag of tricks for efficient text classification”. In: arXiv preprint arXiv:1607.01759 (2016). find
  19. Kaplan, Jared et al. Scaling Laws for Neural Language Models. 2020. arXiv: 2001.08361 [cs.LG]. URL : https: //arxiv.org/abs/2001.08361. find
  20. Kool, Wouter, Herke van Hoof, and Max Welling. “Buy 4 reinforce samples, get a baseline for free!” In: (2019). find
  21. Kwon, Woosuk et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention”. In: Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. 2023. find
  22. Laurençon, Hugo et al. “Obelics: An open web-scale filtered dataset of interleaved image-text documents”. In: Advances in Neural Information Processing Systems 36 (2024). find our card
  23. Li, Jeffrey et al. “Datacomp-lm: In search of the next generation of training sets for language models”. In: arXiv preprint arXiv:2406.11794 (2024). find our card
  24. Li, Ming et al. “From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning”. In: arXiv preprint arXiv:2308.12032 (2023). find our card
  25. Li, Raymond et al. StarCoder: may the source be with you! 2023. arXiv: 2305 . 06161 [cs.CL]. URL : https : //arxiv.org/abs/2305.06161. find
  26. Lightman, Hunter et al. “Let’s Verify Step by Step”. In: arXiv preprint arXiv:2305.20050 (2023). find our card
  27. Liu, Wei et al. “What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning”. In: arXiv preprint arXiv:2312.15685 (2023). find our card
  28. Lozhkov, Anton et al. StarCoder 2 and The Stack v2: The Next Generation. 2024. arXiv: 2402.19173 [cs.SE]. URL : https://arxiv.org/abs/2402.19173. ↗
  29. Lu, Pan et al. “Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts”. In: arXiv preprint arXiv:2310.02255 (2023). find our card
  30. McAleese, Nat et al. LLM Critics Help Catch LLM Bugs. 2024. arXiv: 2407.00215 [cs.SE]. URL : https://arxiv. org/abs/2407.00215. — [arXiv:2407.00215](https://arxiv.org/abs/2407.00215) find
  31. Mei, Jincheng et al. “On principled entropy exploration in policy optimization”. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence. 2019, pp. 3130–3136. find
  32. Muennighoff, Niklas et al. Scaling Data-Constrained Language Models. 2023. arXiv: 2305.16264 [cs.CL]. URL : https://arxiv.org/abs/2305.16264. ↗
  33. Nachum, Ofir et al. “Bridging the gap between value and policy based reinforcement learning”. In: Advances in neural information processing systems 30 (2017). find
  34. OpenAI. “Learning to reason with LLMs”. In: (2024). URL : https://openai.com/index/learning-to-reason-with-llms/. ↗
  35. Ouyang, Long et al. “Training language models to follow instructions with human feedback”. In: Advances in neural information processing systems 35 (2022), pp. 27730–27744. find
  36. Pan, Alexander, Kush Bhatia, and Jacob Steinhardt. “The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models”. In: International Conference on Learning Representations. 2022. URL : https://openreview. net/forum?id=JYtwGwIL7ye. find
  37. Paster, Keiran et al. “Openwebmath: An open dataset of high-quality mathematical web text”. In: arXiv preprint arXiv:2310.06786 (2023). find our card
  38. Penedo, Guilherme et al. “The fineweb datasets: Decanting the web for the finest text data at scale”. In: arXiv preprint arXiv:2406.17557 (2024). find our card
  39. Qin, Ruoyu et al. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. 2024. arXiv: 2407. 00079 [cs.DC]. URL : https://arxiv.org/abs/2407.00079. ↗
  40. Rafailov, Rafael et al. “Direct preference optimization: Your language model is secretly a reward model”. In: Advances in Neural Information Processing Systems 36 (2024). find
  41. Schuhmann, Christoph et al. “Laion-5b: An open large-scale dataset for training next generation image-text models”. In: Advances in Neural Information Processing Systems 35 (2022), pp. 25278–25294. find our card
  42. Shoeybi, Mohammad et al. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. 2020. arXiv: 1909.08053 [cs.CL]. URL : https://arxiv.org/abs/1909.08053. ↗
  43. Silver, David et al. “Mastering the game of go without human knowledge”. In: nature 550.7676 (2017), pp. 354–359. find
  44. Snell, Charlie et al. “Scaling llm test-time compute optimally can be more effective than scaling model parameters”. In: arXiv preprint arXiv:2408.03314 (2024). find our card
  45. Su, Dan et al. “Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset”. In: arXiv preprint arXiv:2412.02595 (2024). find
  46. Su, Jianlin et al. “Roformer: Enhanced transformer with rotary position embedding”. In: Neurocomputing 568 (2024), p. 127063. find
  47. Team, Gemini et al. Gemini: A Family of Highly Capable Multimodal Models. 2024. arXiv: 2312.11805 [cs.CL]. URL : https://arxiv.org/abs/2312.11805. ↗ our card
  48. Tomar, Manan et al. “Mirror descent policy optimization”. In: arXiv preprint arXiv:2005.09814 (2020). — [arXiv:2005.09814](https://arxiv.org/abs/2005.09814) find
  49. Vaswani, Ashish et al. “Attention is All you Need”. In: Advances in Neural Information Processing Systems. Ed. by I. Guyon et al. Vol. 30. Curran Associates, Inc., 2017. URL : https://proceedings.neurips.cc/paper\_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. ↗
  50. Villalobos, Pablo et al. Will we run out of data? Limits of LLM scaling based on human-generated data. 2024. arXiv: 2211.04325 [cs.LG]. URL : https://arxiv.org/abs/2211.04325. ↗
  51. Vinyals, Oriol et al. “Grandmaster level in StarCraft II using multi-agent reinforcement learning”. In: nature 575.7782 (2019), pp. 350–354. find
  52. Wang, Ke et al. “Measuring multimodal mathematical reasoning with math-vision dataset”. In: arXiv preprint arXiv:2402.14804 (2024). find our card
  53. Wei, Haoran et al. “General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model”. In: arXiv preprint arXiv:2409.01704 (2024). — [arXiv:2409.01704](https://arxiv.org/abs/2409.01704) find
  54. Wei, Jason et al. “Chain-of-thought prompting elicits reasoning in large language models”. In: Advances in neural information processing systems 35 (2022), pp. 24824–24837. find our card
  55. Wu, Yangzhen et al. “Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models”. In: arXiv preprint arXiv:2408.00724 (2024). find our card
  56. Xu, Liang et al. “CLUE: A Chinese Language Understanding Evaluation Benchmark”. In: International Conference on Computational Linguistics. 2020. URL : https://arxiv.org/abs/2004.05986. ↗ our card
  57. Yang, Enneng et al. “Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities”. In: arXiv preprint arXiv:2408.07666 (2024). find
  58. Yao, Shunyu et al. “Tree of thoughts: Deliberate problem solving with large language models”. In: Advances in Neural Information Processing Systems 36 (2024). find
  59. Yue, Xiang, Yuansheng Ni, et al. “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 9556–9567. find our card
  60. Yue, Xiang, Xingwei Qu, et al. “Mammoth: Building math generalist models through hybrid instruction tuning”. In: arXiv preprint arXiv:2309.05653 (2023). find our card
  61. Zhang, Lunjun et al. “Generative verifiers: Reward modeling as next-token prediction, 2024”. In: URL https://arxiv. org/abs/2408.15240 (2024). find our card
  62. Zheng, Lianmin et al. SGLang: Efficient Execution of Structured Language Model Programs. 2024. arXiv: 2312.07104 [cs.AI]. URL : https://arxiv.org/abs/2312.07104. ↗ our card
  63. Zhou, Jeffrey et al. “Instruction-Following Evaluation for Large Language Models”. In: ArXiv abs/2311.07911 (2023). URL : https://arxiv.org/abs/2311.07911. ↗ our card
  64. Zhu, Wanrong et al. “Multimodal c4: An open, billion-scale corpus of images interleaved with text”. In: Advances in Neural Information Processing Systems 36 (2024). find