Elamur Research
☰
home / papers / 2025 / Kimi K2: Open Agentic Intelligence

Kimi K2: Open Agentic Intelligence

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-ND 4.0 (arXiv)" does not allow redistributing the text itself. Original — arxiv.org/pdf/2507.20534

Summary

This technical report addresses two obstacles to building agentic LLMs. High-quality pre-training data is scarce, so the learning signal extracted per token matters more. Examples of multi-step planning and tool use are also rare in natural text. The Kimi Team's answer is Kimi K2, a sparse Mixture-of-Experts model with 1.04T total and 32B active parameters, built on MLA in a design similar to DeepSeek-V3. For pre-training on 15.5T tokens, the team created MuonClip. It extends the Muon optimizer with QK-Clip, which shrinks a head's query and key projection weights after an update whenever that head's largest attention logit exceeds a threshold. This prevents the logit blow-ups that destabilised plain Muon, and the full run had no loss spikes. Post-training then adds a pipeline that generates verified tool-use trajectories in simulated and real environments. It also includes an RL stage that combines verifiable rewards with a self-critique rubric reward. Without extended reasoning, K2 leads open non-thinking models on agentic and coding benchmarks and nears the Claude 4 models. Releasing the checkpoints openly gives the research community a strong base for agent work.

Key points

  • Architecture and training scale: an ultra-sparse MoE transformer with 1.04T parameters (32B activated), trained on 15.5T tokens with the MuonClip optimizer.
  • QK-Clip behaviour:
    • In a mid-scale 9B-active/53B-total model trained with vanilla Muon, max attention logits passed 1000.
    • In K2 with τ = 100, logits stayed at the cap and then settled into a normal range after roughly 30% of training.
  • Benchmark results (non-thinking):
    • Agentic and software engineering: Tau2-Bench 66.1, ACEBench (En) 76.5, SWE-Bench Verified 65.8, SWE-Bench Multilingual 47.3.
    • Coding, maths and STEM: LiveCodeBench v6 53.7, OJBench 27.1, AIME 2025 49.5, GPQA-Diamond 75.1.
    • LMSYS Arena (17 July 2025): top open-source model and 5th overall.
  • Safety evaluation:
    • Red-teaming used Promptfoo, comparing K2 with DeepSeek-V3, DeepSeek-R1 and Qwen3-235B-A22B.
    • Base64 attacks had little effect, with pass rates near 100%. Crescendo attacks were the most effective.
  • Known limitations and release:
    • On hard problems or with vague tool definitions, the model can produce overly long outputs, leading to truncation or incomplete tool calls.
    • Enabling tools unnecessarily can hurt performance on some tasks, and one-shot prompting is weaker than an agentic coding framework for building complete software projects.
    • Base and instruct checkpoints are public (moonshotai/Kimi-K2-Instruct on Hugging Face).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

315 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 313 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2507.20534 · arXiv 2507.20534

@misc{team2025kimi,
  title = {{Kimi K2: Open Agentic Intelligence}},
  author = {Team, Kimi and Bai, Yifan and Bao, Yiping and Charles, Y. and Chen, Cheng and Chen, Guanduo and Chen, Haiting and Chen, Huarong and Chen, Jiahao and Chen, Ningxin and Chen, Ruijue and Chen, Yanru and Chen, Yuankun and Chen, Yutian and Chen, Zhuofu and Cui, Jialei and Ding, Hao and Dong, Mengnan and Du, Angang and Du, Chenzhuang and Du, Dikang and Du, Yulun and Fan, Yu and Feng, Yichen and Fu, Kelin and Gao, Bofei and Gao, Chenxiao and Gao, Hongcheng and Gao, Peizhong and Gao, Tong and Ge, Yuyao and Geng, Shangyi and Gu, Qizheng and Gu, Xinran and Guan, Longyu and Guo, Haiqing and Guo, Jianhang and Hao, Xiaoru and He, Tianhong and He, Weiran and He, Wenyang and He, Yunjia and Hong, Chao and Hu, Hao and Hu, Yangyang and Hu, Zhenxing and Huang, Weixiao and Huang, Zhiqi and Huang, Zihao and Jiang, Tao and Jiang, Zhejun and Jin, Xinyi and Kang, Yongsheng and Lai, Guokun and Li, Cheng and Li, Fang and Li, Haoyang and Li, Ming and Li, Wentao and Li, Yang and Li, Yanhao and Li, Yiwei and Li, Zhaowei and Li, Zheming and Lin, Hongzhan and Lin, Xiaohan and Lin, Zongyu and Liu, Chengyin and Liu, Chenyu and Liu, Hongzhang and Liu, Jingyuan and Liu, Junqi and Liu, Liang and Liu, Shaowei and Liu, T.Y. and Liu, Tianwei and Liu, Weizhou and Liu, Yangyang and Liu, Yibo and Liu, Yiping and Liu, Yue and Liu, Zhengying and Lu, Enzhe and Lu, Haoyu and Lu, Lijun and Luo, Yashuo and Ma, Shengling and Ma, Xinyu and Ma, Yingwei and Mao, Shaoguang and Mei, Jie and Men, Xin and Miao, Yibo and Pan, Siyuan and Peng, Yebo and Qin, Ruoyu and Qin, Zeyu and Qu, Bowen and Shang, Zeyu and Shi, Lidong and Shi, Shengyuan and Song, Feifan and Su, Jianlin and Su, Zhengyuan and Sui, Lin and Sun, Xinjie and Sung, Flood and Tai, Yunpeng and Tang, Heyi and Tao, Jiawen and Teng, Qifeng and Tian, Chaoran and Wang, Chensi and Wang, Dinglu and Wang, Feng and Wang, Hailong and Wang, Haiming and Wang, Jianzhou and Wang, Jiaxing and Wang, Jinhong and Wang, Shengjie and Wang, Shuyi and Wang, Si and Wang, Xinyuan and Wang, Yao and Wang, Yejie and Wang, Yiqin and Wang, Yuxin and Wang, Yuzhi and Wang, Zhaoji and Wang, Zhengtao and Wang, Zhengtao and Wang, Zhexu and Wei, Chu and Wei, Qianqian and Wu, Haoning and Wu, Wenhao and Wu, Xingzhe and Wu, Yuxin and Xiao, Chenjun and Xie, Jin and Xie, Xiaotong and Xiong, Weimin and Xu, Boyu and Xu, Jinjing and Xu, L.H. and Xu, Lin and Xu, Suting and Xu, Weixin and Xu, Xinran and Xu, Yangchuan and Xu, Ziyao and Xu, Jing and Xu, Jing and Yan, Junjie and Yan, Yuzi and Yang, Hao and Yang, Xiaofei and Yang, Yi and Yang, Ying and Yang, Zhen and Yang, Zhilin and Yang, Zonghan and Yao, Haotian and Yao, Xingcheng and Ye, Wenjie and Ye, Zhuorui and Yin, Bohong and Yu, Longhui and Yuan, Enming and Yuan, Hongbang and Yuan, Mengjie and Yuan, Siyu and Zhan, Haobing and Zhang, Dehao and Zhang, Hao and Zhang, Wanlu and Zhang, Xiaobin and Zhang, Yadong and Zhang, Yangkun and Zhang, Yichi and Zhang, Yizhi and Zhang, Yongting and Zhang, Yu and Zhang, Yutao and Zhang, Yutong and Zhang, Zheng and Zhao, Haotian and Zhao, Yikai and Zhao, Zijia and Zheng, Huabin and Zheng, Shaojie and Zhong, Longguang and Zhou, Jianren and Zhou, Xinyu and Zhou, Zaida and Zhu, Jinguo and Zhu, Zhen and Zhuang, Weiyu and Zu, Xinxing},
  year = {2025},
  doi = {10.48550/arxiv.2507.20534},
  eprint = {2507.20534},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.20534}
}

References

92 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Austin et al. Program Synthesis with Large Language Models. 2021. arXiv: 2108.07732 [cs.PL]. URL : https://arxiv.org/abs/2108.07732. ↗ our card
  2. Yushi Bai et al. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks. 2025. arXiv: 2412.15204 [cs.CL]. URL : https://arxiv.org/abs/2412.15204. ↗ our card
  3. Victor Barres et al. τ 2 -Bench: Evaluating Conversational Agents in a Dual-Control Environment. 2025. arXiv: 2506.07982 [cs.AI]. URL : https://arxiv.org/abs/2506.07982. ↗
  4. Stella Biderman et al. “Lessons from the trenches on reproducible evaluation of language models”. In: arXiv preprint arXiv:2405.14782 (2024). find
  5. Greg Brockman et al. OpenAI Gym. 2016. arXiv: 1606.01540 [cs.LG]. URL : https://arxiv.org/abs/1606.01540. ↗
  6. Federico Cassano et al. “MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation”. In: IEEE Transactions on Software Engineering 49.7 (2023), pp. 3675–3691. DOI : 10.1109/TSE.2023.3267446. — [doi:10.1109/tse.2023.3267446](https://doi.org/10.1109/tse.2023.3267446) find
  7. Chen Chen et al. “ACEBench: Who Wins the Match Point in Tool Learning?” In: arXiv e-prints (2025), arXiv– 2501. find
  8. Mark Chen et al. “Evaluating Large Language Models Trained on Code”. In: (2021). arXiv: 2107.03374 [cs.LG]. find our card
  9. Peter Clark et al. “Think you have solved question answering? try arc, the ai2 reasoning challenge”. In: arXiv preprint arXiv:1803.05457 (2018). find
  10. Karl Cobbe et al. Training Verifiers to Solve Math Word Problems. 2021. arXiv: 2110.14168 [cs.LG]. URL : https://arxiv.org/abs/2110.14168. ↗ our card
  11. DeepSeek-AI. DeepSeek-V3 Technical Report. 2024. arXiv: 2412 . 19437 [cs.CL]. URL : https : / / arxiv.org/abs/2412.19437. find
  12. Mostafa Dehghani et al. “Scaling vision transformers to 22 billion parameters”. In: International conference on machine learning. PMLR. 2023, pp. 7480–7512. find
  13. Guanting Dong et al. Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models. 2024. arXiv: 2406 . 13542 [cs.CL]. URL : https : / / arxiv . org / abs / 2406 . 13542. find
  14. Xinrun Du et al. “Supergpqa: Scaling llm evaluation across 285 graduate disciplines”. In: arXiv preprint arXiv:2502.14739 (2025). find
  15. Dheeru Dua et al. “DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs”. In: CoRR abs/1903.00161 (2019). arXiv: 1903.00161. URL : http://arxiv.org/abs/1903.00161. ↗ our card
  16. Kazuki Fujii et al. Rewriting Pre-Training Data Boosts LLM Performance in Math and Code. 2025. arXiv: 2505.02881 [cs.LG]. URL : https://arxiv.org/abs/2505.02881. ↗
  17. Paul Gauthier. Aider LLM Leaderboards. https://aider.chat/docs/leaderboards/. 2025. ↗
  18. Aryo Pradipta Gema et al. “Are we done with mmlu?” In: arXiv preprint arXiv:2406.04127 (2024). find
  19. Alex Gu et al. “Cruxeval: A benchmark for code reasoning, understanding and execution”. In: arXiv preprint arXiv:2401.03065 (2024). find our card
  20. Daya Guo et al. “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning”. In: arXiv preprint arXiv:2501.12948 (2025). find our card
  21. Zhicheng Guo et al. “StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models”. In: arXiv preprint arXiv:2403.07714 (2025). find
  22. Aaron Harlap et al. “Pipedream: Fast and efficient pipeline parallel dnn training”. In: arXiv preprint arXiv:1806.03377 (2018). find
  23. Y He et al. “Chinese simpleqa: A chinese factuality evaluation for large language models, 2024a”. In: URL https://arxiv. org/abs/2411.07140 (). find
  24. Dan Hendrycks et al. “Measuring massive multitask language understanding”. In: arXiv preprint arXiv:2009.03300 (2020). find our card
  25. Dan Hendrycks et al. Measuring Mathematical Problem Solving With the MATH Dataset. 2021. arXiv: 2103. 03874 [cs.LG]. URL : https://arxiv.org/abs/2103.03874. ↗ our card
  26. Shengding Hu et al. “Minicpm: Unveiling the potential of small language models with scalable training strategies”. In: arXiv preprint arXiv:2404.06395 (2024). find our card
  27. Jiaxin Huang et al. “Large language models can self-improve”. In: arXiv preprint arXiv:2210.11610 (2022). find our card
  28. Siming Huang et al. OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models. 2025. arXiv: 2411.04905 [cs.CL]. URL : https://arxiv.org/abs/2411.04905. ↗
  29. Yanping Huang et al. “Gpipe: Efficient training of giant neural networks using pipeline parallelism”. In: Advances in neural information processing systems 32 (2019). find
  30. Yuzhen Huang et al. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. 2023. arXiv: 2305.08322 [cs.CL]. URL : https://arxiv.org/abs/2305.08322. ↗ our card
  31. Alon Jacovi et al. The FACTS Grounding Leaderboard: Benchmarking LLMs’ Ability to Ground Responses to Long-Form Input. 2025. arXiv: 2501.03200 [cs.CL]. URL : https://arxiv.org/abs/2501.03200. ↗
  32. Naman Jain et al. “Livecodebench: Holistic and contamination free evaluation of large language models for code”. In: arXiv preprint arXiv:2403.07974 (2024). find our card
  33. Carlos E Jimenez et al. “SWE-bench: Can Language Models Resolve Real-world Github Issues?” In: The Twelfth International Conference on Learning Representations. 2024. URL : https://openreview.net/forum?id=VTF8yNQM66. ↗ our card
  34. Keller Jordan et al. Muon: An optimizer for hidden layers in neural networks. 2024. URL : https : / / kellerjordan.github.io/posts/muon/. find
  35. Mandar Joshi et al. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. 2017. arXiv: 1705.03551 [cs.CL]. URL : https://arxiv.org/abs/1705.03551. ↗ our card
  36. Kimi Team. “Kimi k1. 5: Scaling reinforcement learning with llms”. In: arXiv preprint arXiv:2501.12599 (2025). find our card
  37. Diederik P. Kingma and Jimmy Ba. “Adam: A Method for Stochastic Optimization”. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. 2015. URL : http://arxiv.org/abs/1412.6980. ↗
  38. Satyapriya Krishna et al. Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation. 2025. arXiv: 2409.12941 [cs.CL]. URL : https://arxiv.org/abs/2409.12941. ↗
  39. Joel Lamy-Poirier. “Breadth-first pipeline parallelism”. In: Proceedings of Machine Learning and Systems 5 (2023), pp. 48–67. find
  40. Dmitry Lepikhin et al. “Gshard: Scaling giant models with conditional computation and automatic sharding”. In: arXiv preprint arXiv:2006.16668 (2020). find
  41. Haonan Li et al. CMMLU: Measuring massive multitask language understanding in Chinese. 2024. arXiv: 2306.09212 [cs.CL]. URL : https://arxiv.org/abs/2306.09212. ↗ our card
  42. Jia Li et al. “Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions”. In: Hugging Face repository 13.9 (2024), p. 9. find
  43. Tianle Li et al. “From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline”. In: arXiv preprint arXiv:2406.11939 (2024). find our card
  44. Bill Yuchen Lin et al. ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning. 2025. arXiv: 2502. 01100 [cs.AI]. URL : https://arxiv.org/abs/2502.01100. ↗
  45. Aixin Liu et al. “Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model”. In: arXiv preprint arXiv:2405.04434 (2024). find
  46. Jiawei Liu et al. “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation”. In: Advances in Neural Information Processing Systems 36 (2023), pp. 21558–21572. find our card
  47. Jingyuan Liu et al. “Muon is scalable for LLM training”. In: arXiv preprint arXiv:2502.16982 (2025). find
  48. Ziming Liu et al. “Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency”. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. SC ’23. ACM, Nov. 2023, pp. 1–13. DOI : 10 . 1145 / 3581784 . 3607073.… find
  49. Ilya Loshchilov and Frank Hutter. “Decoupled Weight Decay Regularization”. In: International Conference on Learning Representations. 2019. URL : https://openreview.net/forum?id=Bkg6RiCqY7. ↗
  50. Pratyush Maini et al. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. 2024. arXiv: 2401.16380 [cs.CL]. URL : https://arxiv.org/abs/2401.16380. ↗
  51. Samuel Miserendino et al. “SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?” In: arXiv preprint arXiv:2502.12115 (2025). find
  52. Arindam Mitra et al. “Agentinstruct: Toward generative teaching with agentic flows”. In: arXiv preprint arXiv:2407.03502 (2024). find
  53. Ivan Moshkov et al. “Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset”. In: arXiv preprint arXiv:2504.16891 (2025). find
  54. Deepak Narayanan et al. “Efficient large-scale language model training on gpu clusters using megatron-lm”. In: Proceedings of the international conference for high performance computing, networking, storage and analysis. 2021, pp. 1–15. find
  55. Long Ouyang et al. “Training language models to follow instructions with human feedback”. In: Advances in neural information processing systems 35 (2022), pp. 27730–27744. find
  56. Bowen Peng et al. “Yarn: Efficient context window extension of large language models”. In: arXiv preprint arXiv:2309.00071 (2023). find
  57. Long Phan et al. Humanity’s Last Exam. 2025. arXiv: 2501.14249 [cs.LG]. URL : https://arxiv. org/abs/2501.14249. find our card
  58. Penghui Qi et al. “Zero bubble pipeline parallelism”. In: arXiv preprint arXiv:2401.10241 (2023). find
  59. Yujia Qin et al. “Toolllm: Facilitating large language models to master 16000+ real-world apis”. In: arXiv preprint arXiv:2307.16789 (2023). find our card
  60. Qwen et al. Qwen2.5 Technical Report. 2025. arXiv: 2412.15115 [cs.CL]. URL : https://arxiv. org/abs/2412.15115. find
  61. Samyam Rajbhandari et al. “Zero: Memory optimizations toward training trillion parameter models”. In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE. 2020, pp. 1–16. find
  62. David Rein et al. “Gpqa: A graduate-level google-proof q&a benchmark”. In: First Conference on Language Modeling. 2024. find our card
  63. Keisuke Sakaguchi et al. “Winogrande: An adversarial winograd schema challenge at scale”. In: Communications of the ACM 64.9 (2021), pp. 99–106. find
  64. David Silver and Richard S Sutton. “Welcome to the era of experience”. In: Google AI 1 (2025). find
  65. Ved Sirdeshmukh et al. MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs. 2025. arXiv: 2501 . 17399 [cs.CL]. URL : https : / / arxiv . org / abs / 2501 . 17399. find
  66. Giulio Starace et al. “PaperBench: Evaluating AI’s Ability to Replicate AI Research”. In: arXiv preprint arXiv:2504.01848 (2025). find
  67. Hao Sun et al. ZeroSearch: Incentivize the Search Capability of LLMs without Searching. 2025. arXiv: 2505. 04588 [cs.CL]. URL : https://arxiv.org/abs/2505.04588. ↗
  68. Mirac Suzgun et al. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. 2022. arXiv: 2210.09261 [cs.CL]. URL : https://arxiv.org/abs/2210.09261. ↗ our card
  69. Manveer Singh Tamber et al. “Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards”. In: arXiv preprint arXiv:2505.04847 (2025). find
  70. Gemma Team et al. “Gemma 2: Improving open language models at a practical size”. In: arXiv preprint arXiv:2408.00118 (2024). find
  71. LlaMA Team. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation — ai.meta.com. https://ai.meta.com/blog/llama-4-multimodal-intelligence/. [Accessed 15-07-2025]. ↗
  72. The Terminal-Bench Team. Terminal-Bench: A Benchmark for AI Agents in Terminal Environments. Apr. 2025. URL : https://github.com/laude-institute/terminal-bench. ↗
  73. Ashish Vaswani et al. “Attention is All you Need”. In: Advances in Neural Information Processing Systems. Ed. by I. Guyon et al. Vol. 30. Curran Associates, Inc., 2017. URL : https://proceedings.neurips. cc/paper\_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. ↗
  74. Vectara. Hallucination Evaluation Model (Revision 7437011). 2024. URL : https://huggingface.co/vectara/hallucination\_evaluation\_model. ↗
  75. Joshua Vendrow et al. “Do large language model benchmarks test reliability?” In: arXiv preprint arXiv:2502.03461 (2025). find
  76. Yizhong Wang et al. “Self-instruct: Aligning language models with self-generated instructions”. In: arXiv preprint arXiv:2212.10560 (2022). find
  77. Yubo Wang et al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. 2024. arXiv: 2406.01574 [cs.CL]. URL : https://arxiv.org/abs/2406.01574. ↗ our card
  78. Zhexu Wang et al. OJBench: A Competition Level Code Benchmark For Large Language Models. 2025. arXiv: 2506.16395 [cs.CL]. URL : https://arxiv.org/abs/2506.16395. — [arXiv:2506.16395](https://arxiv.org/abs/2506.16395) ↗
  79. Jason Wei et al. “Measuring short-form factuality in large language models”. In: arXiv preprint arXiv:2411.04368 (2024). find our card
  80. Tianwen Wei et al. CMATH: Can Your Language Model Pass Chinese Elementary School Math Test? 2023. arXiv: 2306.16636 [cs.CL]. URL : https://arxiv.org/abs/2306.16636. ↗
  81. Colin White et al. “LiveBench: A Challenging, Contamination-Free LLM Benchmark”. In: The Thirteenth International Conference on Learning Representations. 2025. find
  82. Mitchell Wortsman et al. “Small-scale proxies for large-scale transformer training instabilities, 2023”. In: URL https://arxiv. org/abs/2309.14322 (). find
  83. Can Xu et al. WizardLM: Empowering large pre-trained language models to follow complex instructions. 2025. arXiv: 2304.12244 [cs.CL]. URL : https://arxiv.org/abs/2304.12244. ↗ our card
  84. Zhangchen Xu et al. KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding. 2025. arXiv: 2503.02951 [cs.LG]. URL : https://arxiv.org/abs/2503.02951. ↗
  85. John Yang et al. SWE-smith: Scaling Data for Software Engineering Agents. 2025. arXiv: 2504 . 21798 [cs.SE]. URL : https://arxiv.org/abs/2504.21798. — [arXiv:2504.21798](https://arxiv.org/abs/2504.21798) ↗
  86. Shunyu Yao et al. “tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains”. In: arXiv preprint arXiv:2406.12045 (2024). find our card
  87. Daoguang Zan et al. “Multi-swe-bench: A multilingual benchmark for issue resolving”. In: arXiv preprint arXiv:2504.02605 (2025). — [arXiv:2504.02605](https://arxiv.org/abs/2504.02605) find
  88. Eric Zelikman et al. “Star: Bootstrapping reasoning with reasoning”. In: Advances in Neural Information Processing Systems 35 (2022), pp. 15476–15488. find our card
  89. Rowan Zellers et al. “Hellaswag: Can a machine really finish your sentence?” In: arXiv preprint arXiv:1905.07830 (2019). find
  90. Wanjun Zhong et al. “Agieval: A human-centric benchmark for evaluating foundation models”. In: arXiv preprint arXiv:2304.06364 (2023). find our card
  91. Jeffrey Zhou et al. “Instruction-Following Evaluation for Large Language Models”. In: ArXiv abs/2311.07911 (2023). URL : https://arxiv.org/abs/2311.07911. ↗ our card
  92. Qin Zhu et al. AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models. 2025. arXiv: 2502 . 16906 [cs.CL]. URL : https : / / arxiv . org / abs / 2502 . 16906. find