Elamur Research⌕
☰
home / papers / 2026 / Kimi K3: Open Frontier Intelligence

Kimi K3: Open Frontier Intelligence

Paper · 2026 · Kimi Team, Tongtong Bai, Yifan Bai et al.

Summary

Open-weight LLMs have advanced quickly in reinforcement learning and test-time reasoning. However, most still sit near the 1T-parameter scale, so their pre-trained base models fall behind proprietary systems. Kimi K3 scales both pre-training and RL. It is a natively multimodal Mixture-of-Experts model with 2.8T total parameters, 104B active parameters and a context of up to 1M tokens. The architecture interleaves Kimi Delta Attention (KDA) with Gated MLA layers. It also uses Attention Residuals (AttnRes) for selective access to earlier layers, Stable LatentMoE for sparse expert routing, and the MoonViT-V2 vision encoder. Post-training runs RL on coding, agentic, reasoning and knowledge environments at several reasoning-effort levels. Specialist policies are then merged through multi-teacher on-policy distillation. Supporting infrastructure includes KDA kernels, MoonEP balanced expert parallelism and resumable microVM sandboxes for million-token agentic RL. The authors claim about 2.5× better scaling efficiency than Kimi K2. Overall, K3 ranks below Claude Fable 5 and GPT-5.6 Sol but above the other models tested. The weights are fully released, making it the first open 3T-class model.

Key points

  • Architecture:
    • Each block has 3 KDA layers and 1 Gated MLA layer, a 3:1 ratio. An extra Gated MLA layer closes the backbone.
    • For each token, Stable LatentMoE routes to just 16 experts out of a pool of 896. Normalization, SiTU-GLU and Quantile Balancing keep training stable at this sparsity.
    • Per-Head Muon is the optimizer.
  • Post-training: RL covers search, software engineering, kernel optimization, vision-in-the-loop tool use, web development and autonomous tasks. Trajectories can involve thousands of tool calls.
  • Infrastructure:
    • KDA Context Parallelism and state-aware prefix caching support long contexts.
    • MoonEP uses static shapes and zero-copy communication for expert parallelism.
    • The RL system relies on partial rollouts and external KV-cache retention.
  • Results: On GDPval-AA v2 (Artificial Analysis, as of July 23, 2026), Kimi K3 scores an Elo of 1686. This places it third, behind Fable 5 (1747) and GPT-5.6 Sol (1736).
  • Release and case studies:
    • Weights are available at huggingface.co/moonshotai/Kimi-K3.
    • Demos include the MiniTriton GPU compiler.
    • In a 48-hour autonomous run, the model designed an INT4 inference-chip prototype (nano-kpu). It runs at 100 MHz with over 8,700 tokens/s in RTL simulation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, M. C., Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y. Charles, H.S. Che, Guanduo Chen, Guangyu Chen, Guanzheng Chen, Huarong Chen, Jia Chen, Jianlong Chen, Jun Chen, Kexin Chen, Peng Chen, Ruijue Chen, Wentao Chen, Xin Chen, Yang Chen, Yanru Chen, Yifei Chen, Yingjiang Chen, Yuankun Chen, Yujie Chen, Yutian Chen, Zhirong Chen, Dazhi Cheng, Yean Cheng, Jialei Cui, Jingbing Cui, Anqi Dai, Jiaqi Deng, Hao Ding, Rui Ding, Shaofeng Ding, Mengfan Dong, Mengnan Dong, Yuhao Dong, Yuxin Dong, Angang Du, Chenzhuang Du, Dikang Du, Jusen Du, Yulun Du, Yu Fan, Jing Feng, Qiulin Feng, Yichen Feng, Kelin Fu, Qiang Fu, Fuxuan Gao, Hongcheng Gao, Jingyue Gao, Tong Gao, Weijia Gao, Shangyi Geng, Jie Gong, Linhu Gong, Shengao Gong, Xiaochen Gong, Qizheng Gu, Yicheng Gu, Shuhao Guan, Haiqing Guo, Shiqi Guo, Xiang Guo, Zhengyan Guo, Beixi Hao, Wenxin Hao, Xiaoru Hao, Dailan He, Haotian He, Lehan He, Qi He, Weiran He, Xinran He, Xinyi He, Yibo He, Yunjia He, Chao Hong, Tiange Hong, Hao Hu, Jiaxi Hu, Ruikun Hu, Weiming Hu, Yangyang Hu, Zhenxing Hu, Liang Hua, Jinbin Huang, Ke Huang, Ruiyuan Huang, Siying Huang, Weixiao Huang, Yan Huang, Zhengjie Huang, Zhiqi Huang, Yulong Hui, Chaobo Jia, Yutong Jiang, Zhejun Jiang, Zuoyou Jiang, Wenyi Jin, Xinyi Jin, Yu Jing, Huanjun Kong, Guokun Lai, Aidi Li, Cheng Li, Chengyuan Li, Cong Li, Fang Li, Guanyu Li, Haoyang Li, Jia Li, Junxiong Li, Lei Li, Letian Li, Lincan Li, Weihong Li, Wentao Li, Xintong Li, Yang Li, Yishen Li, Yiwei Li, Yuxiao Li, Zhaowei Li, Zhaoxi Li, Zheming Li, Zhengxiao Li, Zhiyuan Li, Jiawei Lin, Xiaohan Lin, Yibo Lin, Zichao Lin, Ziyan Lin, Bill Liu, Boxiao Liu, Chuan Liu, Liang Liu, Shaowei Liu, Shudong Liu, Shuran Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yanming Liu, Yibo Liu, Yipeng Liu, Zhengying Liu, Zhiheng Liu, Enzhe Lu, Haoyu Lu, Linqiang Lu, Tingzhan Lu, Zhiyuan Lu, Aotian Luo, G. Luo, Junyu Luo, Yifan Luo, B. Lyu, Wenzhou Lyu, Shaoguang Mao, Yuan Mei, Xin Men, Minqing Ni, Yixuan Niu, Siyuan Pan, Shujun Peng, Zhangyang Qi, Ruoyu Qin, ZeChao Qin, Zeyu Qin, Haiquan Qiu, Jianxin Qiu, Jiezhong Qiu, Bowen Qu, Yuhao Qu, Zeyu Shang, Youbo Shao, Han Shen, Jincheng Shi, Juanfeng Shi, Lidong Shi, Shengyuan Shi, Wingchun Siu, Pengwei Song, Xiaoxi Song, Jianlin Su, Yunfeng Su, Zhaochen Su, Lin Sui, Jingsong Sun, Junyao Sun, Shaoning Sun, Shuzhe Sun, Tongyu Sun, Yujun Sun, Yunpeng Tai, Chuning Tang, Heyi Tang, Sirui Tang, Zecheng Tang, Chaoran Tian, Rongpeng Tian, Yu Tian, Wei Tu, Chensi Wang, Chuang Wang, Chunjie Wang, Dinglu Wang, Feng Wang, Hailong Wang, Haiming Wang, Hao Wang, Hao Wang, Huaqing Wang, Hui Wang, Jiayi Wang, Jinglong Wang, Jinhong Wang, Jiuzheng Wang, Linian Wang, Shaobo Wang, Shenzhi Wang, Shuyi Wang, Si Wang, Siyuan Wang, Tianfu Wang, Wenjue Wang, Xingran Wang, Xinmei Wang, Xinyuan Wang, Xusheng Wang, Yalin Wang, Yangkun Wang, Yao Wang, Yaoyu Wang, Yejie Wang, Yiqin Wang, Yucheng Wang, Yuzhi Wang, Zhaoji Wang, Zhaowei Wang, Zhengtao Wang, Zhenhao Wang, Zhongsheng Wang, Zifan Wang, Chu Wei, Ming Wei, Shouxin Wei, Zichen Wen, Fan Wu, Haoning Wu, Rucong Wu, Wenhao Wu, Xiaoxue Wu, Yingcong Wu, Yongqi Wu, Yuxin Wu, Zijian Wu, Xinglang Xian, Chenxuan Xiang, Yuye Xiang, Bocheng Xiao, Chenjun Xiao, Xin Xiao, Jin Xie, Xiaotong Xie, Yifeng Xie, Zhe Xie, Bowei Xing, Yiming Xiong, Baosheng Xu, Boyu Xu, Jiale Xu, Jianfan Xu, Jing Xu, Jinjing Xu, L.H. Xu, Qingtao Xu, Shuyao Xu, Suting Xu, Tiantian Xu, Tianxiang Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ye Xu, Yueni Xu, Ziyao Xu, Haonan Xue, Junjie Yan, Yaoyao Yan, Fan Yang, Guangyao Yang, Hao Yang, Junwei Yang, Ruoyu Yang, Wenjie Yang, Xiaofei Yang, Xinyu Yang, Yi Yang, Yiling Yang, Ying Yang, Yuchen Yang, Zhen Yang, Zhilin Yang, Zian Yang, Zuhao Yang, Haotian Yao, Dan Ye, Haoran Ye, Wenjie Ye, Zhanbo Ye, Bohong Yin, Haoxiang Yin, Xietong Yin, Chengzhen Yu, Haozhen Yu, Longhui Yu, Shengnan Yu, Shuying Yu, Tianxiang Yu, Enming Yuan, Mengjie Yuan, Tongtian Yue, Wei Yue, Yang Yue, Dunyuan Zha, Haobing Zhan, B.H. Zhang, Dehao Zhang, Fei Zhang, Hao Zhang, Haoyuan Zhang, Huanyu Zhang, Jiapei Zhang, Jiaxuan Zhang, Jin Zhang, Kaiyi Zhang, Miaozhen Zhang, Puqi Zhang, Qinglei Zhang, Rong Zhang, Rui Zhang, Shaoshuai Zhang, Shiyi Zhang, Xiaobin Zhang, Xiaoyun Zhang, Y. Zhang, Yangkun Zhang, Ye Zhang, Yichi Zhang, Yikun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Zijing Zhang, Bin Zhao, Chenguang Zhao, Feifan Zhao, Jinglun Zhao, Jinxiang Zhao, Shuai Zhao, Wenshuo Zhao, Xiangyu Zhao, Xuanle Zhao, Yikai Zhao, Zijia Zhao, Haozhi Zheng, Huabin Zheng, Ruihan Zheng, Shaojie Zheng, Tengyang Zheng, Haofeng Zhong, Lei Zhong, Longguang Zhong, M. Zhou, Qiankang Zhou, Runjie Zhou, Ruozhang Zhou, Xinyu Zhou, Yiqiao Zhou, Zaida Zhou, Jinguo Zhu, Liya Zhu, Xinhao Zhu, Yangjunfeng Zhu, Yuxuan Zhu, Zhen Zhu, Chen Zhuang, Weiyu Zhuang and Xinxing Zu. 2026. Kimi K3: Open Frontier Intelligence. arXiv:2607.24653.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2607.24653 · arXiv 2607.24653

@misc{team2026kimi,
  title = {{Kimi K3: Open Frontier Intelligence}},
  author = {Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, M. C., Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y. Charles, H.S. Che, Guanduo Chen, Guangyu Chen, Guanzheng Chen, Huarong Chen, Jia Chen, Jianlong Chen, Jun Chen, Kexin Chen, Peng Chen, Ruijue Chen, Wentao Chen, Xin Chen, Yang Chen, Yanru Chen, Yifei Chen, Yingjiang Chen, Yuankun Chen, Yujie Chen, Yutian Chen, Zhirong Chen, Dazhi Cheng, Yean Cheng, Jialei Cui, Jingbing Cui, Anqi Dai, Jiaqi Deng, Hao Ding, Rui Ding, Shaofeng Ding, Mengfan Dong, Mengnan Dong, Yuhao Dong, Yuxin Dong, Angang Du, Chenzhuang Du, Dikang Du, Jusen Du, Yulun Du, Yu Fan, Jing Feng, Qiulin Feng, Yichen Feng, Kelin Fu, Qiang Fu, Fuxuan Gao, Hongcheng Gao, Jingyue Gao, Tong Gao, Weijia Gao, Shangyi Geng, Jie Gong, Linhu Gong, Shengao Gong, Xiaochen Gong, Qizheng Gu, Yicheng Gu, Shuhao Guan, Haiqing Guo, Shiqi Guo, Xiang Guo, Zhengyan Guo, Beixi Hao, Wenxin Hao, Xiaoru Hao, Dailan He, Haotian He, Lehan He, Qi He, Weiran He, Xinran He, Xinyi He, Yibo He, Yunjia He, Chao Hong, Tiange Hong, Hao Hu, Jiaxi Hu, Ruikun Hu, Weiming Hu, Yangyang Hu, Zhenxing Hu, Liang Hua, Jinbin Huang, Ke Huang, Ruiyuan Huang, Siying Huang, Weixiao Huang, Yan Huang, Zhengjie Huang, Zhiqi Huang, Yulong Hui, Chaobo Jia, Yutong Jiang, Zhejun Jiang, Zuoyou Jiang, Wenyi Jin, Xinyi Jin, Yu Jing, Huanjun Kong, Guokun Lai, Aidi Li, Cheng Li, Chengyuan Li, Cong Li, Fang Li, Guanyu Li, Haoyang Li, Jia Li, Junxiong Li, Lei Li, Letian Li, Lincan Li, Weihong Li, Wentao Li, Xintong Li, Yang Li, Yishen Li, Yiwei Li, Yuxiao Li, Zhaowei Li, Zhaoxi Li, Zheming Li, Zhengxiao Li, Zhiyuan Li, Jiawei Lin, Xiaohan Lin, Yibo Lin, Zichao Lin, Ziyan Lin, Bill Liu, Boxiao Liu, Chuan Liu, Liang Liu, Shaowei Liu, Shudong Liu, Shuran Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yanming Liu, Yibo Liu, Yipeng Liu, Zhengying Liu, Zhiheng Liu, Enzhe Lu, Haoyu Lu, Linqiang Lu, Tingzhan Lu, Zhiyuan Lu, Aotian Luo, G. Luo, Junyu Luo, Yifan Luo, B. Lyu, Wenzhou Lyu, Shaoguang Mao, Yuan Mei, Xin Men, Minqing Ni, Yixuan Niu, Siyuan Pan, Shujun Peng, Zhangyang Qi, Ruoyu Qin, ZeChao Qin, Zeyu Qin, Haiquan Qiu, Jianxin Qiu, Jiezhong Qiu, Bowen Qu, Yuhao Qu, Zeyu Shang, Youbo Shao, Han Shen, Jincheng Shi, Juanfeng Shi, Lidong Shi, Shengyuan Shi, Wingchun Siu, Pengwei Song, Xiaoxi Song, Jianlin Su, Yunfeng Su, Zhaochen Su, Lin Sui, Jingsong Sun, Junyao Sun, Shaoning Sun, Shuzhe Sun, Tongyu Sun, Yujun Sun, Yunpeng Tai, Chuning Tang, Heyi Tang, Sirui Tang, Zecheng Tang, Chaoran Tian, Rongpeng Tian, Yu Tian, Wei Tu, Chensi Wang, Chuang Wang, Chunjie Wang, Dinglu Wang, Feng Wang, Hailong Wang, Haiming Wang, Hao Wang, Hao Wang, Huaqing Wang, Hui Wang, Jiayi Wang, Jinglong Wang, Jinhong Wang, Jiuzheng Wang, Linian Wang, Shaobo Wang, Shenzhi Wang, Shuyi Wang, Si Wang, Siyuan Wang, Tianfu Wang, Wenjue Wang, Xingran Wang, Xinmei Wang, Xinyuan Wang, Xusheng Wang, Yalin Wang, Yangkun Wang, Yao Wang, Yaoyu Wang, Yejie Wang, Yiqin Wang, Yucheng Wang, Yuzhi Wang, Zhaoji Wang, Zhaowei Wang, Zhengtao Wang, Zhenhao Wang, Zhongsheng Wang, Zifan Wang, Chu Wei, Ming Wei, Shouxin Wei, Zichen Wen, Fan Wu, Haoning Wu, Rucong Wu, Wenhao Wu, Xiaoxue Wu, Yingcong Wu, Yongqi Wu, Yuxin Wu, Zijian Wu, Xinglang Xian, Chenxuan Xiang, Yuye Xiang, Bocheng Xiao, Chenjun Xiao, Xin Xiao, Jin Xie, Xiaotong Xie, Yifeng Xie, Zhe Xie, Bowei Xing, Yiming Xiong, Baosheng Xu, Boyu Xu, Jiale Xu, Jianfan Xu, Jing Xu, Jinjing Xu, L.H. Xu, Qingtao Xu, Shuyao Xu, Suting Xu, Tiantian Xu, Tianxiang Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ye Xu, Yueni Xu, Ziyao Xu, Haonan Xue, Junjie Yan, Yaoyao Yan, Fan Yang, Guangyao Yang, Hao Yang, Junwei Yang, Ruoyu Yang, Wenjie Yang, Xiaofei Yang, Xinyu Yang, Yi Yang, Yiling Yang, Ying Yang, Yuchen Yang, Zhen Yang, Zhilin Yang, Zian Yang, Zuhao Yang, Haotian Yao, Dan Ye, Haoran Ye, Wenjie Ye, Zhanbo Ye, Bohong Yin, Haoxiang Yin, Xietong Yin, Chengzhen Yu, Haozhen Yu, Longhui Yu, Shengnan Yu, Shuying Yu, Tianxiang Yu, Enming Yuan, Mengjie Yuan, Tongtian Yue, Wei Yue, Yang Yue, Dunyuan Zha, Haobing Zhan, B.H. Zhang, Dehao Zhang, Fei Zhang, Hao Zhang, Haoyuan Zhang, Huanyu Zhang, Jiapei Zhang, Jiaxuan Zhang, Jin Zhang, Kaiyi Zhang, Miaozhen Zhang, Puqi Zhang, Qinglei Zhang, Rong Zhang, Rui Zhang, Shaoshuai Zhang, Shiyi Zhang, Xiaobin Zhang, Xiaoyun Zhang, Y. Zhang, Yangkun Zhang, Ye Zhang, Yichi Zhang, Yikun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Zijing Zhang, Bin Zhao, Chenguang Zhao, Feifan Zhao, Jinglun Zhao, Jinxiang Zhao, Shuai Zhao, Wenshuo Zhao, Xiangyu Zhao, Xuanle Zhao, Yikai Zhao, Zijia Zhao, Haozhi Zheng, Huabin Zheng, Ruihan Zheng, Shaojie Zheng, Tengyang Zheng, Haofeng Zhong, Lei Zhong, Longguang Zhong, M. Zhou, Qiankang Zhou, Runjie Zhou, Ruozhang Zhou, Xinyu Zhou, Yiqiao Zhou, Zaida Zhou, Jinguo Zhu, Liya Zhu, Xinhao Zhu, Yangjunfeng Zhu, Yuxuan Zhu, Zhen Zhu, Chen Zhuang, Weiyu Zhuang and Xinxing Zu},
  year = {2026},
  doi = {10.48550/arxiv.2607.24653},
  eprint = {2607.24653},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2607.24653}
}

References

121 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. τ 3 -Banking. Sierra. 2026. URL : https://taubench.com/blog/tau-knowledge.html. ↗
  2. AA-Briefcase: Agentic Knowledge Work Benchmark. Artificial Analysis. 2026. URL : https://artificialanalysis.ai/evaluations/aa-briefcase. ↗
  3. Alexandru Agache et al. “Firecracker: Lightweight Virtualization for Serverless Applications”. In: 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI). 2020, pp. 419–434. find
  4. Agents’ Last Exam. UC Berkeley RDI. 2026. URL : https://agents-last-exam.org/leaderboard. ↗
  5. Jason Ansel et al. “PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation”. In: Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 2024. DOI : 10.1145/3620665.364036…
  6. Anthropic. Claude’s Extended Thinking. https://www.anthropic.com/research/visible-extended-thinking. Accessed: 2026-07-23. Feb. 2025. ↗
  7. Anthropic. Introducing Claude 4. https://www.anthropic.com/news/claude-4. Accessed: 2026-07-23. May 2025. ↗
  8. Artificial Analysis. Artificial Analysis. 2026. URL : https://artificialanalysis.ai/. ↗
  9. Artificial Analysis Long Context Reasoning (AA-LCR). Artificial Analysis. 2026. URL : https://artificialanalysis.ai/evaluations/artificial-analysis-long-context-reasoning. ↗
  10. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. 2014. arXiv: 1409.0473 [cs.CL]. URL : https://arxiv.org/abs/1409.0473. ↗
  11. Chaithanya Bandi et al. MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers. 2026. arXiv: 2602.00933 [cs.SE]. — [arXiv:2602.00933](https://arxiv.org/abs/2602.00933) find
  12. Better MoE Model Inference with Warp Decode. Cursor. 2026. URL : https://cursor.com/blog/warp-decode (visited on 07/20/2026). ↗
  13. Liang Chen et al. BabyVision: Visual Reasoning Beyond Language. 2026. arXiv: 2601.06521 [cs.CV]. URL : https://arxiv.org/abs/2601.06521. — [arXiv:2601.06521](https://arxiv.org/abs/2601.06521) ↗
  14. Yutian Chen et al. FlashKDA: Flash Kimi Delta Attention. 2026. URL : https://github.com/MoonshotAI/FlashKDA. ↗
  15. Claude Code. Anthropic. 2026. URL : https://docs.anthropic.com/en/docs/claude-code. ↗
  16. Claude Fable 5. Anthropic. 2026. URL : https://www.anthropic.com/news/claude-fable-5-mythos-5. ↗
  17. Claude Opus 4.8. Anthropic. 2026. URL : https://www.anthropic.com/news/claude-opus-4-8. ↗
  18. Claude Sonnet 5. Anthropic. 2026. URL : https://www - cdn . anthropic . com / 283ef97c476cf442c91d9a37d5b214242a55bb92/Claude%20Sonnet%205%20System%20Card.pdf. find
  19. Claude Sonnet 5. Anthropic. 2026. URL : https://www.anthropic.com/news/claude-sonnet-5. ↗
  20. Codex. OpenAI. 2026. URL : https://github.com/openai/codex. ↗
  21. CorpFin v2. Vals AI. 2026. URL : https://www.vals.ai/benchmarks/corp_fin_v2. ↗
  22. cuBLAS. NVIDIA. 2026. URL : https://developer.nvidia.com/cublas. ↗
  23. Damai Dai et al. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. 2024. arXiv: 2401.06066 [cs.CL]. URL : https://arxiv.org/abs/2401.06066. ↗
  24. Tri Dao and Albert Gu. “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality”. In: CoRR abs/2405.21060 (2024). DOI : 10.48550/ARXIV.2405.21060. arXiv: 2405.21060. URL : https://doi.org/10.48550/arXiv.2405.21060. ↗
  25. Dao AI Lab. ReplaySSM: Cache SSM Inputs, Not State. https://tridao.me/blog/2026/replayssm/. June 2026. ↗
  26. Yann N. Dauphin et al. “Language Modeling with Gated Convolutional Networks”. In: Proceedings of the 34th International Conference on Machine Learning. Vol. 70. Proceedings of Machine Learning Research. PMLR, 2017, pp. 933–941. URL : https://proceedings.mlr.press/v70/dauphin17a.html. ↗
  27. Soham De et al. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models. 2024. arXiv: 2402.19427 [cs.LG]. URL : https://arxiv.org/abs/2402.19427. ↗
  28. DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. 2024. arXiv: 2405.04434 [cs.CL]. URL : https://arxiv.org/abs/2405.04434. ↗
  29. DeepSeek-AI. “DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence”. In: arXiv preprint arXiv:2606.19348 (2026). 35
  30. DeepSeek-AI et al. DeepSeek-V3 Technical Report. 2024. arXiv: 2412.19437 [cs.CL]. URL : https://arxiv.org/abs/2412.19437. ↗
  31. DeepSWE Benchmark. Datacurve. 2026. URL : https://deepswe.datacurve.ai/. ↗
  32. Venmugil Elango et al. LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts. 2026. arXiv: 2601.18089 [cs.LG]. URL : https://arxiv.org/abs/2601.18089. — [arXiv:2601.18089](https://arxiv.org/abs/2601.18089) ↗
  33. William Fedus, Barret Zoph, and Noam Shazeer. “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity”. In: Journal of Machine Learning Research 23.120 (2022), pp. 1–39. find
  34. Weiqi Feng et al. “Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation”. In: 2025 USENIX Annual Technical Conference (USENIX ATC 25). Boston, MA: USENIX Association, July 2025, pp. 161–177. ISBN : 978-1-939133-48-9. URL : https://www.usenix.org/conference/atc25/presenta… ↗
  35. Finance Agent v2. Vals AI. 2026. URL : https://www.vals.ai/benchmarks/fabv2. ↗
  36. FrontierSWE. 2026. URL : https://www.frontierswe.com/. ↗
  37. Chaoyou Fu et al. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. 2024. arXiv: 2405.21075 [cs.CV]. URL : https://arxiv.org/abs/2405.21075. ↗
  38. GLM-5.2. Z.ai. 2026. URL : https://z.ai/blog/glm-5.2. ↗
  39. GPT-5.5. OpenAI. 2026. URL : https://openai.com/index/introducing-gpt-5-5/. ↗
  40. GPT-5.6 Sol. OpenAI. 2026. URL : https://openai.com/index/previewing-gpt-5-6-sol/. ↗
  41. Daya Guo et al. “DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning”. In: Nature 645.8081 (2025), pp. 633–638. ISSN : 1476-4687. DOI : 10 . 1038 / s41586 - 025 - 09422 - z. URL : http : //dx.doi.org/10.1038/s41586-025-09422-z.
  42. Wentao Guo et al. SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations. 2025. arXiv: 2512. 14080 [cs.LG]. URL : https://arxiv.org/abs/2512.14080. ↗
  43. Harvey LAB: Legal Agent Benchmark. Harvey. 2026. URL : https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark. ↗
  44. Kaiming He et al. Deep Residual Learning for Image Recognition. 2015. arXiv: 1512.03385 [cs.CV]. URL : https://arxiv.org/abs/1512.03385. ↗
  45. Hermes Agent. Nous Research. 2026. URL : https://hermes-agent.nousresearch.com/docs/. ↗
  46. Jordan Hoffmann et al. Training Compute-Optimal Large Language Models. 2022. arXiv: 2203 . 15556 [cs.CL]. URL : https://arxiv.org/abs/2203.15556. ↗
  47. Shengding Hu, Yuge Tu, Xu Han, et al. “MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies”. In: (2024). arXiv: 2404.06395 [cs.CL].
  48. Ailin Huang, Ang Li, Aobo Kong, et al. “Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters”. In: arXiv preprint arXiv:2602.10604 (2026). — [arXiv:2602.10604](https://arxiv.org/abs/2602.10604) find
  49. Yanping Huang et al. “Gpipe: Efficient training of giant neural networks using pipeline parallelism”. In: Advances in neural information processing systems 32 (2019). find
  50. Benoit Jacob et al. “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference”. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018, pp. 2704–2713. find
  51. Sam Ade Jacobs et al. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. 2023. arXiv: 2309.14509 [cs.LG]. URL : https://arxiv.org/abs/2309.14509. ↗
  52. Peijie Jiang et al. PowLU: An Activation Function for Stable Pre-Training of LLMs. 2026. arXiv: 2605.25704 [cs.CL]. URL : https://arxiv.org/abs/2605.25704. — [arXiv:2605.25704](https://arxiv.org/abs/2605.25704) ↗
  53. JobBench: Aligning Agent Work with Human Will. July 24, 2026. URL : https://job-bench.github.io/. ↗
  54. Keller Jordan et al. Muon: An Optimizer for Hidden Layers in Neural Networks. 2024. URL : https://kellerjordan.github.io/posts/muon/. ↗
  55. Jared Kaplan et al. Scaling Laws for Neural Language Models. 2020. arXiv: 2001.08361 [cs.LG]. URL : https://arxiv.org/abs/2001.08361. ↗
  56. Angelos Katharopoulos et al. “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention”. In: Proceedings of ICML. Ed. by Hal Daumé III and Aarti Singh. PMLR, 2020, pp. 5156–5165. URL : https://proceedings.mlr.press/v119/katharopoulos20a.html. ↗
  57. Kimi CLI. Moonshot AI. 2026. URL : https://www.kimi.com/code. ↗
  58. Kimi Team. Attention Residuals. Preprint. 2026. find
  59. Kimi Team. Kimi K2: Open Agentic Intelligence. 2025. arXiv: 2507.20534 [cs.LG].
  60. Kimi Team. “Kimi K2.5: Visual Agentic Intelligence”. In: arXiv preprint arXiv:2602.02276 (2026).
  61. Kimi Team. Kimi K3: Open Frontier Intelligence. Moonshot AI. July 16, 2026. URL : https://www.kimi.com/blog/kimi-k3. ↗
  62. Kimi Team. “Kimi-vl technical report”. In: arXiv preprint arXiv:2504.07491 (2025).
  63. Kimi Team. PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models. 2026. arXiv: 2607.24957 [cs.CV]. URL : https://arxiv.org/abs/2607.24957. — [arXiv:2607.24957](https://arxiv.org/abs/2607.24957) ↗
  64. Kimi Team et al. Kimi Linear: An Expressive, Efficient Attention Architecture. 2025. arXiv: 2510.26692 [cs.CL].
  65. Chris Lattner et al. “MLIR: Scaling Compiler Infrastructure for Domain Specific Computation”. In: 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). 2021, pp. 2–14. DOI : 10.1109/CGO51591.2021.9370308. — [doi:10.1109/cgo51591.2021.9370308](https://doi.org/10.1109/cgo51591.2021.9370308) find
  66. Legal Research Bench. Vals AI. 2026. URL : https://www.vals.ai/benchmarks/legal_research. ↗
  67. Dmitry Lepikhin et al. “Gshard: Scaling giant models with conditional computation and automatic sharding”. In: arXiv preprint arXiv:2006.16668 (2020). find
  68. Mike Lewis et al. “BASE Layers: Simplifying Training of Large, Sparse Models”. In: Proceedings of ICML. 2021. find
  69. Huiba Li et al. “DADI: Block-Level Image Service for Agile and Elastic Application Deployment”. In: 2020 USENIX Annual Technical Conference (USENIX ATC). 2020, pp. 727–740. find
  70. Junlong Li et al. The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution. ICLR 2026. 2025. arXiv: 2510.25726 [cs.CL].
  71. Yuetai Li et al. “JobBench: Aligning Agent Work With Human Will”. In: (2026). arXiv: 2605.26329 [cs.AI]. find
  72. Yuhui Li et al. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. 2025. arXiv: 2503.01840 [cs.CL]. URL : https://arxiv.org/abs/2503.01840. — [arXiv:2503.01840](https://arxiv.org/abs/2503.01840) ↗
  73. Hao Liu, Matei Zaharia, and Pieter Abbeel. “Ring Attention with Blockwise Transformers for Near-Infinite Context”. In: (2023). arXiv: 2310.01889 [cs.CL]. URL : https://arxiv.org/abs/2310.01889. ↗
  74. Jingyuan Liu et al. Muon is Scalable for LLM Training. 2025. arXiv: 2502.16982 [cs.LG]. URL : https://arxiv.org/abs/2502.16982. ↗
  75. LMArena Leaderboard. LMArena. 2026. URL : https://lmarena.ai/leaderboard. ↗
  76. Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism. 2025. URL : https://thinkingmachines.ai/blog/on-policy-distillation/. ↗
  77. Bohan Lyu et al. “MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI”. In: (2026). arXiv: 2605.08678 [cs.LG]. find
  78. Eric Martin and Chris Cundy. “Parallelizing Linear Recurrent Neural Nets Over Sequence Length”. In: Proceedings of ICLR. 2018. URL : https://openreview.net/forum?id=HyUNwulC-. ↗
  79. Mike A Merrill et al. “Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces”. In: arXiv preprint arXiv:2601.11868 (2026). — [arXiv:2601.11868](https://arxiv.org/abs/2601.11868) find
  80. Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax. 2018. arXiv: 1805.02867 [cs.PF]. URL : https://arxiv.org/abs/1805.02867. ↗
  81. Nangate, Inc. Nangate 45nm Open Cell Library. https://si2.org/open - cell - library/. Version PDKv1_3_v2010_12. Donated to and distributed by the Silicon Integration Initiative (Si2). 2010. (Visited on 07/27/2026). ↗
  82. Deepak Narayanan et al. “Efficient large-scale language model training on gpu clusters using megatron-lm”. In: Proceedings of the international conference for high performance computing, networking, storage and analysis. 2021, pp. 1–15. find
  83. NCCL: The NVIDIA Collective Communications Library. NVIDIA. 2026. URL : https://developer.nvidia.com/nccl. ↗
  84. OpenAI. Introducing OpenAI o3 and o4-mini. Apr. 2025. URL : https://openai.com/index/introducing-o3-and-o4-mini/. ↗
  85. OpenAI. Learning to Reason with LLMs. https://openai.com/index/learning-to- reason- with-llms/. Accessed: 2026-07-23. 2024. ↗
  86. OpenAI Harmony Response Format. OpenAI. 2025. URL : https://github.com/openai/harmony. ↗
  87. OpenClaw. OpenClaw. 2026. URL : https://docs.openclaw.ai/. ↗
  88. Krista Opsahl-Ong et al. “OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning”. In: (2026). arXiv: 2603.08655. — [arXiv:2603.08655](https://arxiv.org/abs/2603.08655) find
  89. Linke Ouyang et al. OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations. 2025. arXiv: 2412.07626 [cs.CV]. URL : https://arxiv.org/abs/2412.07626. — [arXiv:2412.07626](https://arxiv.org/abs/2412.07626) ↗
  90. Adam Paszke et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. 2019. arXiv: 1912.01703 [cs.LG].
  91. Tejal Patwardhan et al. GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks. 2025. arXiv: 2510.04374 [cs.LG]. URL : https://arxiv.org/abs/2510.04374. ↗
  92. Bo Peng et al. RWKV-7 "Goose" with Expressive Dynamic State Evolution. 2025. arXiv: 2503.14456 [cs.CL].
  93. Bowen Peng et al. “Yarn: Efficient context window extension of large language models”. In: arXiv preprint arXiv:2309.00071 (2023). find
  94. Long Phan et al. Humanity’s Last Exam. 2025. arXiv: 2501.14249 [cs.LG]. URL : https://arxiv.org/abs/2501.14249. ↗
  95. PostTrainBench. 2026. URL : https://posttrainbench.com/. ↗
  96. ProgramBench. Vals AI. 2026. URL : https://www.vals.ai/benchmarks/programbench. ↗
  97. Ruoyu Qin et al. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. 2024. arXiv: 2407.00079 [cs.DC]. find
  98. Zhen Qin et al. HGRN2: Gated Linear RNNs with State Expansion. 2024. arXiv: 2404.07904 [cs.CL]. URL : https://arxiv.org/abs/2404.07904. — [arXiv:2404.07904](https://arxiv.org/abs/2404.07904) ↗
  99. Haiquan Qiu and Quanming Yao. “Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention”. In: International Conference on Learning Representations (ICLR). 2026. arXiv: 2510.04212 [cs.LG]. — [arXiv:2510.04212](https://arxiv.org/abs/2510.04212) find
  100. Zihan Qiu et al. Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free. 2025. arXiv: 2505.06708 [cs.CL]. — [arXiv:2505.06708](https://arxiv.org/abs/2505.06708) find
  101. Samyam Rajbhandari et al. “Zero: Memory optimizations toward training trillion parameter models”. In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE. 2020, pp. 1–16. find
  102. David Rein et al. “Gpqa: A graduate-level google-proof q&a benchmark”. In: First Conference on Language Modeling. 2024.
  103. Jonathan Roberts et al. ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models. 2025. arXiv: 2502.09696 [cs.CV]. URL : https://arxiv.org/abs/2502.09696. — [arXiv:2502.09696](https://arxiv.org/abs/2502.09696) ↗
  104. Bita Darvish Rouhani et al. “Microscaling Data Formats for Deep Learning”. In: arXiv preprint arXiv:2310.10537 (2023). arXiv: 2310.10537 [cs.LG]. find
  105. Alexander Samarin et al. LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding. 2026. arXiv: 2602.23881 [cs.LG]. URL : https://arxiv.org/abs/2602.23881. — [arXiv:2602.23881](https://arxiv.org/abs/2602.23881) ↗
  106. Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. “Linear Transformers Are Secretly Fast Weight Programmers”. In: Proceedings of ICML. Ed. by Marina Meila and Tong Zhang. PMLR, 2021, pp. 9355–9366. URL : https://proceedings.mlr.press/v139/schlag21a.html. ↗
  107. Manasi Sharma et al. “ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents”. In: The Fourteenth International Conference on Learning Representations. 2026. URL : https : //openreview.net/forum?id=ErnvfmSX0P. find
  108. Noam Shazeer. GLU Variants Improve Transformer. 2020. arXiv: 2002 . 05202 [cs.LG]. URL : https : //arxiv.org/abs/2002.05202. find
  109. Daniel Shepard and Robin Salimans. “AutomationBench”. In: (2026). arXiv: 2604.18934 [cs.AI]. find
  110. Kean Shi et al. SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows? 2026. arXiv: 2605.15777 [cs.AI]. URL : https://arxiv.org/abs/2605.15777. — [arXiv:2605.15777](https://arxiv.org/abs/2605.15777) ↗
  111. Benjamin F. Spector et al. “ThunderKittens: Simple, Fast, and Adorable Kernels”. In: The Thirteenth International Conference on Learning Representations. 2025. URL : https://openreview.net/forum?id=0fJfVOSUra. ↗
  112. Jianlin Su. Muon Optimizer Guide: Quick Start and Key Details. Blog post (in Chinese). Nov. 2025. URL : https://kexue.fm/archives/11416. ↗
  113. Jianlin Su. Travels in MoE: 6. Promoting Load Balance via Optimal Assignment. Blog post (in Chinese). Feb. 2026. URL : https://spaces.ac.cn/archives/11619. ↗
  114. Hanchi Sun et al. Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing. 2026. arXiv: 2603.11535 [cs.AI]. 38 — [arXiv:2603.11535](https://arxiv.org/abs/2603.11535) find
  115. Weigao Sun et al. “LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid”. In: (2025). arXiv: 2502.07563 [cs.LG]. URL : https://arxiv.org/abs/2502.07563. ↗
  116. Weigao Sun et al. “Linear Attention Sequence Parallelism”. In: (2024). arXiv: 2404.02882 [cs.LG]. URL : https://arxiv.org/abs/2404.02882. ↗
  117. Yiyou Sun et al. Agents’ Last Exam. 2026. arXiv: 2606.05405 [cs.AI]. URL : https://arxiv.org/abs/2606.05405. ↗
  118. Yuan Sun. Binary-Integer-Programming Based Algorithm for Expert Load Balancing in Mixture-of-Experts Models. 2025. arXiv: 2502.15451 [cs.LG]. — [arXiv:2502.15451](https://arxiv.org/abs/2502.15451) find
  119. SWE Marathon. 2026. URL : https://www.swe-marathon.org/. ↗
  120. Kimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs. 2025. arXiv: 2501.12599 [cs.AI]. URL : https://arxiv.org/abs/2501.12599. The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution. July 24, 2026. URL : https://toolathlon.xyz/introduction.… ↗
  121. [141] Songlin Yang and Yu Zhang. FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism. Jan. 2024. URL : https://github.com/fla-org/flash-linear-attention. [142] Songlin Yang et al. “Gated Linear Attention Transformers with Hardware-Efficient Training”. In:… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kimi-team-2026-kimi-k3-open-frontier/