Elamur Research⌕
☰
home / papers / 2025 / Kimi-VL Technical Report

Kimi-VL Technical Report

Paper · 2025 · Kimi Team, Angang Du, Bohong Yin et al.

Summary

Open-source vision-language models have trailed text-only LLMs in efficiency, context length and extended reasoning. Earlier MoE-based VLMs relied on fixed-size image encoders and had short contexts or weak fine-grained perception. Kimi-VL addresses this by pairing the Moonlight MoE language model (16B total parameters, 2.8B active) with MoonViT, a 400M native-resolution encoder derived from SigLIP-SO-400M. A two-layer MLP projector connects the two. The model was trained with an enhanced Muon optimizer across four stages totalling 4.4T tokens, ending with a 128K context window. Long chain-of-thought fine-tuning and reinforcement learning then produced thinking variants. The latest, Kimi-VL-A3B-Thinking-2506, reasons better while using shorter outputs and keeps the perception, video, long-document and GUI-agent abilities of the instruct model. Despite its small active size, it rivals GPT-4o-mini, Qwen2.5-VL-7B and Gemma-3-12B-IT, and sometimes GPT-4o. Code and weights are public, giving a cheap, capable multimodal reasoning baseline.

Key points

  • Architecture. MoonViT handles images at native resolution: it packs image patches into 1D sequences in the style of NaViT. Position is encoded by combining interpolated SigLIP embeddings with 2D RoPE. In the 2506 release, MoonViT accepts up to 3.2M pixels per image, four times the original limit.
  • Training. The language model starts from a Moonlight checkpoint pretrained on 5.2T text tokens. Multimodal stages then add 4.4T tokens, and the RoPE base is raised from 50,000 to 800,000 to support long context.
  • Kimi-VL-A3B results.
    • LongVideoBench: 64.5
    • MMLongBench-Doc: 35.1
    • InfoVQA: 83.2
    • ScreenSpot-Pro: 34.5
  • Thinking-2506 reasoning results. Output length drops by about 20% compared with the earlier thinking model.
    • MMMU: 64.0
    • MMMU-Pro: 46.3
    • MathVision: 56.9
    • MathVista: 80.1
    • VideoMMMU: 65.2
  • Thinking-2506 perception and agent results.
    • V*: 83.2
    • ScreenSpot-Pro: 52.8
    • OSWorld-G: 52.5
    • MMLongBench-Doc: 42.1, roughly matching GPT-4o
  • Limitations noted by the authors. The model's small size limits specialized tasks, and long-context ability is still restricted for very long inputs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 510 worksVision & language · 269 works

Cite

Cite as Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang, Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Mengfan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng, Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song, Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinhao Li, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang, Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yuhao Dong, Yulun Du, Yuxin Wu, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Zijia Zhao, Ziwei Chen and Zongyu Lin. 2025. Kimi-VL Technical Report. arXiv:2504.07491.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2504.07491 · arXiv 2504.07491

@misc{team2025kimi,
  title = {{Kimi-VL Technical Report}},
  author = {Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang, Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Mengfan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng, Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song, Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinhao Li, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang, Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yuhao Dong, Yulun Du, Yuxin Wu, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Zijia Zhao, Ziwei Chen and Zongyu Lin},
  year = {2025},
  doi = {10.48550/arxiv.2504.07491},
  eprint = {2504.07491},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.07491}
}

References

62 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amazon Web Services. Amazon Simple Storage Service (Amazon S3). Web. Available at: https://aws.amazon.com/s3/. 2023. URL : https://aws.amazon.com/s3/ (visited on 12/15/2023). ↗
  2. Bai, Shuai et al. Qwen2.5-VL Technical Report. 2025. arXiv: 2502.13923 [cs.CV]. URL : https://arxiv.org/abs/2502.13923. ↗
  3. Bonatti, Rogerio et al. Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale. 2024. arXiv: 2409.08264 [cs.AI]. URL : https://arxiv.org/abs/2409.08264. — [arXiv:2409.08264](https://arxiv.org/abs/2409.08264) ↗
  4. Chen, Lin et al. “Are We on the Right Way for Evaluating Large Vision-Language Models?” In: arXiv preprint arXiv:2403.20330 (2024).
  5. Chen, Tianqi et al. Training Deep Nets with Sublinear Memory Cost. 2016. arXiv: 1604 . 06174 [cs.LG]. URL : https://arxiv.org/abs/1604.06174. ↗
  6. Cheng, Kanzhi et al. “Seeclick: Harnessing gui grounding for advanced visual gui agents”. In: arXiv preprint arXiv:2401.10935 (2024).
  7. Dao, Tri et al. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. 2022. arXiv: 2205.14135 [cs.LG]. URL : https://arxiv.org/abs/2205.14135. ↗
  8. DeepSeek-AI, Daya Guo, et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. 2025. arXiv: 2501.12948 [cs.CL]. URL : https://arxiv.org/abs/2501.12948. ↗
  9. DeepSeek-AI, Aixin Liu, et al. DeepSeek-V3 Technical Report. 2025. arXiv: 2412.19437 [cs.CL]. URL : https://arxiv.org/abs/2412.19437. ↗
  10. Dehghani, Mostafa et al. Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution. 2023. arXiv: 2307.06304 [cs.CV]. URL : https://arxiv.org/abs/2307.06304. — [arXiv:2307.06304](https://arxiv.org/abs/2307.06304) ↗
  11. Fedus, William, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. 2022. arXiv: 2101.03961 [cs.LG]. URL : https://arxiv.org/abs/2101.03961. ↗
  12. Fu, Chaoyou et al. “Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis”. In: arXiv:2405.21075 (2024).
  13. Fu, Xingyu et al. “Blink: Multimodal large language models can see but not perceive”. In: European Conference on Computer Vision. Springer. 2024, pp. 148–166. find
  14. Gadre, Samir Yitzhak et al. “Datacomp: In search of the next generation of multimodal datasets”. In: Advances in Neural Information Processing Systems 36 (2024). find
  15. Grauman, Kristen et al. “Ego4d: Around the world in 3,000 hours of egocentric video”. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022, pp. 18995–19012. find
  16. Guo, Jarvis et al. MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale. 2024. arXiv: 2412.05237 [cs.CL]. URL : https://arxiv.org/abs/2412.05237. ↗
  17. Hu, Kairui et al. “Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos”. In: arXiv preprint arXiv:2501.13826 (2025). — [arXiv:2501.13826](https://arxiv.org/abs/2501.13826) find
  18. Huang, Yanping et al. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. 2019. arXiv: 1811.06965 [cs.CV]. URL : https://arxiv.org/abs/1811.06965. ↗
  19. Jacobs, Sam Ade et al. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. 2023. arXiv: 2309.14509 [cs.LG]. URL : https://arxiv.org/abs/2309.14509. ↗
  20. Jordan, Keller et al. Muon: An optimizer for hidden layers in neural networks. 2024. URL : https://kellerjordan.github.io/posts/muon/. ↗
  21. Kembhavi, Aniruddha et al. “A diagram is worth a dozen images”. In: European conference on computer vision. Springer. 2016, pp. 235–251.
  22. Korthikanti, Vijay et al. Reducing Activation Recomputation in Large Transformer Models. 2022. arXiv: 2205.05198 [cs.LG]. URL : https://arxiv.org/abs/2205.05198. ↗
  23. Laurençon, Hugo et al. “Obelics: An open web-scale filtered dataset of interleaved image-text documents”. In: Advances in Neural Information Processing Systems 36 (2024).
  24. Li, Bo et al. LLaVA-OneVision: Easy Visual Task Transfer. 2024. arXiv: 2408 . 03326 [cs.CV]. URL : https : //arxiv.org/abs/2408.03326.
  25. Li, Dongxu et al. Aria: An Open Multimodal Native Mixture-of-Experts Model. 2024. arXiv: 2410.05993 [cs.CV]. URL : https://arxiv.org/abs/2410.05993. — [arXiv:2410.05993](https://arxiv.org/abs/2410.05993) ↗
  26. Li, Kaixin et al. “ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use”. In: Workshop on Reasoning and Planning for Large Language Models. 2025. find
  27. Li, Shen et al. PyTorch Distributed: Experiences on Accelerating Data Parallel Training. 2020. arXiv: 2006.15704 [cs.DC]. URL : https://arxiv.org/abs/2006.15704. ↗
  28. Liu, Hao, Matei Zaharia, and Pieter Abbeel. Ring Attention with Blockwise Transformers for Near-Infinite Context. 2023. arXiv: 2310.01889 [cs.CL]. URL : https://arxiv.org/abs/2310.01889. ↗
  29. Liu, Jingyuan et al. “Muon is Scalable for LLM Training”. In: arXiv preprint arXiv:2502.16982 (2025). – “Muon is Scalable for LLM Training”. In: arXiv preprint arXiv:2502.16982 (2025). find
  30. Liu, Yuan et al. “MMBench: Is Your Multi-modal Model an All-around Player?” In: arXiv:2307.06281 (2023).
  31. Liu, Yuliang et al. “On the hidden mystery of ocr in large multimodal models”. In: arXiv e-prints (2023), arXiv–2305. find
  32. Lu, Pan et al. “Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts”. In: arXiv preprint arXiv:2310.02255 (2023).
  33. Mangalam, Karttikeya, Raiymbek Akshulakov, and Jitendra Malik. “Egoschema: A diagnostic benchmark for very long-form video language understanding”. In: Advances in Neural Information Processing Systems 36 (2023), pp. 46212–46244. find
  34. Mathew, Minesh et al. “Infographicvqa”. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022, pp. 1697–1706. find
  35. Narayanan, Deepak et al. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. 2021. arXiv: 2104.04473 [cs.CL]. URL : https://arxiv.org/abs/2104.04473. ↗
  36. OpenAI. “Learning to reason with LLMs”. In: (2024). URL : https://openai.com/index/learning-to-reason-with-llms/. ↗
  37. OpenAI et al. GPT-4o System Card. 2024. arXiv: 2410.21276 [cs.CL]. URL : https://arxiv.org/abs/2410.21276. ↗
  38. Rajbhandari, Samyam et al. “Zero: Memory optimizations toward training trillion parameter models”. In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE. 2020, pp. 1– 16. find
  39. Schuhmann, Christoph et al. “Laion-5b: An open large-scale dataset for training next generation image-text models”. In: Advances in Neural Information Processing Systems 35 (2022), pp. 25278–25294.
  40. Shangguan, Ziyao et al. “TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models”. In: International Conference on Learning Representations. 2025. URL : https://openreview.net/forum?id=fCi4o83Mfs. ↗
  41. Su, Dan et al. “Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset”. In: arXiv preprint arXiv:2412.02595 (2024). find
  42. Su, Jianlin et al. RoFormer: Enhanced Transformer with Rotary Position Embedding. 2023. arXiv: 2104 . 09864 [cs.CL]. URL : https://arxiv.org/abs/2104.09864. ↗
  43. Team, Gemini et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. 2024. arXiv: 2403.05530 [cs.CL]. URL : https://arxiv.org/abs/2403.05530. ↗
  44. Team, Gemma et al. Gemma 3 Technical Report. 2025. arXiv: 2503.19786 [cs.CL]. URL : https://arxiv.org/abs/2503.19786. ↗
  45. Team, Kimi et al. “Kimi k1. 5: Scaling reinforcement learning with llms”. In: arXiv preprint arXiv:2501.12599 (2025).
  46. Tong, Shengbang et al. Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. 2024. arXiv: 2406.16860 [cs.CV]. URL : https://arxiv.org/abs/2406.16860. ↗
  47. Wang, Ke et al. “Measuring multimodal mathematical reasoning with math-vision dataset”. In: arXiv preprint arXiv:2402.14804 (2024).
  48. Wei, Haoran et al. “General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model”. In: arXiv preprint arXiv:2409.01704 (2024). — [arXiv:2409.01704](https://arxiv.org/abs/2409.01704) find
  49. Wu, Haoning et al. “Longvideobench: A benchmark for long-context interleaved video-language understanding”. In: Advances in Neural Information Processing Systems 37 (2024), pp. 28828–28857. find
  50. Wu, Zhiyong et al. “Os-atlas: A foundation action model for generalist gui agents”. In: arXiv preprint arXiv:2410.23218 (2024). find
  51. Wu, Zhiyu et al. DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding. 2024. arXiv: 2412.10302 [cs.CV]. URL : https://arxiv.org/abs/2412.10302. x.ai. “Grok-1.5 Vision Preview”. In: (2024). URL : https://x.ai/news/grok-1.5v. ↗
  52. Xie, Tianbao et al. “Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments”. In: Advances in Neural Information Processing Systems 37 (2024), pp. 52040–52094.
  53. Xu, Yiheng et al. Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction. 2024. arXiv: 2412.04454 [cs.CL]. — [arXiv:2412.04454](https://arxiv.org/abs/2412.04454) find
  54. Yang, Jihan et al. “Thinking in space: How multimodal large language models see, remember, and recall spaces”. In: arXiv preprint arXiv:2412.14171 (2024). find
  55. Yu, Jiahui et al. CoCa: Contrastive Captioners are Image-Text Foundation Models. 2022. arXiv: 2205.01917 [cs.CV]. URL : https://arxiv.org/abs/2205.01917. ↗
  56. Yu, Weihao et al. “Mm-vet: Evaluating large multimodal models for integrated capabilities”. In: International conference on machine learning. PMLR. 2024.
  57. Yue, Xiang, Yuansheng Ni, et al. “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 9556–9567.
  58. Yue, Xiang, Xingwei Qu, et al. “Mammoth: Building math generalist models through hybrid instruction tuning”. In: arXiv preprint arXiv:2309.05653 (2023).
  59. Zhai, Xiaohua et al. Sigmoid Loss for Language Image Pre-Training. 2023. arXiv: 2303 . 15343 [cs.CV]. URL : https://arxiv.org/abs/2303.15343. ↗
  60. Zhao, Yilun et al. “MMVU: Measuring Expert-Level Multi-Discipline Video Understanding”. In: arXiv preprint arXiv:2501.12380 (2025). — [arXiv:2501.12380](https://arxiv.org/abs/2501.12380) find
  61. Zhou, Junjie et al. “Mlvu: A comprehensive benchmark for multi-task long video understanding”. In: arXiv preprint arXiv:2406.04264 (2024). find
  62. Zhu, Wanrong et al. “Multimodal c4: An open, billion-scale corpus of images interleaved with text”. In: Advances in Neural Information Processing Systems 36 (2024). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-vl-technical-report/