Elamur Research⌕
☰
home / papers / 2026 / MiMo-V2-Flash Technical Report

MiMo-V2-Flash Technical Report

Paper · 2026 · Xiaomi LLM-Core Team; Bangjun Xiao; Bingquan Xia; Bo Yang; Bofei Gao; Bowen Shen; Chen Zhang; Chenhong He; Chiheng Lou; Fuli Luo; Gang Wang; Gang Xie; Hailin Zhang; Hanglong Lv; Hanyu Li; Heyu Chen; Hongshen Xu; Houbin Zhang; Huaqiu Liu; Jiangshan Duo; Jianyu Wei; Jiebao Xiao; Jinhao Dong; Jun Shi; Junhao Hu; Kainan Bao; Kang Zhou; Lei Li; Liang Zhao; Linghao Zhang; Peidian Li; Qianli Chen; Shaohui Liu; Shihua Yu; Shijie Cao; Shimao Chen; Shouqiu Yu; Shuo Liu; Tianling Zhou; Weijiang Su; Weikun Wang; Wenhan Ma; Xiangwei Deng; Bohan Mao; Bowen Ye; Can Cai; Chenghua Wang; Chengxuan Zhu; Chong Ma; Chun Chen; Chunan Li; Dawei Zhu; Deshan Xiao; Dong Zhang; Duo Zhang; Fangyue Liu; Feiyu Yang; Fengyuan Shi; Guoan Wang; Hao Tian; Hao Wu; Heng Qu; Hongfei Yi; Hongxu An; Hongyi Guan; Xing Zhang; Yifan Song; Yihan Yan; Yihao Zhao; Yingchun Lai; Yizhao Gao; Yu Cheng; Yuanyuan Tian; Yudong Wang; Zhen Tang; Zhengju Tang; Zhengtao Wen; Zhichao Song; Zhixian Zheng; Zihan Jiang; Jian Wen; Jiarui Sun; Jiawei Li; Jinlong Xue; Jun Xia; Kai Fang; Menghang Zhu; Nuo Chen; Qian Tu; Qihao Zhang; Qiying Wang; Rang Li; Rui Ma; Shaolei Zhang; Shengfan Wang; Shicheng Li; Shuhao Gu; Shuhuai Ren; Sirui Deng; Tao Guo; Tianyang Lu; Weiji Zhuang; Weikang Zhang; Weimin Xiong; Wenshan Huang; Wenyu Yang; Xin Zhang; Xing Yong; Xu Wang; Xueyang Xie; Yilin Jiang; Yixin Yang; Yongzhe He; Yu Tu; Yuanliang Dong; Yuchen Liu; Yue Ma; Yue Yu; Yuxing Xiang; Zhaojun Huang; Zhenru Lin; Zhipeng Xu; Zhiyang Chen; Zhonghua Deng; Zihan Zhang; and Zihao Yue

Summary

MiMo-V2-Flash, from Xiaomi's LLM-Core Team, addresses a core problem for reasoning and agentic LLMs: long contexts must be handled both cheaply and well. The model is a sparse Mixture-of-Experts with 309B parameters, of which only 15B are active per token. Most layers use Sliding Window Attention with a very small 128-token window, interleaved with global attention at a 5:1 ratio. A learnable attention sink bias keeps long-context quality from degrading despite this aggressive setup. Pre-training covers 27T tokens in FP8 mixed precision with Multi-Token Prediction (MTP), starting at a 32K context that is later extended to 256K. Post-training introduces Multi-Teacher On-Policy Distillation (MOPD). After general SFT, domain-specific teachers are trained via RL or SFT. The student then learns from their dense token-level signals combined with verifiable outcome rewards. The resulting model is competitive with DeepSeek-V3.2 and Kimi-K2 while being much smaller. It leads open models on software-engineering benchmarks and decodes faster by reusing its MTP layers for speculative decoding. The weights are openly released.

Key points

  • Architecture: 48 layers (39 SWA, 9 global attention), arranged as 8 hybrid blocks. Each MoE layer has 256 experts with 8 active and no shared experts. The design cuts KV-cache and attention cost for long inputs by nearly 6×.
  • The base model, MiMo-V2-Flash-Base, achieves near-perfect long-context retrieval from 32K to 256K. It also degrades only slightly on GSM-Infinite when moving from 16K to 128K.
  • The post-trained model scores 73.4% on SWE-Bench Verified and 71.7% on SWE-Bench Multilingual. It also beats larger full-attention models on LongBench V2 and MRCR.
  • Each MTP block is lightweight (0.33B parameters, dense FFN with SWA). With three MTP layers, acceptance length reaches about 3.6 tokens, giving up to 2.6× faster decoding. Acceptance length correlates closely with next-token entropy (fit R² = 0.995).
  • The model and the 3-layer MTP weights are published at github.com/XiaomiMiMo/MiMo-V2-Flash. The authors acknowledge that a gap remains to the strongest closed-weight models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Xiaomi LLM-Core Team; Bangjun Xiao; Bingquan Xia; Bo Yang; Bofei Gao; Bowen Shen; Chen Zhang; Chenhong He; Chiheng Lou; Fuli Luo; Gang Wang; Gang Xie; Hailin Zhang; Hanglong Lv; Hanyu Li; Heyu Chen; Hongshen Xu; Houbin Zhang; Huaqiu Liu; Jiangshan Duo; Jianyu Wei; Jiebao Xiao; Jinhao Dong; Jun Shi; Junhao Hu; Kainan Bao; Kang Zhou; Lei Li; Liang Zhao; Linghao Zhang; Peidian Li; Qianli Chen; Shaohui Liu; Shihua Yu; Shijie Cao; Shimao Chen; Shouqiu Yu; Shuo Liu; Tianling Zhou; Weijiang Su; Weikun Wang; Wenhan Ma; Xiangwei Deng; Bohan Mao; Bowen Ye; Can Cai; Chenghua Wang; Chengxuan Zhu; Chong Ma; Chun Chen; Chunan Li; Dawei Zhu; Deshan Xiao; Dong Zhang; Duo Zhang; Fangyue Liu; Feiyu Yang; Fengyuan Shi; Guoan Wang; Hao Tian; Hao Wu; Heng Qu; Hongfei Yi; Hongxu An; Hongyi Guan; Xing Zhang; Yifan Song; Yihan Yan; Yihao Zhao; Yingchun Lai; Yizhao Gao; Yu Cheng; Yuanyuan Tian; Yudong Wang; Zhen Tang; Zhengju Tang; Zhengtao Wen; Zhichao Song; Zhixian Zheng; Zihan Jiang; Jian Wen; Jiarui Sun; Jiawei Li; Jinlong Xue; Jun Xia; Kai Fang; Menghang Zhu; Nuo Chen; Qian Tu; Qihao Zhang; Qiying Wang; Rang Li; Rui Ma; Shaolei Zhang; Shengfan Wang; Shicheng Li; Shuhao Gu; Shuhuai Ren; Sirui Deng; Tao Guo; Tianyang Lu; Weiji Zhuang; Weikang Zhang; Weimin Xiong; Wenshan Huang; Wenyu Yang; Xin Zhang; Xing Yong; Xu Wang; Xueyang Xie; Yilin Jiang; Yixin Yang; Yongzhe He; Yu Tu; Yuanliang Dong; Yuchen Liu; Yue Ma; Yue Yu; Yuxing Xiang; Zhaojun Huang; Zhenru Lin; Zhipeng Xu; Zhiyang Chen; Zhonghua Deng; Zihan Zhang; and Zihao Yue. 2026. MiMo-V2-Flash Technical Report. arXiv:2601.02780.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2601.02780 · arXiv 2601.02780

@misc{zhang2026mimo,
  title = {{MiMo-V2-Flash Technical Report}},
  author = {Xiaomi LLM-Core Team; Bangjun Xiao; Bingquan Xia; Bo Yang; Bofei Gao; Bowen Shen; Chen Zhang; Chenhong He; Chiheng Lou; Fuli Luo; Gang Wang; Gang Xie; Hailin Zhang; Hanglong Lv; Hanyu Li; Heyu Chen; Hongshen Xu; Houbin Zhang; Huaqiu Liu; Jiangshan Duo; Jianyu Wei; Jiebao Xiao; Jinhao Dong; Jun Shi; Junhao Hu; Kainan Bao; Kang Zhou; Lei Li; Liang Zhao; Linghao Zhang; Peidian Li; Qianli Chen; Shaohui Liu; Shihua Yu; Shijie Cao; Shimao Chen; Shouqiu Yu; Shuo Liu; Tianling Zhou; Weijiang Su; Weikun Wang; Wenhan Ma; Xiangwei Deng; Bohan Mao; Bowen Ye; Can Cai; Chenghua Wang; Chengxuan Zhu; Chong Ma; Chun Chen; Chunan Li; Dawei Zhu; Deshan Xiao; Dong Zhang; Duo Zhang; Fangyue Liu; Feiyu Yang; Fengyuan Shi; Guoan Wang; Hao Tian; Hao Wu; Heng Qu; Hongfei Yi; Hongxu An; Hongyi Guan; Xing Zhang; Yifan Song; Yihan Yan; Yihao Zhao; Yingchun Lai; Yizhao Gao; Yu Cheng; Yuanyuan Tian; Yudong Wang; Zhen Tang; Zhengju Tang; Zhengtao Wen; Zhichao Song; Zhixian Zheng; Zihan Jiang; Jian Wen; Jiarui Sun; Jiawei Li; Jinlong Xue; Jun Xia; Kai Fang; Menghang Zhu; Nuo Chen; Qian Tu; Qihao Zhang; Qiying Wang; Rang Li; Rui Ma; Shaolei Zhang; Shengfan Wang; Shicheng Li; Shuhao Gu; Shuhuai Ren; Sirui Deng; Tao Guo; Tianyang Lu; Weiji Zhuang; Weikang Zhang; Weimin Xiong; Wenshan Huang; Wenyu Yang; Xin Zhang; Xing Yong; Xu Wang; Xueyang Xie; Yilin Jiang; Yixin Yang; Yongzhe He; Yu Tu; Yuanliang Dong; Yuchen Liu; Yue Ma; Yue Yu; Yuxing Xiang; Zhaojun Huang; Zhenru Lin; Zhipeng Xu; Zhiyang Chen; Zhonghua Deng; Zihan Zhang; and Zihao Yue},
  year = {2026},
  doi = {10.48550/arxiv.2601.02780},
  eprint = {2601.02780},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2601.02780}
}

References

83 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Agarwal, N. Vieillard, Y. Zhou, P. Stańczyk, S. Ramos, M. Geist, and O. Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, 2023. URL https://api.semanticscholar.org/CorpusID: 263610088 . ↗
  2. S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, E. Arbus, R. K. Arora, Y. Bai, B. Baker, H. Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025.
  3. J. Ainslie, J. Lee-Thorp, M. De Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023.
  4. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. ArXiv preprint, abs/2108.07732, 2021. URL https://arxiv.org/abs/2108.07732 . ↗
  5. Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. Acceptance Length 3.2 3.4 3.6 3.0 3.8 1.99× 1.95× 2.11× 2.13× 1.94× 2.12× 2.08× 2.25× 2.28× 2.07× 2.25× 2.21× 2.39×…
  6. M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev. Matharena: Evaluating llms on uncontaminated math competitions. arXiv preprint arXiv:2505.23281, 2025. find
  7. V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan. 𝜏 2 -bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025. find
  8. I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
  9. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  10. F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, et al. Multipl-e: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.08227, 2022. find
  11. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv preprint, abs/1803.05457, 2018. URL https://arxiv.org/abs/1803.05457 . ↗
  12. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168 . ↗
  13. X. Du, Y. Yao, K. Ma, B. Wang, T. Zheng, K. Zhu, M. Liu, Y. Liang, X. Jin, Z. Wei, et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines. ArXiv preprint, abs/2502.14739, 2025. URL https://arxiv.org/abs/2502.14739 . ↗
  14. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa…
  15. W. Fu, J. Gao, X. Shen, C. Zhu, Z. Mei, C. He, S. Xu, G. Wei, J. Mei, J. Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24298, 2025. find
  16. W. Gao, Y. Zhao, D. An, T. Wu, L. Cao, S. Xiong, J. Huang, W. Wang, S. Yang, W. Su, et al. Rollpacker: Mitigating long-tail rollouts for fast, synchronous rl post-training. arXiv preprint arXiv:2509.21009, 2025. find
  17. A. P. Gema, J. O. J. Leang, G. Hong, A. Devoto, A. C. M. Mancino, R. Saxena, X. He, Y. Zhao, X. Du, M. R. G. Madani, et al. Are we done with mmlu? ArXiv preprint, abs/2406.04127, 2024. URL https://arxiv.org/abs/2406.04127 . ↗
  18. Gemma Team. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. find
  19. Gemma Team. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. find
  20. F. Gloeckle, B. Y. Idrissi, B. Rozière, D. Lopez-Paz, and G. Synnaeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404.19737, 2024. find
  21. Google DeepMind. Gemini 3 pro model card. https://storage.googleapis.com/deepmin d-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf , Nov. 2025. ↗
  22. A. Gu, B. Rozière, H. J. Leather, A. Solar-Lezama, G. Synnaeve, and S. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024.
  23. OpenReview.net, 2024a. URL https://openreview.net/forum?id=Ffpg52swvg . ↗
  24. X. Gu, T. Pang, C. Du, Q. Liu, F. Zhang, C. Du, Y. Wang, and M. Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410.10781, 2024b. — [arXiv:2410.10781](https://arxiv.org/abs/2410.10781) find
  25. Y. Gu, L. Dong, F. Wei, and M. Huang. Minillm: Knowledge distillation of large language models. In Proceedings of ICLR, 2024c. find
  26. H. He and T. M. Lab. Defeating nondeterminism in llm inference. Thinking Machines Lab: Connectionism, 2025. doi: 1 0 . 6 4 4 3 4 / t m l . 2 0 2 5 0 9 1 0. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/. ↗
  27. Y. He, S. Li, J. Liu, Y. Tan, W. Wang, H. Huang, X. Bu, H. Guo, C. Hu, B. Zheng, et al. Chinese simpleqa: A chinese factuality evaluation for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 19182–19208,… find
  28. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
  29. OpenReview.net, 2021a. URL https://openreview.net/forum?id=d7KBjmI3GmQ . ↗
  30. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv preprint, abs/2103.03874, 2021b. URL https://arxiv.org/abs/2103.03874 . ↗
  31. C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024.
  32. Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, Y. Fu, M. Sun, and J. He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural I…
  33. N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. ArXiv preprint, abs/2403.07974, 2024. URL https://arxiv.org/abs/2403.07974 . ↗
  34. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
  35. OpenReview.net, 2024a. URL https://openreview.net/forum?id=VTF8yNQM66 . ↗
  36. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024b. URL https://openreview.net/forum?id=VTF8yNQM 66 . ↗
  37. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In R. Barzilay and M.-Y. Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601…
  38. Kimi Team. Kimi linear: An expressive, efficient attention architecture. arXiv:2510.26692, 2025a. find
  39. Kimi Team. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025b.
  40. Kimi Team. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025c.
  41. A. Li, B. Gong, B. Yang, B. Shan, C. Liu, C. Zhu, C. Zhang, C. Guo, D. Chen, D. Li, et al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.08313, 2025. find
  42. H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. ArXiv preprint, abs/2306.09212, 2023. URL https://arxiv.org/abs/2306.09212 . ↗
  43. T. Li, W.-L. Chiang, E. Frick, L. Dunlap, B. Zhu, J. E. Gonzalez, and I. Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL https://lmsys.org/blog/2024-04-19-arena-hard/ . ↗
  44. A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. find
  45. A. Liu, A. Mei, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025.
  46. J. Liu, C. S. Xia, Y. Wang, and L. Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual…
  47. K. Lu and T. M. Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation. — [doi:10.64434/tml.20251026](https://doi.org/10.64434/tml.20251026) ↗
  48. W. Ma, H. Zhang, L. Zhao, Y. Song, Y. Wang, Z. Sui, and F. Luo. Stabilizing moe reinforcement learning by aligning training and inference routers. arXiv preprint arXiv:2510.11370, 2025. — [arXiv:2510.11370](https://arxiv.org/abs/2510.11370) find
  49. MAA. American invitational mathematics examination - aime. In American Invitational Mathematics Examination - AIME, 2024. URL https://maa.org/math-competition s/american-invitational-mathematics-examination-aime . arXiv preprint A. Modarressi, H. Deilamsalehy, F. Dernoncourt, T. Bui, R. A. Rossi, S.… — [arXiv:2502.05167](https://arxiv.org/abs/2502.05167) ↗
  50. P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, et al. Ray: A distributed framework for emerging {AI} applications. In 13th USENIX symposium on operating systems design and implementation (OSDI 18), pages 561–577, 2018. find
  51. OpenAI. Introducing simpleqa. https://openai.com/index/introducing-simpleqa/ , 2024. ↗
  52. L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al. Humanity’s last exam. arXiv preprint arXiv:2501.14249, 2025.
  53. Z. Qiu, Z. Wang, B. Zheng, Z. Huang, K. Wen, S. Yang, R. Men, L. Yu, F. Huang, S. Huang, et al. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. arXiv preprint arXiv:2505.06708, 2025. — [arXiv:2505.06708](https://arxiv.org/abs/2505.06708) find
  54. Qwen Team. Qwen3-next: Towards ultimate training & inference efficiency. https://qwen.a i/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.lates t-advancements-list , Sept. 2025. ↗
  55. RadixArk Team. Introducing miles — rl framework to fire up large-scale moe training. https://lmsys.org/blog/2025-11-19-miles/ , Nov. 2025. ↗
  56. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
  57. A. Romanou, N. Foroutan, A. Sotnikova, Z. Chen, S. H. Nelaturu, S. Singh, R. Maheshwary, M. Altomare, M. A. Haggag, A. Amayuelas, et al. Include: Evaluating multilingual language understanding with regional knowledge. arXiv preprint arXiv:2411.19799, 2024.
  58. K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI… find
  59. AAAI Press, 2020. URL https://aaai.org/ojs/index.php/AAAI/article/view/6399 . ↗
  60. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. find
  61. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300 . ↗
  62. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. find
  63. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. find
  64. S. Singh, A. Romanou, C. Fourrier, D. I. Adelani, J. G. Ngui, D. Vila-Suero, P. Limkonchotiwat, K. Marchisio, W. Q. Leong, Y. Susanto, et al. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation. In Proceedings of the 63rd Annual Meeting of the Associat…
  65. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
  66. M. Sun, X. Chen, J. Z. Kolter, and Z. Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17762, 2024. find
  67. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. Le, E. Chi, D. Zhou, and J. Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Findings of the Association for Computational Linguist…
  68. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  69. K. Vodrahalli, S. Ontanon, N. Tripuraneni, K. Xu, S. Jain, R. Shivanna, J. Hui, N. Dikkala, M. Kazemi, B. Fatemi, et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640, 2024. find
  70. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J…
  71. J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025. find
  72. B. Xia, B. Shen, D. Zhu, D. Zhang, G. Wang, H. Zhang, H. Liu, J. Xiao, J. Dong, L. Zhao, et al. Mimo: Unlocking the reasoning potential of language model–from pretraining to posttraining. arXiv preprint arXiv:2505.07608, 2025. find
  73. C. S. Xia, Y. Deng, S. Dunn, and L. Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489, 2024. find
  74. G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. find
  75. J. Yang, K. Lieret, C. E. Jimenez, A. Wettig, K. Khandpur, Y. Zhang, B. Hui, O. Press, L. Schmidt, and D. Yang. Swe-smith: Scaling data for software engineering agents, 2025. URL https://arxiv.org/abs/2504.21798 . — [arXiv:2504.21798](https://arxiv.org/abs/2504.21798) ↗
  76. F. Yao, L. Liu, D. Zhang, C. Dong, J. Shang, and J. Gao. Your efficient rl framework secretly brings you off-policy rl training, Aug. 2025. URL https://fengyao.notion.site/off-polic y-rl . ↗
  77. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. Traum, and L. Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy, 2019. Associa… find
  78. X. Zhao, Y. Liu, K. Xu, J. Guo, Z. Wang, Y. Sun, X. Kong, Q. Cao, L. Jiang, Z. Wen, Z. Zhang, and J. Zhou. Small leak can sink a great ship–boost rl training on moe with icepop!, Sep 2025. URL https://ringtech.notion.site/icepop . ↗
  79. C. Zheng, S. Liu, M. Li, X.-H. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025.
  80. L. Zheng, L. Yin, Z. Xie, C. L. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37:62557–62583, 2024.
  81. Y. Zhong, Z. Zhang, B. Wu, S. Liu, Y. Chen, C. Wan, H. Hu, L. Xia, R. Ming, Y. Zhu, et al. Optimizing {RLHF} training for large language models with stage fusion. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), pages 489–503, 2025. find
  82. Y. Zhou, H. Liu, Z. Chen, Y. Tian, and B. Chen. Gsm-infinite: How do your llms behave over infinitely increasing context length and reasoning complexity? arXiv preprint arXiv:2502.05252, 2025. find
  83. T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877, 2024.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/team-2026-llm-mimo-v2-flash-technical/