Elamur Research
☰
home / papers / 2026 / DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2606.19348

Summary

Reasoning LLMs improve when they spend more compute at inference time. Standard attention, however, grows quadratically with sequence length, which makes very long inputs and extended agentic workflows costly. DeepSeek-V4 is a preview family of two MoE models, DeepSeek-V4-Pro and DeepSeek-V4-Flash, built to process one-million-token contexts cheaply.

The models keep DeepSeekMoE and MTP from DeepSeek-V3 but change three things:

  • Hybrid attention. Compressed Sparse Attention (CSA) shrinks the KV cache along the sequence and then runs DeepSeek Sparse Attention. Heavily Compressed Attention (HCA) compresses more aggressively but stays dense.
  • mHC. Manifold-Constrained Hyper-Connections replace plain residual connections.
  • Muon. The models are trained with the Muon optimizer.

After pre-training on 32T+ tokens, domain specialists are trained (SFT, then GRPO) and merged into one model through on-policy distillation. At 1M tokens, Pro uses about a quarter of DeepSeek-V3.2's per-token compute and a tenth of its KV cache. In its maximum-reasoning mode, the authors claim state of the art among open models. The checkpoints are public, which lowers the barrier to long-context research.

Key points

  • Model sizes. DeepSeek-V4-Pro has 1.6T total parameters (49B active). DeepSeek-V4-Flash has 284B total (13B active). Both natively support a 1M-token context.
  • Efficiency vs. DeepSeek-V3.2 at 1M tokens. Pro needs about 27% of single-token FLOPs and 10% of the KV cache. Flash needs about 10% and 7%. Routed expert weights are stored in FP4, with FP4 quantization-aware training applied in post-training.
  • Pre-training data. Flash saw 32T tokens and Pro saw 33T. DeepSeek-V4-Flash-Base already beats DeepSeek-V3.2-Base on most benchmarks.
  • Infrastructure. The system includes:
    • a fused MoE kernel that overlaps compute, communication and memory access;
    • kernels written in TileLang;
    • batch-invariant, deterministic kernels for bitwise reproducibility;
    • a heterogeneous KV cache with on-disk storage for reusing shared prefixes.
  • Internal coding evaluation. On 30 curated R&D coding tasks, DeepSeek-V4-Pro clearly beats Claude Sonnet 4.5 and comes close to Opus 4.5. In a developer survey (N = 85), 52% would adopt it as their main coding model and 39% leaned toward yes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

113 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2606.19348 · arXiv 2606.19348

@misc{deepseekai2026deepseek,
  title = {{DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence}},
  author = {DeepSeek-AI and Xu, Anyi and Lin, Bangcai and Xue, Bing and Wang, Bingxuan and Xu, Bingzheng and Wu, Bochao and Zhang, Bowei and Lin, Chaofan and Dong, Chen and Ling, Chenchen and Lu, Chengda and Zhao, Chenggang and Deng, Chengqi and Hou, Chengyu and Xu, Chenhao and Shao, Chenze and Ruan, Chong and Sun, Conner and Dai, Damai and Guo, Daya and Yang, Dejian and Chen, Deli and Li, Donghao and Ji, Dongjie and Li, Erhang and Wei, Fang and Lin, Fangyun and Yuan, Fangzhou and Xia, Feiyu and Dai, Fucong and Hao, Guangbo and Chen, Guanting and Cao, Guoai and Meng, Guolai and Li, Guowei and Yu, Han and Zhang, Han and Xu, Hanwei and Li, Hao and Liang, Haofen and Zhang, Haoling and Luo, Haoming and Wei, Haoran and Yuan, Haotian and Zhang, Haowei and Luo, Haowen and Chen, Haoyu and Ji, Haozhe and Zhang, Hengqing and Ding, Honghui and Tang, Hongxuan and Cao, Huanqi and Gao, Huazuo and Qu, Hui and Zeng, Hui and Yang, J and Zhu, JQ and Luo, Jia and Song, Jia and Yu, Jia and Huang, Jialiang and Cai, Jialu and Liang, Jian and Zhou, Jiangting and Ye, Jiasheng and Li, Jiashi and Xu, Jiaxin and Hu, Jiewen and Yang, Jieyu and Chen, Jin and Yan, Jin and Chen, Jingchang and Zhou, Jingli and Xiang, Jingting and Yuan, Jingyang and Cheng, Jingyuan and Zhou, Jingzi and Zhu, Jinhua and Yu, Jiping and Sun, Joseph and Ran, Jun and Jiang, Junguang and Qiu, Junjie and Li, Junlong and Zheng, Junmin and Song, Junxiao and Dong, Kai and Gao, Kaige and Guan, Kang and Zhou, Kexing and Huang, Kezhao and Yu, Kuai and Wang, Lean and Zhang, Lecong and Wang, Lei and Xia, Leyi and Zhang, Li and Zhao, Liang and Guo, Lihua and Luo, Lingxiao and Ma, Linwang and Zhu, Linyan and Wang, Litong and Cai, Liyu and Zhang, Liyue and Chen, Longhao and Di, MS and Xu, MY and Mei, Max and Wang, Miaojun and Zhang, Mingchuan and Zhang, Minghua and Tang, Minghui and Li, Mingming and Zhou, Mingxu and Han, Minmin and Wang, Ning and Huang, Panpan and Wang, Panpan and Cong, Peixin and Wang, Peiyi and Zhang, Peng and Wang, Qiancheng and Zhu, Qihao and Li, Qingyang and Chen, Qinyu and Du, Qiushi and Jiang, Qiwei and Tian, Rui and Xu, Ruifan and Lu, Ruijie and Xu, Ruiling and Ge, Ruiqi and Zhang, Ruisong and Pan, Ruizhe and Wang, Runji and Chen, Runqian and Yin, Runqiu and Xu, Runxin and Shen, Ruomeng and Zhang, Ruoyu and Chen, Ruyi and Liu, SH and Lu, Shanghao and Sun, Shangmian and Zhou, Shangyan and Chen, Shanhuang and Cai, Shaofei and Nie, Shaoheng and Wu, Shaoqing and Chen, Shaoyuan and Hu, Shengding and Liu, Shengyu and Hu, Shiqiang and Ma, Shirong and Wang, Shiyu and Yu, Shuiping and Zhou, Shunfeng and Pan, Shuting and Yu, Shuying and Zhou, Songyang and Ni, Tao and Yun, Tao and Jin, Tian and Pei, Tian and Ye, Tian and Lin, Tianle and Ji, Tianran and Cui, Tianyi and Yue, Tianyuan and Yu, Tingting and Wang, Tun and Zhang, W and Xiao, WL and Zeng, Wangding and An, Wei and Zhao, Weilin and Liu, Wen and Liang, Wenfeng and Pang, Wenjie and Luo, Wenjing and Yao, Wenjing and Gao, Wenjun and Yang, Wenkai and Huang, Wenlve and Hou, Wenqing and Zhang, Wentao and Ma, Wenting and Gao, Xi and He, Xiang and Wang, Xiangwen and Wang, Xianzu and Bi, Xiao and Liu, Xiaodong and Wang, Xiaohan and Chen, Xiaokang and Zhang, Xiaokang and Nie, Xiaotao and Sun, Xiaowen and Wang, Xiaoxiang and Cheng, Xin and Liu, Xin and Xie, Xin and Liu, Xingchao and Liu, Xingchen and Yu, Xingkai and Li, Xingyou and Yang, Xinyu and Zhang, Xinyu and Chen, Xu and Wang, Xuanyu and Su, Xuecheng and Chen, Xueyin and Lin, Xuheng and Fu, Xuwei and Yan, YC and Wang, YQ and Ma, YW and Luo, Yanfeng and Zhang, Yang and Xu, Yanhong and Ma, Yanru and Huang, Yanwen and Li, Yao and Li, Yao and Xu, Yao and Zhao, Yao and Sun, Yaofeng and Wang, Yaohui and Qian, Yi and Shao, Yi and Yu, Yi and Zhang, Yichao and Ding, Yifan and Shi, Yifan and Wu, Yijia and Xiong, Yiliang and Ma, Yiling and He, Ying and Tang, Ying and Zhou, Ying and Luo, Yingjia and Zhong, Yinmin and Piao, Yishi and Wang, Yisong and Zhang, Yixiang and Chen, Yixiao and Tan, Yixuan and Wei, Yixuan and Ma, Yiyang and Liu, Yiyuan and Yang, Yonglun and Guo, Yongqiang and Wu, Yongtong and Wu, Yu and Li, YuKun and Cheng, Yuan and Ou, Yuan and Xu, Yuanfan and Li, Yuanhao and Wang, Yuduan and Yang, Yuehan and Xu, Yuer and Wu, Yuhan and Meng, Yuhao and Zou, Yuheng and Zha, Yukun and Xiong, Yunfan and Chen, Yupeng and Lin, Yuping and Cao, Yuqian and Wang, Yuqian and Zhang, Yushun and Yan, Yuting and Lin, Yutong and Gu, Yuxian and Luo, Yuxiang and You, Yuxiang and Liu, Yuxuan and Zhou, Yuxuan and Zhou, Yuyang and Huang, Yuzhen and Wu, ZF and Wang, Zehao and Zhao, Zehua and Ren, Zehui and Zhang, Zekai and Sha, Zhangli and Fu, Zhe and Ju, Zhe and Xu, Zhean and Xie, Zhenda and Zhang, Zhengyan and Gao, Zheren and Hao, Zhewen and Gou, Zhibin and Ma, Zhicheng and Yan, Zhigang and Shao, Zhihong and Huang, Zhixian and Chen, Zhixuan and Wu, Zhiyu and Ren, Zhizhou and Wu, Zhongyu and Li, Zhuoshu and Zhang, Zhuping and Xu, Zian and Wang, Zihao and Qu, Zihua and Gu, Zihui and Zhu, Zijia and Li, Zilin and Zhang, Zipeng and Xie, Ziwei and Gao, Ziyi and Wan, Ziyi and Pan, Zizheng and Yao, Zongqing},
  year = {2026},
  doi = {10.48550/arxiv.2606.19348},
  eprint = {2606.19348},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.19348}
}

References

109 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AA. Gdpval-aa leaderboard, 2025. URL https://artificialanalysis.ai/methodolog y/intelligence-benchmarking\#gdpval-aa . ↗
  2. T. Achim, A. Best, A. Bietti, K. Der, M. Fédérico, S. Gukov, D. Halpern-Leistner, K. Henningsgard, Y. Kudryashov, A. Meiburg, et al. Aristotle: Imo-level automated theorem proving. arXiv preprint arXiv:2510.01346, 2025. find
  3. A. Agache, M. Brooker, A. Florescu, A. Iordache, A. Liguori, R. Neugebauer, P. Piwonka, and D.-M. Popa. Firecracker: lightweight virtualization for serverless applications. In Proceedings of the 17th Usenix Conference on Networked Systems Design and Implementation, NSDI’20, page 419–434, USA, 2020.… find
  4. O. J. Aimuyo, B. Oh, and R. Singh. Flashmoe: Fast distributed moe in a single kernel. Advances in Neural Information Processing Systems, 2025. URL https://neurips.cc/virtual/2 025/poster/119124 . ↗
  5. J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023. find our card
  6. J. Asher. LeanExplore: A search engine for Lean 4 declarations, 2025. URL https://arxiv.or g/abs/2506.11085 . — [arXiv:2506.11085](https://arxiv.org/abs/2506.11085) ↗
  7. Y. Bai, Y. Bao, G. Chen, J. Chen, N. Chen, R. Chen, Y. Chen, Y. Chen, Y. Chen, Z. Chen, J. Cui, H. Ding, M. Dong, A. Du, C. Du, D. Du, Y. Du, Y. Fan, Y. Feng, K. Fu, B. Gao, H. Gao, P. Gao, T. Gao, X. Gu, L. Guan, H. Guo, J. Guo, H. Hu, X. Hao, T. He, W. He, W. He, C. Hong, Y. Hu, Z. Hu, W. Huang, Z… find our card
  8. Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find our card
  9. M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev. Matharena: Evaluating llms on uncontaminated math competitions. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmark, 2025. find
  10. C. Bandi, B. Hertzberg, G. Boo, T. Polakam, J. Da, S. Hassaan, M. Sharma, A. Park, E. Hernandez, D. Rambado, et al. Mcp-atlas: A large-scale benchmark for tool-use competency with real mcp servers. arXiv preprint arXiv:2602.00933, 2026. — [arXiv:2602.00933](https://arxiv.org/abs/2602.00933) find
  11. F. Bellard. Qemu, a fast and portable dynamic translator. In Proceedings of the Annual Conference on USENIX Annual Technical Conference, ATEC ’05, page 41, USA, 2005. USENIX Association. find
  12. I. Bello, H. Pham, Q. V. Le, M. Norouzi, and S. Bengio. Neural combinatorial optimization with reinforcement learning, 2017. URL https://openreview.net/forum?id=rJY3vK9eg . ↗
  13. J. Chen, W. Chen, J. Du, J. Hu, Z. Jiang, A. Jie, X. Jin, X. Jin, C. Li, W. Shi, Z. Wang, M. Wang, C. Wei, S. Wei, H. Xin, F. Yang, W. Gao, Z. Yuan, T. Zhan, Z. Zheng, T. Zhou, and T. H. Zhu. Seed-prover 1.5: Mastering undergraduate-level theorem proving via learning from experience, 2025. URL https… find
  14. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such,… find our card
  15. T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, H. Shen, M. Cowan, L. Wang, Y. Hu, L. Ceze, C. Guestrin, and A. Krishnamurthy. TVM: An automated End-to-End optimizing compiler for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 578–594, Carlsba… find our card
  16. A. Cheng, A. Jacovi, A. Globerson, B. Golan, C. Kwong, C. Alberti, C. Tao, E. Ben-David, G. S. Tomar, L. Haas, et al. The facts leaderboard: A comprehensive benchmark for large language model factuality. arXiv preprint arXiv:2512.10791, 2025. — [arXiv:2512.10791](https://arxiv.org/abs/2512.10791) find
  17. X. Cheng, W. Zeng, D. Dai, Q. Chen, B. Wang, Z. Xie, K. Huang, X. Yu, Z. Hao, Y. Li, H. Zhang, H. Zhang, D. Zhao, and W. Liang. Conditional memory via scalable lookup: A new axis of sparsity for large language models. CoRR, abs/2601.07372, 2026. doi: 10.48550/ARXIV.2601.07372. URL https://doi.org/10… find
  18. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. find our card
  19. D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y. K. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. CoRR, abs/2401.06066, 2024. URL https://doi.org/10.48550/ar… ↗
  20. T. Dao, D. Haziza, F. Massa, and G. Sizov. Flash-decoding for long-context inference, 2023. URL https://pytorch.org/blog/flash-decoding/ . ↗
  21. L. De Moura and N. Bjørner. Z3: an efficient smt solver. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS’08/ETAPS’08, page 337–340, Berlin, Heidelberg, 2008. Springer-Verlag. ISBN 3540787… find
  22. DeepSeek-AI. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. CoRR, abs/2406.11931, 2024. URL https://doi.org/10.48550/arXiv.2406.11 931 . ↗ our card
  23. DeepSeek-AI. Deepseek-v3 technical report. CoRR, abs/2412.19437, 2024. URL https://doi. org/10.48550/arXiv.2412.19437 . find
  24. DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRR, abs/2405.04434, 2024. URL https://doi.org/10.48550/arXiv.2405.04 434 . ↗
  25. DeepSeek-AI. Fire-flyer file system, 2025. URL https://github.com/deepseek-ai/3FS . ↗
  26. DeepSeek-AI. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nat., 645(8081):633–638, 2025. URL https://doi.org/10.1038/s41586-025-09422-z . ↗ our card
  27. DeepSeek-AI. Deepseek-v3.2: Pushing the frontier of open large language models, 2025. URL https://arxiv.org/abs/2512.02556 . ↗ our card
  28. X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath, M. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, V. Bharadwaj, J. Holm, R. Aluri, C. B. C. Zhang, N. Jacobson, B. Liu, and B. Kenstler. Swe-bench pro: Can ai agents solve long-horizon software engi… — [arXiv:2509.16941](https://arxiv.org/abs/2509.16941) find
  29. H. Ding, Z. Wang, G. Paolini, V. Kumar, A. Deoras, D. Roth, and S. Soatto. Fewer truncations improve language modeling. arXiv preprint arXiv:2404.10830, 2024. — [arXiv:2404.10830](https://arxiv.org/abs/2404.10830) find
  30. X. Dong, Y. Fu, S. Diao, W. Byeon, Z. CHEN, A. S. Mahabaleshwarkar, S.-Y. Liu, M. V. keirsbilck, M.-H. Chen, Y. Suhara, Y. C. Lin, J. Kautz, and P. Molchanov. Hymba: A hybrid-head architecture for small language models. In The Thirteenth International Conference on Learning Representations, 2025. UR… find
  31. X. Du, Y. Yao, K. Ma, B. Wang, T. Zheng, K. Zhu, M. Liu, Y. Liang, X. Jin, Z. Wei, et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739, 2025. find
  32. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa… find our card
  33. X. Gao, M. Dong, X. Miao, W. Du, C. Yu, and H. Chen. Erofs: a compression-friendly readonly file system for resource-scarce devices. In Proceedings of the 2019 USENIX Conference on Usenix Annual Technical Conference, USENIX ATC ’19, page 149–162, USA, 2019. USENIX Association. ISBN 9781939133038. find
  34. A. P. Gema, J. O. J. Leang, G. Hong, A. Devoto, A. C. M. Mancino, R. Saxena, X. He, Y. Zhao, X. Du, M. R. G. Madani, C. Barale, R. McHardy, J. Harris, J. Kaddour, E. van Krieken, and P. Minervini. Are we done with mmlu? CoRR, abs/2406.04127, 2024. URL https://doi.or g/10.48550/arXiv.2406.04127 . ↗
  35. F. Gloeckle, B. Y. Idrissi, B. Rozière, D. Lopez-Paz, and G. Synnaeve. Better & faster large language models via multi-token prediction. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. find
  36. OpenReview.net, 2024. URL https://openreview.net/forum?id=pEWAcejiU2 . ↗
  37. Y. Gu, L. Dong, F. Wei, and M. Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024. find
  38. L. Haas, G. Yona, G. D’Antonio, S. Goldshtein, and D. Das. Simpleqa verified: A reliable factuality benchmark to measure parametric knowledge. arXiv preprint arXiv:2509.07968, 2025. find
  39. Y. He, S. Li, J. Liu, Y. Tan, W. Wang, H. Huang, X. Bu, H. Guo, C. Hu, B. Zheng, et al. Chinese simpleqa: A chinese factuality evaluation for large language models. arXiv preprint arXiv:2411.07140, 2024. find
  40. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. find our card
  41. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. find our card
  42. Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023. find our card
  43. D. Hupkes and N. Bogoychev. Multiloko: a multilingual local knowledge benchmark for llms spanning 31 languages. CoRR, abs/2504.10356, 2025. doi: 10.48550/ARXIV.2504.10356. URL https://doi.org/10.48550/arXiv.2504.10356 . ↗
  44. B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. find
  45. N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. find our card
  46. K. Jordan, Y. Jin, V. Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein. Muon: An optimizer for hidden layers in neural networks. Cited on, page 10, 2024. find
  47. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In R. Barzilay and M.-Y. Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601… find our card
  48. H. Li, Y. Yuan, R. Du, K. Ma, L. Liu, and W. Hsu. DADI: Block-Level image service for agile and elastic application deployment. In 2020 USENIX Annual Technical Conference (USENIX ATC 20), pages 727–740. find
  49. USENIX Association, July 2020. ISBN 978-1-939133-14-4. URL https://www.usenix.org/conference/atc20/presentation/li-huiba . ↗
  50. H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023. find our card
  51. J. Li, W. Zhao, J. Zhao, W. Zeng, H. Wu, X. Wang, R. Ge, Y. Cao, Y. Huang, W. Liu, et al. The tool decathlon: Benchmarking language agents for diverse, realistic, and long-horizon task execution. arXiv preprint arXiv:2510.25726, 2025. find
  52. Y. Li, F. Wei, C. Zhang, and H. Zhang. EAGLE: speculative sampling requires rethinking feature uncertainty. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. find our card
  53. OpenReview.net, 2024. URL https://openreview.net /forum?id=1NdN7eXyb4 . ↗
  54. J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du, Y. Qin, W. Xu, E. Lu, J. Yan, Y. Chen, H. Zheng, Y. Liu, S. Liu, B. Yin, W. He, H. Zhu, Y. Wang, J. Wang, M. Dong, Z. Zhang, Y. Kang, H. Zhang, X. Xu, Y. Zhang, Y. Wu, X. Zhou, and Z. Yang. Muon is scalable for LLM training. CoRR, abs/2502.16982, 2025.… find
  55. I. Loshchilov and F. Hutter. arXiv:1711.05101, 2017. find
  56. Decoupled weight decay regularization. K. Lu and T. M. Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation. — [doi:10.64434/tml.20251026](https://doi.org/10.64434/tml.20251026) ↗
  57. Z. Lu, C. Li, Y. Shi, W. Shen, M. Yan, and F. Huang. Corpusqa: A 10 million token benchmark for corpus-level analysis and reasoning. arXiv preprint arXiv:2601.14952, 2026. — [arXiv:2601.14952](https://arxiv.org/abs/2601.14952) find
  58. T. Luong, D. Hwang, H. H. Nguyen, G. Ghiasi, Y. Chervonyi, I. Seo, J. Kim, G. Bingham, J. Lee, S. Mishra, A. Zhai, C. H. Hu, H. Michalewski, J. Kim, J. Ahn, J. Bae, X. Song, T. H. Trinh, Q. V. Le, and J. Jung. Towards robust mathematical reasoning. In Proceedings of the 2025 Conference on Empirical… find
  59. M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. — [arXiv:2601.11868](https://arxiv.org/abs/2601.11868) find
  60. MiniMax. Meet minimax-m2, 2025. URL https://github.com/MiniMax-AI/MiniMax-M2 . ↗
  61. L. d. Moura and S. Ullrich. The lean 4 theorem prover and programming language. In International Conference on Automated Deduction, pages 625–635. Springer, 2021. find our card
  62. Y. Nesterov. A method of solving a convex programming problem with convergence rate 𝑂 (1/ 𝑘 2 ). Soviet Mathematics Doklady, 27:372–376, 1983. find
  63. NVIDIA Corporation. cublas documentation, 2024. URL https://docs.nvidia.com/cuda /cublas/ . Version 12.4. Accessed: 2024-09-16. ↗
  64. OpenAI. Multilingual massive multitask language understanding (mmmlu), 2024a. URL https://huggingface.co/datasets/openai/MMMLU . arXiv preprint OpenAI. Openai mrcr: Long context multiple needle in a haystack benchmark, 2024b. URL https://huggingface.co/datasets/openai/mrcr . ↗
  65. OpenAI. Learning to reason with llms, 2024c. URL https://openai.com/index/learnin g-to-reason-with-llms . ↗
  66. OpenAI. Introducing SimpleQA, 2024d. URL https://openai.com/index/introducing -simpleqa/ . ↗
  67. OpenAI. Introducing SWE-bench verified we’re releasing a human-validated subset of swe-bench that more, 2024e. URL https://openai.com/index/introducing-swe-bench-v erified/ . ↗
  68. OpenAI. gpt-oss-120b & gpt-oss-20b model card. CoRR, abs/2508.10925, 2025. doi: 10.48550/A RXIV.2508.10925. URL https://doi.org/10.48550/arXiv.2508.10925 . ↗ our card
  69. M. Osama, D. Merrill, C. Cecka, M. Garland, and J. D. Owens. Stream-k: Work-centric parallel decomposition for dense matrix-matrix multiplication on the gpu. In Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, pages 429–431, 2023. find
  70. T. Patwardhan, R. Dias, E. Proehl, G. Kim, M. Wang, O. Watkins, S. P. Fishman, M. Aljubeh, P. Thacker, L. Fauconnet, et al. Gdpval: Evaluating ai model performance on real-world economically valuable tasks. arXiv preprint arXiv:2510.04374, 2025. find
  71. L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al. Humanity’s last exam. arXiv preprint arXiv:2501.14249, 2025. find our card
  72. W. Qi, Y. Yan, Y. Gong, D. Liu, N. Duan, J. Chen, R. Zhang, and M. Zhou. Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training. In T. Cohn, Y. He, and Y. Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volum… — [doi:10.18653/v1/2020.f](https://doi.org/10.18653/v1/2020.f) find
  73. Qwen. Qwen3 technical report. CoRR, abs/2505.09388, 2025. doi: 10.48550/ARXIV.2505.09388. URL https://doi.org/10.48550/arXiv.2505.09388 . ↗ our card
  74. S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. find
  75. J. K. Reed, Z. DeVito, H. He, A. Ussery, and J. Ansel. Torch.fx: Practical program capture and transformation for deep learning in python, 2022. URL https://arxiv.org/abs/2112.0 8429 . ↗
  76. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. find our card
  77. G. T. M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ram’e, J. Ferret, P. Liu, P. D. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stańczyk, S. Girgin, N. Momchev, M. Hoffman, S. Thako… find
  78. S. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2… find
  79. B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolf, S. Dusan, V. Elango, M. Golub, A. Heinecke, P. James-Roxby, D. Jani, G. Kolhe, M. Langhammer, A. Li, L. Melnick, M. Mesmakhosroshahi, A. Rodriguez, M. Schulte, R. Shafipour, L. Shao,… find
  80. K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. find
  81. Z. Shao, Y. Luo, C. Lu, Z. Z. Ren, J. Hu, T. Ye, Z. Gou, S. Ma, and X. Zhang. Deepseekmath-v2: Towards self-verifiable mathematical reasoning, 2025. URL https://arxiv.org/abs/25 11.22570 . ↗
  82. N. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150 . ↗
  83. N. Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. find
  84. F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, D. Das, and J. Wei. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. find our card
  85. OpenReview.net, 2023. URL https://openreview.net/forum?i d=fR3wGCk-IXp . ↗
  86. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. find
  87. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. find our card
  88. G. Tsoukalas, J. Lee, J. Jennings, J. Xin, M. Ding, M. Jennings, A. Thakur, and S. Chaudhuri. Putnambench: Evaluating neural theorem-provers on the putnam mathematical competition, 2024. URL https://arxiv.org/abs/2407.11214 . ↗
  89. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. find
  90. L. Wang, H. Gao, C. Zhao, X. Sun, and D. Dai. Auxiliary-loss-free load balancing strategy for mixture-of-experts. CoRR, abs/2408.15664, 2024a. URL https://doi.org/10.48550/arX iv.2408.15664 . — [doi:10.48550/arx](https://doi.org/10.48550/arx) ↗
  91. L. Wang, Y. Cheng, Y. Shi, Z. Mo, Z. Tang, W. Xie, T. Wu, L. Ma, Y. Xia, J. Xue, et al. Tilelang: Bridge programmability and performance in modern neural kernels. In The Fourteenth International Conference on Learning Representations, 2026. find
  92. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. CoRR, abs/2406.01574, 2024b. URL https://doi.org/10.48550/arXi… ↗ our card
  93. J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025. find
  94. T. Wei, J. Luan, W. Liu, S. Dong, and B. Wang. Cmath: Can your language model pass chinese elementary school math test?, 2023. find
  95. G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. find
  96. OpenReview.net, 2024. URL https://openreview .net/forum?id=NG7sS51zVF . find
  97. Z. Xie, Y. Wei, H. Cao, C. Zhao, C. Deng, J. Li, D. Dai, H. Gao, J. Chang, K. Yu, L. Zhao, S. Zhou, Z. Xu, Z. Zhang, W. Zeng, S. Hu, Y. Wang, J. Yuan, L. Wang, and W. Liang. mhc: Manifold-constrained hyper-connections, 2026. URL https://arxiv.org/abs/2512.24880 . — [arXiv:2512.24880](https://arxiv.org/abs/2512.24880) ↗
  98. L. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese langu… find our card
  99. J. Yang, K. Lieret, C. E. Jimenez, A. Wettig, K. Khandpur, Y. Zhang, B. Hui, O. Press, L. Schmidt, and D. Yang. Swe-smith: Scaling data for software engineering agents, 2025. URL https: //arxiv.org/abs/2504.21798 . — [arXiv:2504.21798](https://arxiv.org/abs/2504.21798) find
  100. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 20… find
  101. C. Zhang, K. Du, S. Liu, W. Kwon, X. Mo, Y. Wang, X. Liu, K. You, Z. Li, M. Long, J. Zhai, J. Gonzalez, and I. Stoica. Jenga: Effective memory management for serving llm with heterogeneity. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, SOSP ’25, page 446–461, New Y… find
  102. S. Zhang, N. Zheng, H. Lin, Z. Jiang, W. Bao, C. Jiang, Q. Hou, W. Cui, S. Zheng, L.-W. Chang, Q. Chen, and X. Liu. Comet: Fine-grained computation-communication overlapping for mixture-of-experts. 2025b. URL https://arxiv.org/abs/2502.19811 . ↗
  103. C. Zhao, L. Zhao, J. Li, Z. Xu, and C. Xu. Deepgemm: clean and efficient fp8 gemm kernels with fine-grained scaling. https://github.com/deepseek-ai/DeepGEMM , 2025. ↗
  104. W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. AGIEval: A human-centric benchmark for evaluating foundation models. CoRR, abs/2304.06364, 2023. doi: 10.48550/arXiv.2304.06364. URL https://doi.org/10.48550/arXiv.2304.06364 . ↗ our card
  105. D. Zhu, H. Huang, Z. Huang, Y. Zeng, Y. Mao, B. Wu, Q. Min, and X. Zhou. Hyper-connections. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. find
  106. OpenReview.net, 2025. URL https://openreview.net /forum?id=9FqARW7dwB . ↗
  107. X. Zhu, D. Cheng, H. Li, K. Zhang, E. Hua, X. Lv, N. Ding, Z. Lin, Z. Zheng, and B. Zhou. How to synthesize text data without model collapse? arXiv preprint arXiv:2412.14689, 2024. — [arXiv:2412.14689](https://arxiv.org/abs/2412.14689) find
  108. T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, S. Brunner, C. Gong, J. Hoang, A. R. Zebaze, X. Hong, W. Li, J. Kaddour, M. Xu, Z. Zhang, P. Yadav, and et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instru… find our card
  109. OpenReview.net, 2025. URL http s://openreview.net/forum?id=YrycTjllL0 . find