Elamur Research⌕
☰
home / papers / 2025 / Kimi Linear: An Expressive, Efficient Attention Architecture

Kimi Linear: An Expressive, Efficient Attention Architecture

Paper · 2025 · Kimi Team, Yu Zhang, Zongyu Lin et al.

Summary

Softmax attention becomes a bottleneck for agentic and RL-heavy LLM workloads. Its cost grows quadratically with sequence length, and its KV cache keeps expanding during long decoding. Earlier linear-attention and hybrid designs saved compute but usually lost quality or were only tested at small scale. The Kimi Team proposes Kimi Linear, which mixes Kimi Delta Attention (KDA) and Multi-Head Latent Attention (MLA) layers at a 3:1 ratio. KDA builds on Gated DeltaNet, replacing its single per-head forget gate with separate decay rates for each channel. This gives finer control over the fixed-size recurrent memory. A custom chunkwise kernel based on a constrained DPLR transition keeps training fast. Under an identical recipe with 1.4T training tokens, a 48B-parameter MoE model (3B active) beat full MLA on short-context, long-context and RL-style evaluations. It also cut KV cache size by up to 75% and decoded about six times faster at 1M tokens. The result suggests hybrid linear attention can directly replace full attention in long-output, decoding-heavy settings.

Key points

  • Architecture: Three KDA layers alternate with one global MLA layer; the full-attention layers use NoPE, which the authors found simplifies context extension. The model is a 48B-total / 3B-activated MoE.
  • Benchmark results (1.4T-token fair comparison):
    • MMLU-Pro (4k): Kimi Linear 51.0, GDN-H 47.9, MLA 47.2.
    • RULER (128k): Kimi Linear 84.3, MLA 81.3, GDN-H 80.5, with ~3.98× speedup.
  • Decoding speed: At 1M context, time per output token is 1.84 ms versus 11.48 ms for MLA (6.3×), and KV cache use drops by up to 75%.
  • How KDA works: It uses channel-wise gating in the style of GLA together with the delta rule. Its specialized DPLR variant needs less computation than general DPLR, and training relies on WY/UT transforms.
  • Released resources: The KDA kernel is in flash-linear-attention (fla/ops/kda), alongside a vLLM integration and pre-trained and instruction-tuned checkpoints on Hugging Face (Kimi-Linear-48B-A3B-Instruct).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 242 works

Cite

Cite as Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang, Yu Fan, Longguang Zhong, Enming Yuan, Dehao Zhang, Yizhi Zhang, T.Y. Liu, Haiming Wang, Shengjun Fang, Weiran He, Shaowei Liu, Yiwei Li, Jianlin Su, Jiezhong Qiu, Bo Pang, Junjie Yan, Zhejun Jiang, Weixiao Huang, Bohong Yin, Jiacheng You, Chu Wei, Zhengtao Wang, Chao Hong, Yutian Chen, Guanduo Chen, Yucheng Wang, Huabin Zheng, Feng Wang, Yibo Liu, Mengnan Dong, Zheng Zhang, Siyuan Pan, Wenhao Wu, Yuhao Wu, Longyu Guan, Jiawen Tao, Guohong Fu, Xinran Xu, Yuzhi Wang, Guokun Lai, Yuxin Wu, Xinyu Zhou, Zhilin Yang and Yulun Du. 2025. Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv:2510.26692.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2510.26692 · arXiv 2510.26692

@misc{team2025kimi,
  title = {{Kimi Linear: An Expressive, Efficient Attention Architecture}},
  author = {Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang, Yu Fan, Longguang Zhong, Enming Yuan, Dehao Zhang, Yizhi Zhang, T.Y. Liu, Haiming Wang, Shengjun Fang, Weiran He, Shaowei Liu, Yiwei Li, Jianlin Su, Jiezhong Qiu, Bo Pang, Junjie Yan, Zhejun Jiang, Weixiao Huang, Bohong Yin, Jiacheng You, Chu Wei, Zhengtao Wang, Chao Hong, Yutian Chen, Guanduo Chen, Yucheng Wang, Huabin Zheng, Feng Wang, Yibo Liu, Mengnan Dong, Zheng Zhang, Siyuan Pan, Wenhao Wu, Yuhao Wu, Longyu Guan, Jiawen Tao, Guohong Fu, Xinran Xu, Yuzhi Wang, Guokun Lai, Yuxin Wu, Xinyu Zhou, Zhilin Yang and Yulun Du},
  year = {2025},
  doi = {10.48550/arxiv.2510.26692},
  eprint = {2510.26692},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.26692}
}

References

126 references, 46 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandhini Agarwal et al. “gpt-oss-120b & gpt-oss-20b model card”. In: arXiv preprint arXiv:2508.10925 (2025).
  2. Joshua Ainslie et al. “Colt5: Faster long-range transformers with conditional computation”. In: arXiv preprint arXiv:2303.09752 (2023).
  3. Zeyuan Allen-Zhu. “Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers”. In: SSRN Electronic Journal (May 2025). Available at SSRN: https://ssrn.com/abstract=5240330 or http://dx.doi.org/10.2139/ssrn.5240330. DOI : 10.2139/ssrn.5240330. — [doi:10.2139/ssrn.5240330](https://doi.org/10.2139/ssrn.5240330) ↗
  4. Simran Arora et al. “Simple linear attention language models balance the recall-throughput tradeoff”. In: Forty-first International Conference on Machine Learning. 2024. URL : https://openreview.net/forum?id=e93ffDcpH3. ↗
  5. Simran Arora et al. Zoology: Measuring and Improving Recall in Efficient Language Models. 2023. arXiv: 2312.04927 [cs.CL].
  6. Yushi Bai et al. “Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks”. In: arXiv preprint arXiv:2412.15204 (2024).
  7. Federico Barbero et al. “Round and Round We Go! What makes Rotary Positional Encodings useful?” In: Proceedings of ICLR. 2025. URL : https://openreview.net/forum?id=GtvuNrk58a. ↗
  8. Ali Behrouz et al. “Atlas: Learning to optimally memorize the context at test time”. In: arXiv preprint arXiv:2505.23735 (2025). — [arXiv:2505.23735](https://arxiv.org/abs/2505.23735) find
  9. Amanda Bertsch et al. “Unlimiformer: Long-range transformers with unlimited length input”. In: Advances in NeurIPS 36 (2023), pp. 35522–35543. find
  10. Stella Biderman et al. “Lessons from the trenches on reproducible evaluation of language models”. In: arXiv preprint arXiv:2405.14782 (2024). find
  11. Christian Bischof and Charles Van Loan. “The WY Representation for Products of Householder Matrices”. In: SIAM Journal on Scientific and Statistical Computing (1987), s2–s13. URL : https://doi.org/10.1137/0908009. — [doi:10.1137/0908009](https://doi.org/10.1137/0908009) ↗
  12. Aaron Blakeman et al. “Nemotron-h: A family of accurate and efficient hybrid mamba-transformer models”. In: arXiv preprint arXiv:2504.03624 (2025).
  13. Egor Bogomolov et al. “Long code arena: a set of benchmarks for long-context code models”. In: arXiv preprint arXiv:2406.11612 (2024). — [arXiv:2406.11612](https://arxiv.org/abs/2406.11612) find
  14. Peter Clark et al. “Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge”. In: arXiv:1803.05457v1 (2018).
  15. Ganqu Cui et al. “The entropy mechanism of reinforcement learning for reasoning language models”. In: arXiv preprint arXiv:2505.22617 (2025). find
  16. Tri Dao and Albert Gu. “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality”. In: CoRR abs/2405.21060 (2024). DOI : 10.48550/ARXIV.2405.21060. arXiv: 2405.21060. URL : https://doi.org/10.48550/arXiv.2405.21060. ↗
  17. Tri Dao et al. “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness”. In: Advances in NeurIPS. 2022, pp. 16344–16359. URL : https://proceedings.neurips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf. ↗
  18. DeepSeek-AI. DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention. 2025. find
  19. DeepSeek-AI et al. DeepSeek-V3 Technical Report. 2025. arXiv: 2412.19437 [cs.CL]. URL : https://arxiv.org/abs/2412.19437. ↗
  20. Jiayu Ding et al. LongNet: Scaling Transformers to 1,000,000,000 Tokens. 2023. arXiv: 2307.02486 [cs.CL]. URL : https://arxiv.org/abs/2307.02486. ↗
  21. Juechu Dong et al. Flex Attention: A Programming Model for Generating Optimized Attention Kernels. 2024. arXiv: 2412.05496 [cs.LG]. URL : https://arxiv.org/abs/2412.05496. ↗
  22. Xin Dong et al. Hymba: A Hybrid-head Architecture for Small Language Models. 2024. arXiv: 2411.13676 [cs.CL]. URL : https://arxiv.org/abs/2411.13676. ↗
  23. Jusen Du et al. “Mom: Linear sequence modeling with mixture-of-memories”. In: arXiv preprint arXiv:2502.13685 (2025).
  24. Jusen Du et al. “Native Hybrid Attention for Efficient Sequence Modeling”. In: arXiv preprint arXiv:2510.07019 (2025). find
  25. Tianyu Fu et al. “Moa: Mixture of sparse attention for automatic large language model compression”. In: arXiv preprint arXiv:2406.14909 (2024). find
  26. Aryo Pradipta Gema et al. “Are we done with mmlu?” In: arXiv preprint arXiv:2406.04127 (2024). find
  27. Riccardo Grazzi et al. “Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues”. In: Proceedings of ICLR. 2025. URL : https://openreview.net/forum?id=UvTo3tVBk2. ↗
  28. Joseph F. Grcar. “How ordinary elimination became Gaussian elimination”. In: Historia Mathematica 38.2 (May 2011), pp. 163–218. ISSN : 0315-0860. DOI : 10.1016/j.hm.2010.06.003. URL : http://dx.doi.org/10.1016/j.hm.2010.06.003. — [doi:10.1016/j.hm.2010.06.003](https://doi.org/10.1016/j.hm.2010.06.003) ↗
  29. Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. 2023. arXiv: 2312.00752 [cs.LG]. find
  30. Albert Gu, Karan Goel, and Christopher Ré. Efficiently Modeling Long Sequences with Structured State Spaces. 2022. arXiv: 2111.00396 [cs.LG]. find
  31. Xiangming Gu et al. When Attention Sink Emerges in Language Models: An Empirical View. 2025. arXiv: 2410.10781 [cs.CL]. URL : https://arxiv.org/abs/2410.10781. — [arXiv:2410.10781](https://arxiv.org/abs/2410.10781) ↗
  32. Yuxian Gu et al. Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search. 2025. arXiv: 2508.15884 [cs.CL]. URL : https://arxiv.org/abs/2508.15884. — [arXiv:2508.15884](https://arxiv.org/abs/2508.15884) ↗
  33. Daya Guo et al. “DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning”. In: Nature 645.8081 (2025), pp. 633–638. find
  34. Han Guo et al. “Log-linear attention”. In: arXiv preprint arXiv:2506.04761 (2025). — [arXiv:2506.04761](https://arxiv.org/abs/2506.04761) find
  35. Qipeng Guo et al. “Star-transformer”. In: arXiv preprint arXiv:1902.09113 (2019). find
  36. Dan Hendrycks et al. Measuring Massive Multitask Language Understanding. 2021. arXiv: 2009.03300 [cs.CY]. URL : https://arxiv.org/abs/2009.03300. ↗
  37. Jordan Hoffmann et al. Training Compute-Optimal Large Language Models. 2022. arXiv: 2203 . 15556 [cs.CL]. URL : https://arxiv.org/abs/2203.15556. ↗
  38. Cheng-Ping Hsieh et al. “RULER: What’s the Real Context Size of Your Long-Context Language Models?” In: arXiv preprint arXiv:2404.06654 (2024).
  39. Jiaxi Hu et al. “Attractor memory for long-term time series forecasting: A chaos perspective”. In: Advances in NeurIPS 37 (2024), pp. 20786–20818. find
  40. Jiaxi Hu et al. “Comba: Improving Nonlinear RNNs with Closed-loop Control”. In: arXiv preprint arXiv:2506.02475 (2025). — [arXiv:2506.02475](https://arxiv.org/abs/2506.02475) find
  41. Ermo Hua et al. “Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization”. In: arXiv preprint arXiv:2412.17739 (2024). find
  42. Weizhe Hua et al. “Transformer Quality in Linear Time”. In: Proceedings of ICML. Ed. by Kamalika Chaudhuri et al. PMLR, 2022, pp. 9099–9117. URL : https://proceedings.mlr.press/v162/hua22a.html. ↗
  43. Yuzhen Huang et al. “C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models”. In: Advances in NeurIPS 36 (2023), pp. 62991–63010.
  44. Naman Jain et al. “Livecodebench: Holistic and contamination free evaluation of large language models for code”. In: arXiv preprint arXiv:2403.07974 (2024).
  45. Samy Jelassi et al. Repeat After Me: Transformers are Better than State Space Models at Copying. 2024. arXiv: 2402.01032 [cs.LG]. find
  46. Thierry Joffrain et al. “Accumulating Householder transformations, revisited”. In: (2006), pp. 169–179. URL : https://doi.org/10.1145/1141885.1141886. — [doi:10.1145/1141885.1141886](https://doi.org/10.1145/1141885.1141886) ↗
  47. Mandar Joshi et al. “Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension”. In: arXiv preprint arXiv:1705.03551 (2017).
  48. Angelos Katharopoulos et al. “Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention”. In: Proceedings of ICML. Ed. by Hal Daumé III and Aarti Singh. PMLR, 2020, pp. 5156–5165. URL : https://proceedings.mlr.press/v119/katharopoulos20a.html. ↗
  49. Amirhossein Kazemnejad et al. “The impact of positional encoding on length generalization in transformers”. In: Advances in NeurIPS 36 (2023), pp. 24892–24928. find
  50. Team Kimi et al. “Kimi k2: Open agentic intelligence”. In: arXiv preprint arXiv:2507.20534 (2025).
  51. Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. “Reformer: The efficient transformer”. In: arXiv preprint arXiv:2001.04451 (2020).
  52. Satyapriya Krishna et al. “Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation”. In: arXiv preprint arXiv:2409.12941 (2024). find
  53. Hanyu Lai et al. “A Survey of Post-Training Scaling in Large Language Models”. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025, pp. 2771– 2791.
  54. Disen Lan et al. “Liger: Linearizing Large Language Models to Gated Recurrent Structures”. In: arXiv preprint arXiv:2503.01496 (2025).
  55. Haonan Li et al. “CMMLU: Measuring massive multitask language understanding in Chinese”. In: Findings of the Association for Computational Linguistics: ACL 2024. Ed. by Lun-Wei Ku, Andre Martins, and Vivek Srikumar. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 11260–1…
  56. Yixing Li et al. “Transmamba: Flexibly switching between transformer and mamba”. In: arXiv preprint arXiv:2503.24067 (2025). — [arXiv:2503.24067](https://arxiv.org/abs/2503.24067) find
  57. Opher Lieber et al. Jamba: A Hybrid Transformer-Mamba Language Model. 2024. arXiv: 2403 . 19887 [cs.CL].
  58. Zhixuan Lin et al. “Forgetting transformer: Softmax attention with a forget gate”. In: arXiv preprint arXiv:2503.02130 (2025). — [arXiv:2503.02130](https://arxiv.org/abs/2503.02130) find
  59. Bo Liu et al. “Longhorn: State Space Models are Amortized Online Learners”. In: ArXiv abs/2407.14207 (2024). URL : https://api.semanticscholar.org/CorpusID:271310065. ↗
  60. Jiawei Liu et al. “Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation”. In: Thirty-seventh Conference on NeurIPS. 2023. URL : https://openreview.net/forum?id=1qvx610Cu7. ↗
  61. Jiawei Liu et al. “Repoqa: Evaluating long context code understanding”. In: arXiv preprint arXiv:2406.06025 (2024). — [arXiv:2406.06025](https://arxiv.org/abs/2406.06025) find
  62. Jingyuan Liu et al. Muon is Scalable for LLM Training. 2025. arXiv: 2502.16982 [cs.LG]. URL : https://arxiv.org/abs/2502.16982. ↗
  63. Enzhe Lu et al. MoBA: Mixture of Block Attention for Long-Context LLMs. 2025. arXiv: 2502.13189 [cs.LG]. URL : https://arxiv.org/abs/2502.13189. ↗
  64. William Merrill, Jackson Petty, and Ashish Sabharwal. “The illusion of state in state-space models”. In: arXiv preprint arXiv:2404.08819 (2024). find
  65. William Merrill and Ashish Sabharwal. “The Parallelism Tradeoff: Limitations of Log-Precision Transformers”. In: Transactions of the Association for Computational Linguistics 11 (2023), pp. 531–545. DOI : 10.1162/tacl_a_00562. URL : https://aclanthology.org/2023.tacl-1.31/. ↗
  66. MiniMax et al. MiniMax-01: Scaling Foundation Models with Lightning Attention. 2025. arXiv: 2501.08313 [cs.CL].
  67. Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention. 2024. arXiv: 2404.07143 [cs.CL]. find
  68. Tsendsuren Munkhdalai and Adam Trischler. Metalearning with Hebbian Fast Weights. 2018. arXiv: 1807. 05076 [cs.NE]. URL : https://arxiv.org/abs/1807.05076. — [arXiv:1807.05076](https://arxiv.org/abs/1807.05076) ↗
  69. Tsendsuren Munkhdalai et al. “Metalearned Neural Memory”. In: ArXiv abs/1907.09720 (2019). URL : https://api.semanticscholar.org/CorpusID:198179407. — [arXiv:1907.09720](https://arxiv.org/abs/1907.09720) ↗
  70. Long Ouyang et al. “Training language models to follow instructions with human feedback”. In: Advances in NeurIPS 35 (2022), pp. 27730–27744.
  71. Bo Peng et al. RWKV-7 "Goose" with Expressive Dynamic State Evolution. 2025. arXiv: 2503.14456 [cs.CL].
  72. Bowen Peng et al. “Yarn: Efficient context window extension of large language models”. In: arXiv preprint arXiv:2309.00071 (2023). find
  73. Piotr Pi˛ekos, Róbert Csordás, and Jürgen Schmidhuber. “Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing”. In: arXiv preprint arXiv:2505.00315 (2025). — [arXiv:2505.00315](https://arxiv.org/abs/2505.00315) find
  74. Aske Plaat et al. “Reasoning with large language models, a survey”. In: CoRR (2024). — Large Language Models: A Survey
  75. Ofir Press, Noah Smith, and Mike Lewis. “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation”. In: Proceedings of ICLR. 2022. URL : https://openreview.net/forum?id=R8sQPpGCv0. ↗
  76. Krishna C. Puvvada et al. SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling. 2025. arXiv: 2504.08719 [cs.CL]. — [arXiv:2504.08719](https://arxiv.org/abs/2504.08719) find
  77. Zhen Qin et al. HGRN2: Gated Linear RNNs with State Expansion. 2024. arXiv: 2404.07904 [cs.CL]. — [arXiv:2404.07904](https://arxiv.org/abs/2404.07904) find
  78. Zhen Qin et al. TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer. 2024. arXiv: 2307.14995 [cs.CL].
  79. Zihan Qiu et al. Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free. 2025. arXiv: 2505.06708 [cs.CL]. — [arXiv:2505.06708](https://arxiv.org/abs/2505.06708) find
  80. Xiaoye Qu et al. “A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond”. In: arXiv preprint arXiv:2503.21614 (2025).
  81. Qwen Team. Qwen3-Next: Towards Ultimate Training & Inference Efficiency. Accessed: 2025-10-27. Sept. 2025. find
  82. David Rein et al. “Gpqa: A graduate-level google-proof q&a benchmark”. In: First Conference on Language Modeling. 2024.
  83. Keisuke Sakaguchi et al. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. 2019. arXiv: 1907.10641 [cs.CL]. URL : https://arxiv.org/abs/1907.10641. ↗
  84. Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. “Linear Transformers Are Secretly Fast Weight Programmers”. In: Proceedings of ICML. Ed. by Marina Meila and Tong Zhang. PMLR, 2021, pp. 9355–9366. URL : https://proceedings.mlr.press/v139/schlag21a.html. ↗
  85. Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. Learning Associative Inference Using Fast Weight Memory. 2021. arXiv: 2011.07831 [cs.LG]. URL : https://arxiv.org/abs/2011.07831. — [arXiv:2011.07831](https://arxiv.org/abs/2011.07831) ↗
  86. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-Attention with Relative Position Representations. 2018. arXiv: 1803.02155 [cs.CL].
  87. Jianlin Su. Linear Attention: A Brief History of Imitation, Innovation, and Feedback. June 2025. URL : https://kexue.fm/archives/11033. ↗
  88. Jianlin Su et al. “Roformer: Enhanced transformer with rotary position embedding”. In: Neurocomputing 568 (2024), p. 127063.
  89. Weigao Sun et al. Speed Always Wins: A Survey on Efficient Architectures for Large Language Models. 2025. arXiv: 2508.09834 [cs.CL]. URL : https://arxiv.org/abs/2508.09834. — [arXiv:2508.09834](https://arxiv.org/abs/2508.09834) ↗
  90. Yu Sun et al. “Learning to (Learn at Test Time): RNNs with Expressive Hidden States”. In: ArXiv abs/2407.04620 (2024). URL : https://api.semanticscholar.org/CorpusID:271039606. ↗
  91. Yutao Sun et al. “Efficient attention mechanisms for large language models: A survey”. In: arXiv preprint arXiv:2507.19595 (2025). — [arXiv:2507.19595](https://arxiv.org/abs/2507.19595)
  92. Yutao Sun et al. Retentive Network: A Successor to Transformer for Large Language Models. 2023. arXiv: 2307.08621 [cs.CL].
  93. Yutao Sun et al. You Only Cache Once: Decoder-Decoder Architectures for Language Models. 2024. arXiv: 2405.05254 [cs.CL]. URL : https://arxiv.org/abs/2405.05254. ↗
  94. Mirac Suzgun et al. “Challenging big-bench tasks and whether chain-of-thought can solve them”. In: arXiv preprint arXiv:2210.09261 (2022).
  95. Kimi Team et al. Kimi k1.5: Scaling Reinforcement Learning with LLMs. 2025. arXiv: 2501.12599 [cs.AI]. URL : https://arxiv.org/abs/2501.12599. ↗
  96. MiniCPM Team et al. MiniCPM4: Ultra-Efficient LLMs on End Devices. 2025. arXiv: 2506.07900 [cs.CL]. URL : https://arxiv.org/abs/2506.07900. ↗
  97. Tencent Hunyuan Team et al. “Hunyuan-turbos: Advancing large language models through mamba-transformer synergy and adaptive chain-of-thought”. In: arXiv preprint arXiv:2505.15431 (2025).
  98. Hugo Touvron et al. LLaMA: Open and Efficient Foundation Language Models. 2023. arXiv: 2302.13971 [cs.CL].
  99. Ashish Vaswani et al. “Attention is All you Need”. In: Advances in NeurIPS. Ed. by I. Guyon et al. Curran Associates, Inc., 2017. URL : https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. ↗
  100. Roger Waleffe et al. An Empirical Study of Mamba-based Language Models. 2024. arXiv: 2406 . 07887 [cs.LG]. URL : https://arxiv.org/abs/2406.07887. ↗
  101. Sinong Wang et al. Linformer: Self-Attention with Linear Complexity. 2020. arXiv: 2006.04768 [cs.LG]. find
  102. Yaoyu Wang. Understanding DeltaNet from the Perspective of Inference Frameworks. May 2025. URL : https://yywangcs.notion.site/DeltaNet-1fefc9f5d80580a496f8eb406a496f09. ↗
  103. Yubo Wang et al. “Mmlu-pro: A more robust and challenging multi-task language understanding benchmark”. In: Advances in NeurIPS 37 (2024), pp. 95266–95290.
  104. Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu. “Rnns are not transformers (yet): The key bottleneck on in-context retrieval”. In: arXiv preprint arXiv:2402.18510 (2024). find
  105. Colin White et al. “Livebench: A challenging, contamination-free llm benchmark”. In: arXiv preprint arXiv:2406.19314 4 (2024). find
  106. Yuhuai Wu et al. “Memorizing transformers”. In: arXiv preprint arXiv:2203.08913 (2022). find
  107. Guangxuan Xiao et al. “Efficient streaming language models with attention sinks”. In: arXiv preprint arXiv:2309.17453 (2023). find
  108. Wenhan Xiong et al. Effective Long-Context Scaling of Foundation Models. 2023. arXiv: 2309.16039 [cs.CL]. URL : https://arxiv.org/abs/2309.16039. ↗
  109. Ruyi Xu et al. “Xattention: Block sparse attention with antidiagonal scoring”. In: arXiv preprint arXiv:2503.16428 (2025).
  110. Bowen Yang et al. Rope to Nope and Back Again: A New Hybrid Attention Strategy. 2025. arXiv: 2501.18795 [cs.CL]. URL : https://arxiv.org/abs/2501.18795. — [arXiv:2501.18795](https://arxiv.org/abs/2501.18795) ↗
  111. Songlin Yang, Jan Kautz, and Ali Hatamizadeh. “Gated Delta Networks: Improving Mamba2 with Delta Rule”. In: Proceedings of ICLR. 2025. URL : https://openreview.net/forum?id=r8H7xhYPwz. ↗
  112. Songlin Yang and Bailin Wang. “Parallelizing Linear Transformers with the Delta Rule over Sequence Length”. In: ArXiv abs/2406.06484 (2024). URL : https://api.semanticscholar.org/CorpusID:270371554. ↗
  113. Songlin Yang and Yu Zhang. FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism. 2024. URL : https://github.com/fla-org/flash-linear-attention. ↗
  114. Songlin Yang et al. “Gated Linear Attention Transformers with Hardware-Efficient Training”. In: Proceedings of ICML. PMLR, 2024. find
  115. Songlin Yang et al. “PaTH Attention: Position Encoding via Accumulating Householder Transformations”. In: arXiv preprint arXiv:2505.16381 (2025). — [arXiv:2505.16381](https://arxiv.org/abs/2505.16381) find
  116. Feng Yao et al. Your Efficient RL Framework Secretly Brings You Off-Policy RL Training. Aug. 2025. URL : https://fengyao.notion.site/off-policy-rl. ↗
  117. Morris Yau et al. “Sequential-Parallel Duality in Prefix Scannable Models”. In: arXiv preprint arXiv:2506.10918 (2025). — [arXiv:2506.10918](https://arxiv.org/abs/2506.10918) find
  118. Howard Yen et al. “HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly”. In: International Conference on Learning Representations (ICLR). 2025. find
  119. Jingyang Yuan et al. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. 2025. arXiv: 2502.11089 [cs.CL]. URL : https://arxiv.org/abs/2502.11089. ↗
  120. Manzil Zaheer et al. “Big bird: Transformers for longer sequences”. In: Advances in NeurIPS 33 (2020), pp. 17283–17297.
  121. Rowan Zellers et al. “HellaSwag: Can a Machine Really Finish Your Sentence?” In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019. find
  122. Biao Zhang and Rico Sennrich. “Root mean square layer normalization”. In: Advances in NeurIPS 32 (2019). find
  123. Michael Zhang et al. “Lolcats: On low-rank linearizing of large language models”. In: arXiv preprint arXiv:2410.10254 (2024). — [arXiv:2410.10254](https://arxiv.org/abs/2410.10254) find
  124. Yu Zhang et al. Gated Slot Attention for Efficient Linear-Time Sequence Modeling. 2024. arXiv: 2409.07146 [cs.CL]. — [arXiv:2409.07146](https://arxiv.org/abs/2409.07146) find
  125. Shu Zhong et al. “Understanding Transformer from the Perspective of Associative Memory”. In: arXiv preprint arXiv:2505.19488 (2025). — [arXiv:2505.19488](https://arxiv.org/abs/2505.19488) find
  126. Jingwei Zuo et al. Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance. 2025. arXiv: 2507.22448 [cs.CL]. URL : https://arxiv.org/abs/2507.22448. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-linear-expressive-efficient/