Elamur Research⌕
☰
home / papers / 2024 / Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

Paper · 2024 · Ruoyu Qin, Zheming Li, Weiran He et al.

Summary

Mooncake is the production serving system behind Kimi, Moonshot AI's LLM service. It tackles a constrained optimization problem: maximize useful throughput while still meeting latency targets for time to first token (TTFT) and time between tokens (TBT), and do so under frequent heavy overload. The design runs prefill and decoding on separate GPU pools. It also pools the otherwise idle CPU memory, DRAM and SSD of the cluster into a distributed KVCache store, connected over RDMA. A global scheduler called Conductor routes each request based on where its reusable KVCache lives. Other components include:

  • chunked pipeline parallelism for long prompts,
  • layer-wise streaming of KVCache from prefill to decoding nodes,
  • heuristic replication of popular cache blocks,
  • an early-rejection policy that uses load prediction.

Experiments replay real traces on a dummy LLaMA2-70B-style model. In some simulated settings throughput rises by up to 525% while SLOs are still met, and on real workloads Kimi serves 75% more requests. The work matters because most serving research assumes every request will eventually be served, while real providers must decide which requests to drop.

Key points

  • Architecture: Prefill and decoding pools are separate, and a KVCache pool spread over CPU, DRAM and SSD sits between them. For each request, Conductor picks a prefill/decode instance pair, loads reusable cache, runs prefill in chunks or layers while streaming KVCache onward, and adds the request to continuous batching on the decode node.
  • Results: Up to 525% higher throughput under SLOs in simulated scenarios, and 75% more requests handled on Kimi's real workloads.
  • Evaluation setup:
    • The model is a dummy with the LLaMA2-70B architecture.
    • The SLOs are TTFT P90 = 10× and TBT P90 = 5×, measured relative to an interference-free single request.
    • The released trace contains only arrival times, input/output token counts and remapped block hashes. It is published at github.com/kvcache-ai/Mooncake.
  • Cache reuse:
    • In Kimi's traces, at most about 55% of KVCache could be reused, even assuming unlimited storage and unlimited TTFT budget.
    • In some scenarios reuse reaches about 90%, for example the chat-to-paper service papers.cool.
  • Long prompts: Chunked pipeline parallelism (CPP) spreads one long prompt across several nodes. Compared with sequence parallelism, it uses less network traffic and needs less frequent elastic rescaling.

Interesting ideas

  • I1. Schedule around the cache, not just compute: Treating the location of reusable intermediate state as the main scheduling input generalizes beyond LLMs. Any pipeline with expensive reusable artifacts, such as embeddings or preprocessed corpora, could route jobs to where the data already sits.
  • I2. Use idle memory as a shared cache tier: GPU servers often leave CPU, DRAM and SSD underused. Turning that spare capacity into a cluster-wide cache is a cheap way to save recomputation without buying new hardware.
  • I3. Reject early, but with prediction: If a request cannot get a decoding slot after prefill, dropping it early saves wasted work. However, a naive version of this policy caused the load to oscillate, so the authors predicted short-term future load instead. This lesson applies to any admission-control system under overload.
  • I4. Copy hot items by heuristic: Popular cache blocks are replicated to several nodes to avoid fetch congestion, and cold blocks are swapped out. A heuristic is used instead of precise usage forecasts, which is a practical pattern for any distributed cache with skewed demand.
  • I5. Share sanitized traces: Releasing only timing, token counts and remapped hashes lets others reproduce workload behavior without exposing user content. This is a useful template for sharing production data that is private.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🏢Alibaba GroupHangzhou · China201 works · 17,172 citations🏢Moonshot AIBeijing · China2 works · 241 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksInformation Systems · 1,730 worksHardware and Architecture · 167 worksComputer Networks and Communications · 100 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng and Xinran Xu. 2024. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. FAST 2025.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2407.00079

@inproceedings{qin2024mooncake,
  title = {{Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving}},
  author = {Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng and Xinran Xu},
  booktitle = {FAST 2025},
  year = {2024},
  doi = {10.1145/3773772},
  eprint = {2407.00079},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3773772}
}

References

69 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. OpenAI. Introducing chatgpt. https://openai.com/blog/chatgpt, 2022. ↗
  2. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv.org/abs/2307.09288, 2023. ↗
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  4. Charles Packer, Vivian Fang, Shishir G Patil, Kevin Lin, Sarah Wooders, and Joseph E Gonzalez. Memgpt: Towards llms as operating systems. arXiv preprint arXiv:2310.08560, 2023. find
  5. Moonshot AI. Kimi. https://kimi.moonshot.cn, 2023. ↗
  6. NVIDIA. Nvidia h100 tensor core gpu architecture. https://resources.nvidia.com/en-us-tensor-core, 2022. ↗
  7. Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. arXiv preprint arXiv:2311.18677, 2023.
  8. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. arXiv preprint arXiv:2401.09670, 2024.
  9. Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al. Inference without interference: Disaggregate llm inference for mixed downstream workloads. arXiv preprint arXiv:2401.11181, 2024. — [arXiv:2401.11181](https://arxiv.org/abs/2401.11181) find
  10. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  11. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
  12. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, 2022. — Orca: A Distributed Serving System for Transformer-Based Generative Models
  13. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626, 2023.
  14. Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism, 2024.
  15. Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in llm inference with sarathi-serve. arXiv preprint arXiv:2403.02310, 2024.
  16. Google. Our next-generation model: Gemini 1.5. https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024, 2024. ↗
  17. Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023.
  18. Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.
  19. William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. Striped attention: Faster ring attention for causal transformers, 2023. find
  20. Dacheng Li, Rulin Shao, Anze Xie, Eric P Xing, Joseph E Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. Lightseq:: Sequence level parallelism for distributed training of long context transformers. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource O… find
  21. Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5:341–353, 2023.
  22. Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2391–2404, 2023.
  23. Jiarui Fang and Shangchun Zhao. Usp: A unified sequence parallelism approach for long context generative ai, 2024. find
  24. Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. Terapipe: Token-level pipeline parallelism for training large-scale language models. In International Conference on Machine Learning, pages 6543–6552. PMLR, 2021. find
  25. OpenAI. Batch api. https://platform.openai.com/docs/guides/batch, 2024. ↗
  26. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Lingui…
  27. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models, 2023.
  28. NVIDIA Corporation. FasterTransformer, 2019. Fastertransformer. find
  29. NVIDIA Corporation. Tensorrt-llm. https://github.com/NVIDIA/TensorRT-LLM, 2023. ↗
  30. Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Perf… find
  31. Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pages 31094–31116. PMLR, 20…
  32. Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920, 2023.
  33. In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. arXiv preprint arXiv:2311.04934, 2023.
  34. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Efficiently programming large language models using sglang. arXiv preprint arXiv:2312.07104, 2023.
  35. Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Attentionstore: Cost-effective attention reuse across multi-turn conversations in large language model serving. arXiv preprint arXiv:2403.19708, 2024.
  36. Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, and Yiying Zhang. Preble: Efficient distributed prompt scheduling for llm serving. https://escholarship.org/uc/item/1bm0k1w0, May 2024. ↗
  37. Xiaoxuan Yang, Bonan Yan, Hai Li, and Yiran Chen. Retransformer: Reram-based processing-in-memory architecture for transformer acceleration. In Proceedings of the 39th International Conference on Computer-Aided Design, ICCAD ’20, New York, NY, USA, 2020. Association for Computing Machinery. find
  38. Ann Franchesca Laguna, Arman Kazemi, Michael Niemier, and X. Sharon Hu. In-memory computing based accelerator for transformer networks for long sequences. In 2021 Design, Automation and Test in Europe Conference & Exhibition (DATE), pages 1839–1844, 2021. find
  39. Wantong Li, Madison Manley, James Read, Ankit Kaul, Muhannad S. Bakir, and Shimeng Yu. H3datten: Heterogeneous 3-d integrated hybrid analog and digital compute-in-memory accelerator for vision transformer self-attention. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 31(10):1592–1… find
  40. Shrihari Sridharan, Jacob R. Stevens, Kaushik Roy, and Anand Raghunathan. X-former: In-memory acceleration of transformers. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 31(8):1223–1233, 2023. find
  41. Negar Akbarzadeh, Sina Darabi, Atiyeh Gheibi-Fetrat, Amir Mirzaei, Mohammad Sadrosadati, and Hamid Sarbazi-Azad. H3dm: A high-bandwidth high-capacity hybrid 3d memory design for gpus. Proc. ACM Meas. Anal. Comput. Syst., 8(1), feb 2024. find
  42. The Korea Economic Daily. Samsung to unveil 3D DRAM in 2025 to lead AI chip market. https://www.kedglobal.com/korean-chipmakers/newsView/ked202404020016, 2023. ↗
  43. Jishen Zhao, Guangyu Sun, Gabriel H. Loh, and Yuan Xie. Optimizing gpu energy efficiency with 3d die-stacking graphics memory and reconfigurable memory interface. ACM Trans. Archit. Code Optim., 10(4), dec 2013. find
  44. Jishen Zhao, Qiaosha Zou, and Yuan Xie. Overview of 3-d architecture design opportunities and techniques. IEEE Design & Test, 34(4):60–68, 2017. https://github.com/NVIDIA/ ↗
  45. Duckhwan Kim, Jaeha Kung, Sek Chai, Sudhakar Yalamanchili, and Saibal Mukhopadhyay. Neurocube: A programmable digital neuromorphic architecture with high-density 3d memory. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pages 380–392, 2016. find
  46. Shaoyuan Chen, Yutong Lin, Mingxing Zhang, and Yongwei Wu. Efficient and economic large language model inference with attention offloading, 2024. find
  47. DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, H… find
  48. Hao Yu, Zelan Yang, Shen Li, Yong Li, and Jianxin Wu. Effectively compress kv heads for llm, 2024. find
  49. Zefan Cai., Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling, 2024.
  50. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi : Plug-and-play 2bit kv cache quantization with streaming asymmetric quantization. 2023. find
  51. Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient kv cache quantization with salient token identification, 2024. find
  52. Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact, 2024. find
  53. Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Minicache: Kv cache compression in depth dimension for large language models, 2024. find
  54. Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters, 2024. find
  55. Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective l 2 norm-based strategy for kv cache compression, 2024. find
  56. Yao Yao, Zuchao Li, and Hai Zhao. Sirllm: Streaming infinite retentive llm, 2024. find
  57. Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference, 2024.
  58. Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J. Nair, Ilya Soloveychik, and Purushotham Kamath. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference, 2024. find
  59. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation, 2024.
  60. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H 2 o: Heavy-hitter oracle for efficient generative inference of large language models, 2023. — H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  61. Zayd Muhammad Kawakibi Zuhri, Muhammad Farid Adilazuarda, Ayu Purwarianti, and Alham Fikri Aji. Mlkv: Multi-layer key-value heads for memory efficient transformer decoding, 2024. find
  62. Haoyi Wu and Kewei Tu. Layer-condensed kv cache for efficient inference of large language models, 2024. find
  63. Character.AI. Optimizing AI Inference at Character.AI. https://research.character.ai/optimizing-inference/, 2023. ↗
  64. Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models, 2024. find
  65. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. find
  66. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, F… find
  67. Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. find
  68. Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman,… — Jamba: A Hybrid Transformer-Mamba Language Model
  69. Aleksandar Botev, Soham De, Samuel L Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Sertan Girgin, Olivier Bachem, Alek Andreev, Kathleen Kenealy, Thomas Mesnard, Cassidy Hardin, Sury… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qin-2024-mooncake-kvcache-centric-disaggregated-architecture/