Elamur Research⌕
☰
home / papers / 2026 / SpecCache: Speculative KV Cache Reuse for Efficient RAG Serving

SpecCache: Speculative KV Cache Reuse for Efficient RAG Serving

Paper · 2026 · Zijian Wen, Tao Zhang, Shuangwu Chen et al.

Summary

RAG prompts grow long when many retrieved chunks are added, which makes the prefill stage slow and delays the first token. Reusing precomputed per-chunk KV caches saves computation but loses cross-chunk attention. Existing selective-reuse methods recompute only some tokens, but they pick those tokens with weak signals: EPIC uses fixed positions, and CacheBlend uses first-layer deviations.

The authors show that the gap between cached and fully recomputed KV states is small in shallow layers and grows with depth. They also find that the deep layers of a small speculative model largely agree with the large target model on which tokens matter. SpecCache, a training-free framework implemented on vLLM, works in three steps:

  • It scores tokens using deep-layer hidden-state norms from the small model.
  • It expands the selected tokens into complete semantic spans.
  • It maps the selection across tokenizers, so the target model recomputes only those tokens.

On six tasks with three model architectures, it outperforms CacheBlend and EPIC in quality while staying close to full recomputation. It also substantially lowers latency, which makes accurate cache reuse more practical for real-time RAG serving.

Key points

  • Speed vs. full KV recomputation: TTFT is 2.17–3.95× faster, inference throughput is 2.7–5.2× higher, and quality loss is negligible.
  • Quality vs. baselines: task scores are 2.1–23.2% higher than CacheBlend and 3.4–30.08% higher than EPIC.
  • Importance scoring: each token's score is its ℓ2 hidden-state norm, averaged over the last third of the speculative model's layers.
  • Evidence that the small model is a good proxy: Jaccard overlap of the top-20% tokens on 2WikiMQA is higher between speculative and target models than between the target's own layer 0 and its deeper layers. Models used were Llama3-1B and Qwen2.5-1.5B as speculative models, with Llama3-8B as the target.
  • Choosing the speculative model: larger same-family speculative models (Qwen2.5, 0.5B–3B) give better results, while a cross-family LLaMA3-3B does worse. A MuSiQue case study shows SpecCache recovering the bridge entity that links evidence across chunks.
  • Stated limitations: the speculative models are off-the-shelf rather than fine-tuned, and evaluation covers only 7B–13B models on academic benchmarks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Science and Technology of ChinaHefei · China145 works · 114,409 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 297 works

Cite

Cite as Zijian Wen, Tao Zhang, Shuangwu Chen, Shenghao Ye, Yu Guo, Qirui Chen, Jingxian Shuai, Yunpeng Hou, Huasen He and Jianyang. 2026. SpecCache: Speculative KV Cache Reuse for Efficient RAG Serving. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.859

@inproceedings{wen2026speccache,
  title = {{SpecCache: Speculative KV Cache Reuse for Efficient RAG Serving}},
  author = {Zijian Wen, Tao Zhang, Shuangwu Chen, Shenghao Ye, Yu Guo, Qirui Chen, Jingxian Shuai, Yunpeng Hou, Huasen He and Jianyang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.859},
  url = {https://doi.org/10.18653/v1/2026.acl-long.859}
}

References

30 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Shubham Agarwal, Sai Sundaresan, Subrata Mitra, De-babrata Mahapatra, Archit Gupta, Rounak Sharma, Nirmal Joshua Kapu, Tong Yu, and Shiv Saini. 2025. Cache-craft: Managing chunk-caches for efficient retrieval-augmented generation. Proceedings of the ACM on Management of Data, 3(3):1–28. find
  3. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, and 1 others. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd annual meeting of the association for computational lin…
  4. Eleftheria Briakou, Colin Cherry, and George Foster. 2023. Searching for needles in a haystack: On the role of incidental bilingualism in palm’s translation capability. In The 61st Annual Meeting Of The Association For Computational Linguistics. find
  5. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. find
  6. Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations.
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  8. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2(1).
  9. In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6:325–338.
  10. Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. 2024. Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21158–21166. find
  11. Junhao Hu, Wenrui Huang, Weidong Wang, Haoyi Wang, Hao Feng, Xusheng Chen, Yizhou Shan, Tao Xie, and 1 others. 2025. Epic: Efficient position-independent caching for serving large language models. In Forty-second International Conference on Machine Learning. find
  12. Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong Park. 2024. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. In NAACL-HLT.
  13. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  14. Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Shufan Liu, Xuanzhe Liu, and Xin Jin. 2025. Ragcache: Efficient knowledge caching for retrieval-augmented generation. ACM Transactions on Computer Systems, 44(1):1–27. find
  15. Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  16. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626.
  17. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR. find
  18. Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. 2022. A survey on retrieval-augmented text generation. arXiv preprint arXiv:2202.01110. — [arXiv:2202.01110](https://arxiv.org/abs/2202.01110) find
  19. Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. Eagle: speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference on Machine Learning, pages 28935–28948.
  20. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  21. Jingyu Liu, Beidi Chen, and Ce Zhang. 2025. Speculative prefill: Turbocharging ttft with lightweight and training-free token importance estimation. In Forty-second International Conference on Machine Learning. find
  22. Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, and 1 others. 2024. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 38–5… find
  23. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  24. Nadav Timor, Jonathan Mamou, Daniel Korat, Moshe Berchansky, Gaurav Jain, Oren Pereg, Moshe Wasserblat, and David Harel. 2025. Accelerating llm inference with lossless speculative decoding algorithms for heterogeneous vocabularies. In Forty-second International Conference on Machine Learning. find
  25. Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. 2023. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3909–3925.
  26. Huan Yang, Renji Zhang, Mingzhe Huang, Weijun Wang, Yin Tang, Yuanchun Li, Yunxin Liu, and Deyu Zhang. 2025a. Kvshare: An llm service system with efficient and effective multi-tenant kv cache reuse. arXiv preprint arXiv:2503.16525. — [arXiv:2503.16525](https://arxiv.org/abs/2503.16525) find
  27. Jingbo Yang, Bairu Hou, Wei Wei, Yujia Bao, and Shiyu Chang. 2025b. Kvlink: Accelerating large language models via efficient kv cache reuse. arXiv preprint arXiv:2502.16002. — [arXiv:2502.16002](https://arxiv.org/abs/2502.16002) find
  28. Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. Cacheblend: Fast large language model serving for rag with cached knowledge fusion. In Proceedings of the Twentieth European Conference on Computer Systems, pages 94–109. find
  29. Meta. 2024. meta-llama/llama-3.2-1b-instruct. Hugging Face model card. Accessed: 2025-12-20. serving. In 18th USENIX Symposium on Operating Systems Design and Implementation, pages 193–210. find
  30. Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021. Generation-augmented retrieval for open-domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confere…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wen-2026-speccache-speculative-kv-cache-reuse/