Elamur Research⌕
☰
home / papers / 2026 / Thesis Proposal: Efficient KV Cache Reuse for Multi-Document Retrieval-Augmented Generation

Thesis Proposal: Efficient KV Cache Reuse for Multi-Document Retrieval-Augmented Generation

Paper · 2026 · Zhipeng Zhang and Dmitry Ilvovsky

Summary

In retrieval-augmented generation (RAG), the prefill pass over long retrieved documents grows quadratically with input length. Standard KV caching only accelerates decoding. Because the same documents are often retrieved again across queries and sessions, storing and reusing their per-layer KV cache is attractive. Naive reuse, however, breaks correctness in three ways:

  • cached entries carry outdated positional encodings;
  • they were never conditioned on the new preceding context (prompt, query, earlier documents);
  • deviations can compound through residual layers.

This HSE University thesis proposal argues that existing systems such as RAGCache and CacheBlend work in practice but rely on heuristics, with no guarantees on when reuse is safe. The authors frame reuse as a controlled structural perturbation of full recomputation. Their planned framework combines three components:

  • a Position Propagation Operator for exact realignment under RoPE and ALiBi;
  • Softmax-Jacobian-based bounds on attention shifts;
  • contraction analysis with adaptive gating and selective recomputation to limit layer-wise drift.

No experimental results are reported yet. The planned evaluation covers standard benchmarks, noise robustness tests and domain workloads such as legal QA. The work matters because it would give high-stakes deployments principled safety limits without retraining or architectural changes.

Key points

  • Three core failure modes of document-level KV reuse are identified: position misalignment, context misalignment, and error amplification across layers.
  • Prior work cited for context: RAGCache reports up to 4× lower TTFT and 2.1× throughput versus vLLM+Faiss. CacheBlend reports 2.2–3.3× lower TTFT and 2.8–5× throughput versus full recomputation.
  • The roadmap has three phases: theoretical foundation, system integration (measuring layer-wise contraction factors under varying γ and Top-R settings), and validation. Each phase has explicit tolerance-based criteria for moving on.
  • Stated scope limits: causal decoder-only models, RoPE/ALiBi/absolute position encodings, and GQA attention only. MLA (e.g., DeepSeek-V2) is left for future work.
  • Funding came from the HSE University Basic Research Program and a Huawei grant, with computation on HSE HPC facilities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Research University Higher School of EconomicsMoscow · Russia24 works · 39,585 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Information retrieval & search · 552 worksEfficient models & compression · 297 works

Cite

Cite as Zhipeng Zhang and Dmitry Ilvovsky. 2026. Thesis Proposal: Efficient KV Cache Reuse for Multi-Document Retrieval-Augmented Generation. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-srw.11

@inproceedings{zhang2026thesis,
  title = {{Thesis Proposal: Efficient KV Cache Reuse for Multi-Document Retrieval-Augmented Generation}},
  author = {Zhipeng Zhang and Dmitry Ilvovsky},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.11},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.11}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895– 4…
  2. DeepSeek-AI. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint, arXiv:2405.04434. find
  3. Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, Yuqi Tang, Ziwen Xu, Shumin Deng, Yunzhi Yao, Mengru Wang, Shuofei Qiao, Huajun Chen, and Ningyu Zhang. 2025. Lightmem: Lightweight and efficient memory-augmented generation. arXiv preprint, arXiv:2510.18866. — [arXiv:2510.18866](https://arxiv.org/abs/2510.18866) find
  4. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-augmented generation for large language models: A survey. arXiv preprint, arXiv:2312.10997.
  5. Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Shufan Liu, Xuanzhe Liu, and Xin Jin. 2025. Ragcache: Efficient knowledge caching for retrieval-augmented generation. ACM Transactions on Computer Systems, 44(1):1–27. find
  6. Anastassia Kornilova and Vlad Eidelman. 2019. Bill-Sum: A corpus for automatic summarization of US legislation. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 48–56. find
  7. Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. 2025. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. arXiv preprint, arXiv:2409.12941. find
  8. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. arXiv preprint, arXiv:2309.06180.
  9. Chien-Yu Lin, Keisuke Kamahori, Yiyu Liu, Xiaoxiang Shi, Madhav Kashyap, Yile Gu, Rulin Shao, Zihao Ye, Kan Zhu, Stephanie Wang, Arvind Krishnamurthy, Rohan Kadekodi, Luis Ceze, and Baris Kasikci. 2025. TeleRAG: Efficient retrieval-augmented generation inference with lookahead retrieval. arXiv prepr… find
  10. Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. 2024. CacheGen: KV cache compression and streaming for fast large language model serving. arXiv prepr… — [arXiv:2310.07240](https://arxiv.org/abs/2310.07240) find
  11. Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint, arXiv:2108.12409. find
  12. Xianbiao Qi, Jianan Wang, and Lei Zhang. 2023. Understanding optimization of deep learning via jacobian matrix and lipschitz constant. arXiv preprint, arXiv:2306.09338. — [arXiv:2306.09338](https://arxiv.org/abs/2306.09338) find
  13. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. RoFormer: Enhanced transformer with rotary position embedding. arXiv preprint, arXiv:2104.09864.
  14. Robert Friel, Masha Belyi, and Atindriyo Sanyal. 2025. Ragbench: Explainable benchmark for retrieval-augmented generation systems. arXiv preprint, arXiv:2407.11005. — [arXiv:2407.11005](https://arxiv.org/abs/2407.11005) find
  15. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2023. Attention is all you need. arXiv preprint, arXiv:1706.03762.
  16. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. arXiv preprint, arXiv:2309.17453. find
  17. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. 2020. On layer normalization in the transformer architecture. arXiv preprint, arXiv:2002.04745. find
  18. Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast large language model serving for RAG with cached knowledge fusion. arXiv preprint, arXiv:2405.16444. — [arXiv:2405.16444](https://arxiv.org/abs/2405.16444) find
  19. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, and 1 others. 2025. A survey of large language models. arXiv…
  20. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2023. Efficiently programming large language models using SGLang. arXiv preprint, arXiv:2312.07104.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-thesis-proposal-efficient-kv-cache/