Elamur Research⌕
☰
home / papers / 2026 / RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction

RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction

Paper · 2026 · Sihao Liu, YuFan Xiong, Zhonghua Jiang et al.

Summary

Multimodal LLMs that process long or high-resolution visual inputs accumulate a very large KV cache. Most compression methods evict tokens on the assumption that tokens with low attention now will stay unimportant. The authors argue this assumption fails for images in two ways. Some visual tokens look irrelevant early in decoding but become crucial later ("deferred importance"). Dropping individual patches also breaks the spatial coherence of the image.

RetentiveKV addresses this by folding tokens marked for eviction into a fixed-size, State Space Model–style recurrent memory instead of deleting them. Attention entropy from text to visual tokens estimates which low-attention tokens may still matter, and this signal guides the state transitions. Two kinds of state are kept: visual-dominant states, which preserve patch layout, and recall-oriented states, which hold longer-range semantics. A query-conditioned retrieval step brings stored content back into use when it becomes relevant.

On multimodal benchmarks, the method reports 5.0× KV compression and 1.5× faster decoding. It also holds up better than eviction baselines at very small cache budgets, which matters for running MLLMs on limited hardware.

Key points

  • Core reformulation: KV eviction is recast as continuous memory evolution via a linear-attention/SSM recurrence with decay factor γ. Absorbed tokens therefore add only constant memory overhead.
  • Ablation components: the method combines an entropy-driven retention metric, modality-separated states, and Query-Conditioned Retrieval. Removing state retention and retrieval causes the largest drop, about 4.6–5.4%. Using a single unified state costs about 3.1% on average.
  • Efficiency (single A100, versus full cache): at a 5% budget, memory falls from 2.24 GiB to 0.23 GiB. Latency drops from 32.15 to 18.42 ms/token, roughly 1.75×; the abstract's headline figure is 1.5×.
  • Accuracy: the method gains about +2.1 over importance-based baselines on DocVQA and TextVQA. Experiments on Qwen3-VL-8B with budgets from 5% to 60% show it beating SnapKV and Meda. At a 5% budget on Text Needle-In-A-Haystack, its score is double Meda's.
  • Stated limitation: testing covers models of at most 8B parameters. Extensions to models above 30B and to audio/video are left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China201 works · 17,172 citations🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations

Also on the byline: Shanghai Advanced Research Institute.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 811 worksEfficient models & compression · 297 works

Cite

Cite as Sihao Liu, YuFan Xiong, Zhonghua Jiang, Zhaode Wang, Chengfei lv and Shengyu Zhang. 2026. RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.934

@inproceedings{liu2026retentivekv,
  title = {{RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction}},
  author = {Sihao Liu, YuFan Xiong, Zhonghua Jiang, Zhaode Wang, Chengfei lv and Shengyu Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.934},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.934}
}

References

26 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Baddeley. 2000. The episodic buffer: a new component of working memory? Trends in cognitive sciences, 4(11):417–423. find
  2. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024a. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer.
  3. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and 1 others. 2024b. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330.
  4. Albert Gu and Tri Dao. 2024. Mamba: Linear-time sequence modeling with selective state spaces. In First conference on language modeling. find
  5. Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. 2020. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474– 1487. find
  6. Albert Gu, Karan Goel, and Christopher Ré. 2021. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396. find
  7. Ahmadreza Jeddi, Negin Baghbanzadeh, Elham Dolatabadi, and Babak Taati. 2025. Similarity-aware token pruning: Your vlm but faster. arXiv preprint arXiv:2503.11549. — [arXiv:2503.11549](https://arxiv.org/abs/2503.11549) find
  8. Zhonghua Jiang, Kui Chen, Kunxi Li, Keting Yin, Yiyun Zhou, Zhaode Wang, Chengfei Lv, and Shengyu Zhang. 2025a. Acckv: Towards efficient audio-video llms inference via adaptive-focusing and cross-calibration kv cache optimization. Preprint, arXiv:2511.11106. — [arXiv:2511.11106](https://arxiv.org/abs/2511.11106) find
  9. Zhonghua Jiang, Kunxi Li, Yiyun Zhou, Sihao Liu, Zhaode Wang, Chengfei lv, and Shengyu Zhang. 2025b. Purekv: Plug-and-play kv cache optimization with spatial-temporal sparse attention for vision-language large models. Preprint, arXiv:2510.25600. — [arXiv:2510.25600](https://arxiv.org/abs/2510.25600) find
  10. Kunxi Li, Zhonghua Jiang, Zhouzhou Shen, Zhaode-Wang ZhaodeWang, Chengfei Lv, Shengyu Zhang, Fan Wu, and Fei Wu. 2025a. Madakv: Adaptive modality-perception kv cache eviction for efficient multimodal long-context inference. In Proceedings of the 63rd Annual Meeting of the Association for Computation… find
  11. Kunxi Li, Yufan Xiong, Zhonghua Jiang, Yiyun Zhou, Zhaode Wang, Chengfei Lv, and Shengyu Zhang. 2025b. Flowmm: Cross-modal information flow guided kv cache merging for efficient multimodal context inference. Preprint, arXiv:2511.05534. — [arXiv:2511.05534](https://arxiv.org/abs/2511.05534) find
  12. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947–22970.
  13. Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. 2025. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 5334–5342. find
  14. Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems, 36:5… find
  15. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR).
  16. Minesh Mathew, Dimosthenis Karatzas, R Manmatha, and CV Jawahar. 2020. Docvqa: A dataset for vqa on document images. corr abs/2007.00398 (2020). arXiv preprint arXiv:2007.00398.
  17. Jiahe Shi, Zhengqi Gao, Ching-Yun Ko, and Duane Boning. 2025. Earl: Entropy-aware rl alignment of llms for reliable rtl code generation. arXiv preprint arXiv:2511.12033. — [arXiv:2511.12033](https://arxiv.org/abs/2511.12033) find
  18. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326. find
  19. Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. 2024. Milebench: Benchmarking mllms in long context. arXiv preprint arXiv:2404.18532. — [arXiv:2404.18532](https://arxiv.org/abs/2404.18532) find
  20. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. 2023. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621.
  21. Zhongwei Wan, Hui Shen, Xin Wang, Che Liu, Zheda Mai, and Mi Zhang. 2025. Meda: Dynamic kv cache allocation for efficient multimodal long-context inference. arXiv preprint arXiv:2502.17599. find
  22. Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. 2024. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference. arXiv preprint arXiv:2406.18139. find
  23. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453. find
  24. Jing Xiong, Jianghan Shen, Fanghua Ye, Chaofan Tao, Zhongwei Wan, Jianqiao Lu, Xun Wu, Chuanyang Zheng, Zhijiang Guo, Min Yang, and 1 others. 2025. Uncomp: Can matrix entropy uncover sparsity?—a compressor design from an uncertainty-aware perspective. In Proceedings of the 2025 Conference on Empiric… find
  25. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vi…
  26. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, and 1 others. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:346… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-retentivekv-state-space-memory-uncertainty/