Elamur Research⌕
☰
home / papers / 2026 / HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding

HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding

Paper · 2026 · Haowei Zhang, Shudong Yang, Jinlan Fu et al.

Summary

Multimodal LLMs handle offline video well, but streaming video is harder. A system must stay accurate, answer quickly, and keep GPU memory bounded, even though future frames and questions are unknown. Earlier cache-based methods such as ReKV and LiveVLM move KV entries off the GPU and retrieve them when a question arrives, which slows responses. HERMES is a training-free, plug-and-play alternative. It is based on an attention analysis that treats the KV cache as layered memory:

  • Shallow layers behave like sensory memory and favour the newest frames.
  • Middle layers act as working memory.
  • Deep layers keep sparse per-frame anchor tokens, functioning as long-term memory.

The method has three parts: layer-specific token eviction, cross-layer memory smoothing, and position re-indexing. The compact cache is reused directly, so nothing extra is computed when a query arrives. Compared with the previous training-free SOTA, time-to-first-token is about 10× lower, and GPU memory stays constant. Accuracy matches or beats uniform sampling while using up to 68% fewer video tokens. This makes HERMES practical for real-time video assistants with limited hardware.

Key points

  • Attention study: The authors analysed 300 randomly sampled VideoMME video–question pairs with LLaVA-OV-7B. In deep layers, attention peaks recur every 196 tokens, which equals one frame's token count. The authors read these peaks as frame-level anchors.
  • Eviction scoring by layer:
    • Shallow layers score tokens with an exponential forgetting curve inspired by Ebbinghaus.
    • Deep layers score tokens by attention to a generic guidance prompt that stands in for the unknown query.
    • In deep layers, evicted tokens are merged into a summary token.
  • Efficiency: On a single A800 GPU, TTFT stays below 30 ms as the number of frames grows, with no OOM risk and no external CPU or disk storage.
  • Evaluation and accuracy:
    • Evaluated on StreamingBench, OVO-Bench and VideoMME with LLaVA-OV-7B and Qwen2.5-VL-7B.
    • Gains reach up to 11.4% on streaming benchmarks.
    • VideoMME average for LLaVA-OV-7B rises from 57.67 to 58.44.
  • Ablations and limits:
    • Lazy re-indexing works best for streaming and eager re-indexing for offline use.
    • Results barely change when the guidance prompt is reworded.
    • 72B models were not tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 297 works

Cite

Cite as Haowei Zhang, Shudong Yang, Jinlan Fu, See-Kiong Ng and Xipeng Qiu. 2026. HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.381

@inproceedings{zhang2026hermes,
  title = {{HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding}},
  author = {Haowei Zhang, Shudong Yang, Jinlan Fu, See-Kiong Ng and Xipeng Qiu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.381},
  url = {https://doi.org/10.18653/v1/2026.acl-long.381}
}

References

55 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. Claude 3.5 sonnet. find
  2. R.C. Atkinson and R.M. Shiffrin. 1968. Human memory: A proposed system and its control processes. find
  3. Alan D. Baddeley and Graham Hitch. 1974. Working memory. find
  4. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. Qwen3-vl technical report. Preprint, a…
  5. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. Qwen2.5-vl technical report. Preprint, arXiv:2502.1…
  6. Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. 2024. Titans: Learning to memorize at test time. Preprint, arXiv:2501.00663. find
  7. Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. 2024a. Videollm-online: Online video large language model for streaming video. Preprint, arXiv:2406.11816. find
  8. Xueyi Chen, Keda Tao, Kele Shao, and Huan Wang. 2025a. Streamingtom: Streaming token compression for efficient video understanding. Preprint, arXiv:2510.18269. — [arXiv:2510.18269](https://arxiv.org/abs/2510.18269) find
  9. Yilong Chen, Xiang Bai, Zhibin Wang, Chengyu Bai, Yuhan Dai, Ming Lu, and Shanghang Zhang. 2025b. Streamkv: Streaming video question-answering with segment-based kv cache retrieval and compression. Preprint, arXiv:2511.07278. — [arXiv:2511.07278](https://arxiv.org/abs/2511.07278) find
  10. Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, and 16 others. 2024b. How far are we to gpt-4v? closing the gap to commercial m…
  11. Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. Preprint, arXiv:2406.07476.
  12. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 other…
  13. Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Tao Zhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, and Hao Jiang. 2025. Streaming video question-answering with in-context video kv-cache retrieval. Preprint, arXiv:2503.00540. find
  14. Hermann Ebbinghaus. 2013. Memory: A contribution to experimental psychology. Annals of neurosciences, 20(4):155. find
  15. Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. 2024. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. Preprint, arXiv:2408.14023. — [arXiv:2408.14023](https://arxiv.org/abs/2408.14023) find
  16. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, and 2 others. 2025. Video-mme: The first-ever comprehensive evaluati…
  17. Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong X… find
  18. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, and 6 others. 2024. Minicpm: Unveiling the potential of small lan…
  19. Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. 2024b. Kangaroo: A powerful video-language model supporting long-context video input. Preprint, arXiv:2408.15542. — [arXiv:2408.15542](https://arxiv.org/abs/2408.15542) find
  20. Yuyang Hu, Shichun Liu, Yanwei Yue, Guibin Zhang, Boyang Liu, Fangyi Zhu, Jiahang Lin, Honglin Guo, Shihan Dou, Zhiheng Xi, Senjie Jin, Jiejun Tan, Yanbin Yin, Jiongnan Liu, Zeyu Zhang, Zhongxiang Sun, Yutao Zhu, Hao Sun, Boci Peng, and 28 others. 2025. Memory in the age of ai agents. Preprint, arXi… — [arXiv:2512.13564](https://arxiv.org/abs/2512.13564) find
  21. Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2023. Egoschema: A diagnostic benchmark for very long-form video language understanding. Preprint, arXiv:2308.09126.
  22. Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. 2020. Movienet: A holistic dataset for movie understanding. Preprint, arXiv:2007.10937. — [arXiv:2007.10937](https://arxiv.org/abs/2007.10937) find
  23. Zhenyu Ning, Guangda Liu, Qihao Jin, Wenchao Ding, Minyi Guo, and Jieru Zhao. 2025. Livevlm: Efficient online video understanding via streaming-oriented kv cache and retrieval. Preprint, arXiv:2505.15269. — [arXiv:2505.15269](https://arxiv.org/abs/2505.15269) find
  24. Minsoo Kim, Kyuhong Shim, Jungwook Choi, and Simyung Chang. 2024. Infinipot: Infinite context processing on memory-constrained llms. Preprint, arXiv:2410.01518. find
  25. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card.…
  26. Minsoo Kim, Kyuhong Shim, Jungwook Choi, and Simyung Chang. 2025. Infinipot-v: Memory-constrained kv cache compression for streaming video understanding. Preprint, arXiv:2506.15745. find
  27. Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Memgpt: Towards llms as operating systems. Preprint, arXiv:2310.08560. find
  28. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. Llava-onevision: Easy visual task transfer. Preprint, arXiv:2408.03326.
  29. Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Step… find
  30. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. 2024b. Mvbench: A comprehensive multimodal video understanding benchmark. Preprint, arXiv:2311.17005.
  31. Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, and Jiaqi Wang. 2025. Ovo-bench: How far is your video-llms from real-world online video understanding? Preprint, arXiv:2501.0… — [arXiv:2501.05510](https://arxiv.org/abs/2501.05510) find
  32. Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. 2024a. Vila: On pre-training for visual language models. Preprint, arXiv:2312.07533.
  33. Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025. Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reaction. Preprint, arXiv:2501.03218. — [arXiv:2501.03218](https://arxiv.org/abs/2501.03218) find
  34. Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. 2016. Ntu rgb+d: A large scale dataset for 3d human activity analysis. Preprint, arXiv:1604.02808. find
  35. Lianlei Shan, Shixian Luo, Zezhou Zhu, Yu Yuan, and Yong Wu. 2025. Cognitive memory in large language models. Preprint, arXiv:2504.02441. — [arXiv:2504.02441](https://arxiv.org/abs/2504.02441) find
  36. Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. 2024b. Streamingbench: Assessing the gap for mllms to achieve streaming video understanding. Preprint, arXiv:2411.03628. — [arXiv:2411.03628](https://arxiv.org/abs/2411.03628) find
  37. Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra. 2024. Longvu: Spatiotemporal adaptive c… — [arXiv:2410.17434](https://arxiv.org/abs/2410.17434) find
  38. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024a. Llava-next: Improved reasoning, ocr, and world knowledge. find
  39. Haoran Sun and Shaoning Zeng. 2025. Hierarchical memory for high-efficiency long-term reasoning in llm agents. Preprint, arXiv:2507.22925. — [arXiv:2507.22925](https://arxiv.org/abs/2507.22925) find
  40. Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. 2025. Dycoke: Dynamic compression of tokens for fast video large language models. Preprint, arXiv:2411.15024. — [arXiv:2411.15024](https://arxiv.org/abs/2411.15024) find
  41. Haibo Wang, Bo Feng, Zhengfeng Lai, Mingze Xu, Shiyu Li, Weifeng Ge, Afshin Dehghan, Meng Cao, and Ping Huang. 2025a. Streambridge: Turning your offline video large language model into a proactive streaming assistant. Preprint, arXiv:2505.05467. — [arXiv:2505.05467](https://arxiv.org/abs/2505.05467) find
  42. Han Wang, Yuxiang Nie, Yongjie Ye, Deng GuanYu, Yanjie Wang, Shuai Li, Haiyang Yu, Jinghui Lu, and Can Huang. 2024a. Dynamic-vlm: Simple dynamic visual token compression for videollm. Preprint, arXiv:2412.09530. — [arXiv:2412.09530](https://arxiv.org/abs/2412.09530) find
  43. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the…
  44. Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. 2025b. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. Preprint, arXiv:2405.19209. find
  45. Haomiao Xiong, Zongxin Yang, Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Jiawen Zhu, and Huchuan Lu. 2025. Streaming video understanding and multi-round interaction with memory-enhanced knowledge. Preprint, arXiv:2501.13468. — [arXiv:2501.13468](https://arxiv.org/abs/2501.13468) find
  46. Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, and Song Han. 2025. Stream-ingvlm: Real-time understanding for infinite video streams. Preprint, arXiv:2510.09608. find
  47. Haolin Yang, Feilong Tang, Lingxiao Zhao, Xiang An, Ming Hu, Huifa Li, Xinlin Zhuang, Yifan Lu, Xiaofeng Zhang, Abdalla Swikir, Junjun He, Zongyuan Ge, and Imran Razzak. 2025a. Streamagent: Towards anticipatory agents for streaming video understanding. Preprint, arXiv:2508.01875. — [arXiv:2508.01875](https://arxiv.org/abs/2508.01875) find
  48. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2024. Visionzip: Longer is better but not necessary in vision language models. Preprint, arXiv:2412.04467. — [arXiv:2412.04467](https://arxiv.org/abs/2412.04467) find
  49. Yanlai Yang, Zhuokai Zhao, Satya Narayan Shukla, Aashu Singh, Shlok Kumar Mishra, Lizhu Zhang, and Mengye Ren. 2025b. Streammem: Query-agnostic kv cache memory for streaming video understanding. Preprint, arXiv:2508.15717. — [arXiv:2508.15717](https://arxiv.org/abs/2508.15717) find
  50. Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, Lingpeng Kong, Qi Liu, Yuanxing Zhang, and Xu Sun. 2025. Timechat-online: 80% visual tokens are naturally redundant in streaming videos. Preprint, arXiv:2504.17343. — [arXiv:2504.17343](https://arxiv.org/abs/2504.17343) find
  51. Xiangyu Zeng, Kefan Qiu, Qingyu Zhang, Xinhao Li, Jing Wang, Jiaxin Li, Ziang Yan, Kun Tian, Meng Tian, Xinhai Zhao, Yi Wang, and Limin Wang. 2025. Streamforest: Efficient online video understanding with persistent event memory. Preprint, arXiv:2509.24871. — [arXiv:2509.24871](https://arxiv.org/abs/2509.24871) find
  52. Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. 2024a. Flash-vstream: Memory-based real-time understanding for long video streams. Preprint, arXiv:2406.08085. — [arXiv:2406.08085](https://arxiv.org/abs/2406.08085) find
  53. Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. 2024b. Flash-vstream: Memory-based real-time understanding for long video streams. Preprint, arXiv:2406.08085. — [arXiv:2406.08085](https://arxiv.org/abs/2406.08085) find
  54. Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024c. Long context transfer from language to vision. Preprint, arXiv:2406.16852.
  55. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2025. Llava-video: Video instruction tuning with synthetic data. Preprint, arXiv:2410.02713.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-hermes-kv-cache-as-hierarchical/