Elamur Research⌕
☰
home / papers / 2026 / HqeKV: Towards Hybrid Quantization and Eviction for KV Cache in Long-Context LLM Inference

HqeKV: Towards Hybrid Quantization and Eviction for KV Cache in Long-Context LLM Inference

Paper · 2026 · He Wang, Yu Gu, Fangfang Li et al.

Summary

Storing key-value (KV) pairs during autoregressive decoding avoids recomputation, but the cache grows with sequence length and can outweigh the model's own memory in long-context serving. Existing compressors mostly choose one tool, either quantization or eviction. They offer few granularity options and judge token importance mainly from keys. HqeKV merges both tools. Cached pairs are ranked by importance and each receives one of five treatments: keep in FP16, quantize to 4, 2 or 1 bit, or drop. An offline search picks the share of each tier for a given memory budget. Importance is scored by a new joint K–V metric that adds the numeric range of value vectors to cumulative attention weights. The authors back this choice with a derivation and empirical evidence that quantization error rises with range. Uniform quantization serves the higher bit-widths and normalized quantization the lowest ones. On LongBench with several LLMs, HqeKV beats prior methods at equal memory. This matters for running long-context inference on limited GPU memory.

Key points

  • Five compression actions per cached element: FP16, 4-bit, 2-bit, 1-bit, or eviction. The cache is re-quantized periodically during decoding, after every T new tokens.
  • Range as an error signal: the correlation (PCC) between conversion loss and value-vector range is 0.96 for 4-bit V Cache tokens, versus 0.88 for random normal vectors. This motivates the joint metric over prior K-only or VATP-style scoring.
  • Headline result: on LongBench under the same memory limit, quality rises from 40.53 for the best competitor to 49.98. Baselines include eviction methods (H2O, SnapKV, PyramidKV, CAKE), quantization methods (KIVI, KVTuner) and hybrids (MiniKV, EVICPRESS).
  • Offline ratio search: for a 3.2-bit average budget it took about 7 min 46 s. The allocation was roughly 0.65 / 0.28 / 0.06 / 0.02 for 4-bit / 2-bit / 1-bit / eviction, and 200 versus 300 iterations made little difference.
  • Best metric hyperparameters: α = 2 and β = 4 scored 56.75 on HotpotQA with Llama-3.1-8B-Instruct. Code is published at github.com/skywclouds/HqeKV.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northeastern UniversityBoston · United States57 works · 6,533 citations🎓Northeastern UniversityShenyang · China42 works · 1,940 citations🎓Guangzhou UniversityGuangzhou · China4 works · 7 citations🎓Ocean University of ChinaQingdao · China2 works · 2 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as He Wang, Yu Gu, Fangfang Li, Zhigang Wang, Zhenghao Liu, Ning Wang, Xiaohua Li and Ge Yu. 2026. HqeKV: Towards Hybrid Quantization and Eviction for KV Cache in Long-Context LLM Inference. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.201

@inproceedings{wang2026hqekv,
  title = {{HqeKV: Towards Hybrid Quantization and Eviction for KV Cache in Long-Context LLM Inference}},
  author = {He Wang, Yu Gu, Fangfang Li, Zhigang Wang, Zhenghao Liu, Ning Wang, Xiaohua Li and Ge Yu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.201},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.201}
}

References

31 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. 2025. Matharena: Evaluating llms on uncontaminated math competitions. find
  2. James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. Advances in neural information processing systems, 24. find
  3. Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. 2024. Pyramidkv: Dynamic KV cache compression based on pyramidal information funneling. CoRR, abs/2406.02069.
  4. Zhihang Cai, Xingjun Zhang, Zhendong Tan, and Zheng Wei. 2025. Nqkv: A kv cache quantization scheme based on normal distribution characteristics. arXiv preprint arXiv:2505.16210. — [arXiv:2505.16210](https://arxiv.org/abs/2505.16210) find
  5. Wen Cheng, Shichen Dong, Jiayu Qin, and Wei Wang. 2025. Qaq: Quality adaptive quantization for llm kv cache. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2542– 2550. find
  6. Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. 2024. Skvq: Sliding-window key and value cache quantization for large language models. arXiv preprint arXiv:2405.06219. find
  7. Shaoting Feng, Yuhan Liu, Hanchen Li, Xiaokun Chen, Samuel Shen, Kuntai Du, Zhuohan Gu, Rui Zhang, Yuyang Huang, Yihua Cheng, and 1 others. 2025. Evicpress: Joint kv-cache compression and eviction for efficient llm serving. arXiv preprint arXiv:2512.14946. find
  8. Ravi Ghadia, Avinash Kumar, Gaurav Jain, Prashant J. Nair, and Poulami Das. 2025. Dialogue without limits: Constant-sized KV caches for extended response in LLMs. In Forty-second International Conference on Machine Learning. find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. 2024. Attention score is not all you need for token importance indicator in KV cache reduction: Value also matters. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21158–21166, Miami, Florida, USA. Ass… find
  11. Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. 2024. Zipcache: Accurate and efficient kv cache quantization with salient token identification. In Advances in Neural Information Processing Systems, volume 37, pages 68287–68307. Curran Associates, Inc. find
  12. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. In Advances in Neural Information Processing Systems, volume 37, pages 1270– 1303. Curran…
  13. Minsu Kim, Seongmin Hong, RyeoWook Ko, Soongyu Choi, Hunjong Lee, Junsoo Kim, Joo-Young Kim, and Jongse Park. 2025. Oaken: Fast and efficient llm serving with online-offline hybrid kv cache quantization. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, ISCA ’25, pa… find
  14. Xing Li, Zeyu XING, Yiming Li, Linping Qu, Hui-Ling Zhen, Yiwu Yao, Wulong Liu, Sinno Jialin Pan, and Mingxuan Yuan. 2025. KVTuner: Sensitivity-aware layer-wise mixed-precision KV cache quantization for efficient and nearly lossless LLM inference. In Forty-second International Conference on Machine… find
  15. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. SnapKV: LLM knows what you are looking for before generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  16. Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2025. Spinquant: LLM quantization with learned rotations. In The Thirteenth International Conference on Learning Representations. find
  17. Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems, 36:5… find
  18. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen (Henry) Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. Kivi: a tuning-free asymmetric 2bit quantization for kv cache. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
  19. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. Preprint, arXiv:1609.07843.
  20. Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. 2025. CAKE: Cascading and adaptive KV cache eviction with layer preferences. In The Thirteenth International Conference on Learning Representations. find
  21. Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, and Minjia Zhang. 2025. MiniKV: Pushing the limits of 2-bit KV cache via compression and system co-design for efficient long context inference. In Findings of the Association for Computational Linguistics: ACL 2025, pages 18506–18523, Vienna, Au… find
  22. Yi Su, Yuechi Zhou, Quantong Qiu, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, and Min Zhang. 2025. Accurate KV cache quantization with outlier tokens tracing. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 128… find
  23. Wei Tao, Bin Zhang, Xiaoyang Qu, Jiguang Wan, and Jianzong Wang. 2025. Cocktail: Chunk-adaptive mixed-precision quantization for long-context llm inference. In 2025 Design, Automation & Test in Europe Conference (DATE), pages 1–7. IEEE. find
  24. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. find
  25. Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. 2025. Think: Thinner key cache by query-driven pruning. In The Thirteenth International Conference on Learning Representations. find
  26. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  27. June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. 2024. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. CoRR, abs/2402.18096. — [arXiv:2402.18096](https://arxiv.org/abs/2402.18096) find
  28. Amir Zandieh, Majid Daliri, and Insu Han. 2025. Qjl: 1-bit quantized jl transform for kv cache quantization with zero overhead. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25805–25813. find
  29. Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. 2025. Pqcache: Product quantization-based kvcache for long context llm inference. Proceedings of the ACM on Management of Data, 3(3):1–30. find
  30. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, and 1 others. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:346… find
  31. Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems, 6:196–209. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-hqekv-hybrid-quantization-eviction-kv/