Elamur Research⌕
☰
home / papers / 2026 / MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning

MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning

Paper · 2026 · Tao Zhang, Ziqian Zeng, Hao Peng et al.

Summary

Reasoning-oriented LLMs produce very long Chain-of-Thought outputs, so their KV cache consumes large amounts of GPU memory and bandwidth. Existing KV cache quantization methods perform poorly at about 2 bits on hard reasoning tasks. Fixed-precision schemes cannot handle outlier key channels. Mixed-precision schemes allocate bits by raw quantization error, which ignores how much a channel actually affects attention.

The authors argue that a key channel deserves higher precision only if it is both hard to quantize and relevant to the current query. A large-magnitude key channel matters little when the matching query activations are small. MixKVQ applies this idea through a lightweight, tuning-free heuristic that combines both signals to choose which key channels stay in higher precision. Keys are quantized per channel with mixed precision, and values are quantized per token.

On AIME 2024-2025, MATH 500, GPQA-Diamond, LiveCodeBench and LongBench, using Llama-3.1, Qwen2.5 and Mistral models, MixKVQ nearly matches BF16 and beats KIVI, KVTuner, KVQuant, SKVQ and RotateKV. This makes memory-efficient long-context reasoning more practical.

Key points

  • On Qwen-32B, MixKVQ averages 66.04% across the reasoning benchmarks, against 67.84% for BF16. The 4-bit RotateKV baseline reaches 64.51%, and 2-bit KIVI reaches 58.89%.
  • On LongBench, the method runs at an effective 2.70 bits on Mistral-7B-Instruct-v0.3 and Llama-3.1-8B-Instruct with almost no loss compared with BF16.
  • Efficiency was tested on Llama2-13B-chat with ShareGPT workloads in a vLLM-style setup. At similar peak memory, MixKVQ allows up to 2.25× larger batches and 2.63–2.81× higher throughput than FP16.
  • The motivating example: Qwen2.5-32B with batch size 64 and 32,768 tokens needs about 512 GB for its KV cache, which is 8.59× the memory of its weights.
  • Stated limitations: managing mixed-precision memory adds overhead, Multi-Head Latent Attention (MLA) is not covered, and prefill costs are not fully analyzed. Code is available at github.com/ZeroNLP/MixKVQ.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Beihang UniversityBeijing · China65 works · 8,452 citations🎓South China University of TechnologyGuangzhou · China22 works · 1,554 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksEfficient models & compression · 242 works

Cite

Cite as Tao Zhang, Ziqian Zeng, Hao Peng, Huiping Zhuang and Cen Chen. 2026. MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.326

@inproceedings{zhang2026mixkvq,
  title = {{MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning}},
  author = {Tao Zhang, Ziqian Zeng, Hao Peng, Huiping Zhuang and Cen Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.326},
  url = {https://doi.org/10.18653/v1/2026.acl-long.326}
}

References

48 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 7. find
  2. Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213–100240.
  3. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Associat… find
  4. Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. 2024. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. Preprint, arXiv:2406.02069.
  5. Chi-Chih Chang, Chien-Yu Lin, Yash Akhauri, Wei-Cheng Lin, Kai-Chiang Wu, Luis Ceze, and Mohamed S. Abdelfattah. 2025. xkv: Cross-layer svd for kv-cache compression. Preprint, arXiv:2503.18893. — [arXiv:2503.18893](https://arxiv.org/abs/2503.18893) find
  6. Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. 2024. Palu: Compressing kv-cache with low-rank projection. Preprint, arXiv:2407.21118. find
  7. Yidong Chen, Chen Zhang, Rongchao Dong, Haoyuan Zhang, Yonghua Zhang, Zhonghua Lu, and Jidong Zhai. 2024. Mixq: Taming dynamic outliers in mixed-precision quantization by online prediction. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15… find
  8. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 179 others. 2025. Deepseek-r1: Incentivizing reasoning capability i…
  9. Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Tao Zhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, and Hao Jiang. 2025. Streaming video question-answering with in-context video kv-cache retrieval. arXiv preprint arXiv:2503.00540. find
  10. Yanhao Dong, Yubo Miao, Weinan Li, Xiao Zheng, Chao Wang, and Feng Lyu. 2025. Accelerating llm inference throughput via asynchronous kv cache prefetching. arXiv preprint arXiv:2504.06319. find
  11. Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng ZHANG, and Dahua Lin. 2024. SKVQ: Sliding-window key and value cache quantization for large language models. In First Conference on Language Modeling. find
  12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others.… — The Llama 3 Herd of Models
  13. Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. 2025. Identify critical kv cache in llm inference from an output perturbation perspective. Preprint, arXiv:2502.03805. — [arXiv:2502.03805](https://arxiv.org/abs/2502.03805) find
  14. Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. Lm-infinite: Zero-shot extreme length generalization for large language models. Preprint, arXiv:2308.16137.
  15. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. In Advances in Neural Information Processing Systems, volume 37, pages 1270– 1303. Curran…
  16. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Livecodebench: Holistic and contamination free evaluation of large language models for code. The Thirteenth International Conference on Learning Representations.
  17. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611…
  18. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  19. Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2025. QServe:w4a8KV4 quantization and system co-design for efficient LLM serving. In Eighth Conference on Machine Learning and Systems. find
  20. Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, and Lu Hou. 2025a. Quantization hurts reasoning? an empirical study on quantized reasoning models. arXiv preprint arXiv:2504.04823. — [arXiv:2504.04823](https://arxiv.org/abs/2504.04823) find
  21. Tengxuan Liu, Shiyao Li, Jiayi Yang, Tianchen Zhao, Feng Zhou, Xiaohui Song, Guohao Dai, Shengen Yan, Huazhong Yang, and Yu Wang. 2025b. Pm-kvq: Progressive mixed-precision kv cache quantization for long-cot llms. ArXiv, abs/2505.18610. — [arXiv:2505.18610](https://arxiv.org/abs/2505.18610) find
  22. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Rese…
  23. MAA. 2024. American invitational mathematics examination - aime. In American Invitational Mathematics Examination - AIME 2024. find
  24. MAA. 2025. American invitational mathematics examination - aime. In American Invitational Mathematics Examination - AIME 2025. find
  25. Janghwan Lee, Seongmin Park, Sukjin Hong, Minsoo Kim, Du-Seong Chang, and Jungwook Choi. 2024a. Improving conversational abilities of quantized large language models via direct preference alignment. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:… find
  26. OpenAI. 2024. Introducing openai o1. find
  27. Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024b. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 155–172, Santa Clara, CA. USENIX Association. find
  28. Tao Qian, Yu Wenyuan, and Zhou Jingren. 2025. Asymkv: Enabling 1-bit quantization of KV cache with layer-wise asymmetric quantization configurations. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2316– 2328, Abu Dhabi, UAE. Association for Computational Ling… find
  29. Xing Li, Zeyu XING, Yiming Li, Linping Qu, Hui-Ling Zhen, Yiwu Yao, Wulong Liu, Sinno Jialin Pan, and Mingxuan Yuan. 2025. Kvtuner: Sensitivity-aware layer-wise mixed-precision kv cache quantization Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2024. Moon… find
  30. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems, volume 5, pages 606–624. Curan. find
  31. Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. 2025. CAKE: Cascading and adaptive KV cache eviction with layer preferences. In The Thirteenth International Conference on Learning Representations. find
  32. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sotti… — Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
  33. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
  34. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations. find
  35. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  36. Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, and 25 others. 2024. Qwen2.5 technical report. ArXiv, a… find
  37. Zebin Ren, Krijn Doekemeijer, Tiziano De Matteis, Christian Pinto, Radu Stoica, and Animesh Trivedi. 2025. An i/o characterizing study of offloading llm models and kv caches to nvme ssd. In Proceedings of the 5th Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems, p… find
  38. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, Carlsbad, CA. USENIX Association. find
  39. Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. 2024. Eigen attention: Attention in low-rank space for kv cache compression. Preprint, arXiv:2408.05646. find
  40. Yi Su, Yuechi Zhou, Quantong Qiu, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, and Min Zhang. 2025a. Accurate KV cache quantization with outlier tokens tracing. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12… find
  41. Zunhai Su, Zhe Chen, Wang Shen, Hanyu Wei, Linge Li, Huangqi Yu, and Kehong Yuan. 2025b. Rotatekv: Accurate and robust 2-bit kv cache quantization for llms via outlier-aware adaptive rotations. Preprint, arXiv:2501.16383. — [arXiv:2501.16383](https://arxiv.org/abs/2501.16383) find
  42. Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong, Yiwu Yao, and Gongyi Wang. 2024. Razorattention: Efficient kv cache compression through retrieval heads. arXiv preprint arXiv:2407.15891. — [arXiv:2407.15891](https://arxiv.org/abs/2407.15891) find
  43. Yuxuan Tian, Zihan Wang, Yebo Peng, Aomufei Yuan, Zhiming Wang, Bairen Yi, Xin Liu, Yong Cui, and Tong Yang. 2025. Keepkv: Eliminating output perturbation in kv cache compression for efficient llms inference. Preprint, arXiv:2504.09936. — [arXiv:2504.09936](https://arxiv.org/abs/2504.09936) find
  44. Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. 2024. Llm inference unveiled: Survey and roofline model insights. ArXiv, abs/2402.16363. find
  45. Ahmet Caner Yüzügüler, Jiawei Zhuang, and Lukas Cavigelli. 2025. Preserve: Prefetching model weights and kv-cache in distributed llm serving. arXiv preprint arXiv:2501.08192. find
  46. Rongzhi Zhang, Kuang Wang, Liyuan Liu, Shuohang Wang, Hao Cheng, Chao Zhang, and Yelong Shen. 2024. Lorc: Low-rank compression for llms kv cache with a progressive compression strategy. Preprint, arXiv:2410.03111. find
  47. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Informati… find
  48. Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, Longyue Wang, and Mi Zhang. 2025. D2o: Dynamic discriminative operations for efficient long-context inference of large language models. Preprint, arXiv:2406.13035. — [arXiv:2406.13035](https://arxiv.org/abs/2406.13035) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-mixkvq-query-aware-mixed-precision/