大语言模型推理中的键值缓存压缩方法综述

Summary

This survey from Tsinghua University examines KV cache compression for large language model inference. During decoding, the keys and values of previous tokens are stored so they do not have to be recomputed. That cache grows linearly with context length, model depth and the number of concurrent sessions. It causes two problems. GPU memory runs out, which limits how many requests can be served in parallel. Long-term conversational memory also forces providers to keep huge caches on SSDs, which costs money and slows loading.

The authors set out three design challenges: preserving answer quality, keeping compression overhead low, and generalizing across tasks, models and context lengths. They group existing techniques into five families: merging, quantization, token eviction, sharing, and attention-head reduction. For each family they describe how representative systems work, and they compare them by compression dimension, extra computation, architectural changes and how the compression ratio is tuned. Their outlook calls for:

• better generalization across tasks and scenarios,
• cheaper compression,
• automatic method selection with adaptive parameters.

The work is a practical map for anyone building efficient LLM serving.

Key points

• The survey appeared in Journal of Computer Research and Development (计算机研究与发展), vol. 63(4), pp. 1021–1038, 2026.
• The taxonomy's representative methods are:
  • Merging: KVMerger, MiniCache
  • Quantization: KVQuant, CacheGen, Ecco
  • Token eviction: Scissorhands, Impress, ClusterKV
  • Sharing: KVSharer, LCKV
  • Head reduction: MQA (2019), GQA (2023)
• Scale example: a prompt of roughly 2,000 tokens can produce up to about 5 GB of KV cache. Even 40/80 GB NVIDIA A100 GPUs can therefore hold only a few concurrent requests, and Claude supports contexts of up to 200K tokens.
• KVMerger was evaluated on LLaMa2-7b-chat with SynthWiki (average length about 4,000 tokens). Key vectors of neighbouring tokens showed strong locality, with some cosine similarities above 90%. Tokens are merged with Gaussian-kernel weighting around a pivot token chosen by attention score.
• Cited evaluations show weak generalization. Compressed caches perform poorly on arithmetic reasoning such as GSM8K and in long-context QA. Answer quality on instruction-tuned models can fall by as much as 25.53%.