大语言模型推理中的存储优化技术综述

Summary

This survey, published in the Journal of Computer Research and Development (2025), examines why serving large language models is increasingly limited by storage rather than compute. Model parameters keep growing, and users want long-context generation. Both the weights and the key-value (KV) cache compete for scarce GPU memory, and distributed deployments often waste cluster resources and recover poorly from failures. The authors first explain the prefill and decode stages of Transformer inference: prefill is compute-bound, while decoding is limited by memory bandwidth. They then sort the literature into three families:

• Memory optimization: KV cache management, KV cache compression, and attention-operator redesign.
• Heterogeneous storage: GPU memory, CPU RAM and SSD combined through tensor placement, asynchronous transfers that overlap compute, and smart allocation and prefetching.
• Distributed storage: batching, multi-level scheduling, and redundant replication across machines.

The survey ends with open problems and future directions. It matters as a structured map for anyone who needs to deploy LLMs on limited hardware.

Key points

• Worked memory example: LLaMA2-70B in FP16 needs about 140 GB for its weights alone. With batch size 8, sequence length 4,096, hidden size 8,192 and 80 layers, the KV cache adds roughly 80 GB. A single NVIDIA A100 offers at most 80 GB.
• KV cache size formula: the cache scales as 2 × batch × sequence length × hidden size × layers × precision.
• ORCA's fragmentation problem: ORCA reserves space for the maximum token count. In the paper's example, 2,048 token slots are reserved but only 512 are used, leaving 1,536 slots as internal fragmentation.
• vLLM and Infinite-LLM: vLLM's PagedAttention stores the KV cache in non-contiguous, fixed-size blocks, much like OS paging, so waste per request is limited to one block. Infinite-LLM extends this idea with DistAttention: it uses rBlocks, a per-instance rManager, and a global gManager that lends memory between instances.
• Other systems cited: Splitwise, DéjàVu (fault-tolerant KV-cache streaming) and ServerlessLLM. The keywords also flag fault recovery and serverless inference as areas the survey covers.