📄 DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference

Summary

LLM serving systems must deliver high throughput and low latency together, which is hard when prompts are long. In systems such as vLLM and TGI, processing a new prompt pauses ongoing generation, so streamed tokens arrive late. Orca instead runs whole prompts next to generation, which can still make some forward passes very large. DeepSpeed-FastGen combines DeepSpeed-MII (frontend and serving) with DeepSpeed-Inference (scheduling and kernels). It adds a batching strategy called Dynamic SplitFuse. The scheduler keeps every forward pass at a fixed token budget. It cuts long prompts into chunks spread over several iterations and packs short prompts and generation tokens together to fill the budget exactly. The design rests on three profiling observations: token count drives latency, throughput saturates once a pass has enough tokens, and the throughput curve is concave. Measured against vLLM, the system achieves effective throughput as much as 2.3x higher, average latency about 2x lower, and per-token tail latency reduced by as much as 3.7x. The alpha release is pip-installable and supports LLaMA, Mistral and OPT families.

Key points

• Reported gains over vLLM: up to 2.3x effective throughput, 2x lower mean latency, and up to 3.7x lower P95 token-generation latency.
• Builds on two existing techniques: a blocked (paged) KV cache, as in PagedAttention, and continuous batching, as in Orca.
• Benchmarks use 1–32 concurrent clients sending 512 requests in total. One setup is Llama 2 7b on an A6000, with prompt and output lengths drawn from normal distributions (means 2600 and 60, 30% variance).
• The alpha release supports HuggingFace models from the LLaMA/LLaMA-2, Mistral and Facebook OPT families. It offers two deployment modes:
  • a non-persistent pipeline for interactive scripts;
  • a persistent GRPC server with a built-in load balancer.
• Prebuilt kernels ship as DeepSpeed-Kernels, targeting NVIDIA Ampere+ GPUs, CUDA 11.6+ and Ubuntu 20+. FlashAttention and FasterTransformer kernels are used with modifications.

Interesting ideas

• I1. Schedule by a single signal. Profiling showed that forward-pass latency depends mainly on the total token count, not on how many sequences are in the batch. A scheduler can therefore control just one variable, which is a useful simplification for any batched GPU workload.
• I2. Use concavity to justify equal splits. Because throughput grows concavely with tokens per pass, dividing a pool of tokens evenly across passes is optimal. This short mathematical argument could guide chunk sizing in other pipelines, such as batch embedding or tagging of large corpora.
• I3. Chunk long inputs and top up short ones. Splitting long prompts and fusing short ones into fixed-size passes keeps latency per step steady and avoids stalls from preemption. The same idea fits mixed-length document processing, where a few huge documents should not block streaming results for everyone else.
• I4. Measure user-facing metrics, not only averages. Beyond throughput-latency curves, the paper evaluates an "effective throughput" tied to the chat experience: time to first token and the steady pace of streamed tokens. This is a good template for judging any interactive NLP service.
• I5. Offer two deployment modes with a tiny API. A throwaway in-script pipeline and a persistent load-balanced server share the same model identifiers. This lowers the barrier from experimentation to production for small teams.