📄 SpotServe: Serving Generative Large Language Models on Preemptible Instances

Summary

SpotServe addresses the high cost of serving generative LLMs. It runs inference on preemptible cloud GPUs, such as AWS spot instances, which can be up to 90% cheaper than on-demand machines but can be reclaimed at any time. Earlier spot-based serving systems assumed that a model fits on one instance and handled preemptions by rerouting requests or adding redundancy. LLMs instead need data, tensor and pipeline parallelism across many machines, so one preemption disrupts a whole pipeline. SpotServe has three main components:

• Adaptive parallelization: it re-chooses the parallelization configuration whenever instance availability or request load changes.
• Cheap migration: it frames the assignment of surviving instances to the new layout as bipartite matching, solved with the Kuhn-Munkres algorithm, so existing parameters and key/value cache are reused.
• Grace-period use: it saves decoding progress token by token, so interrupted requests resume instead of restarting.

On real preemption traces and several LLMs, SpotServe cut P99 tail latency by 2.4–9.1× compared with existing serving systems. It also lowered monetary cost by 54% compared with on-demand-only deployment. This makes large-model serving more affordable for budget-limited teams.

Key points

• The authors describe SpotServe as the earliest system to serve generative LLMs in a distributed fashion using preemptible instances. It appeared at ASPLOS'24, and the code is available at github.com/Hsword/SpotServe.
• The paper illustrates the cost problem with two examples:
  • Serving one GPT-3 (175B) pipeline on AWS needs more than 16 A100-40GB GPUs and costs over $66 per hour.
  • Reloading a 120B-parameter GPT model from persistent storage on AWS takes more than 2 minutes, so restarting from scratch is expensive.
• AWS gives about 30 seconds of grace period before a spot instance is taken away. SpotServe uses this window through stateful inference recovery with a just-in-time decision on when to move the KV cache.
• Reported results:
  • P99 tail latency is 2.4–9.1× lower than the best existing LLM serving systems.
  • Cost is up to 54% lower than on-demand instances, with similar average latency.
  • The authors found AlpaServe about 3× slower than FasterTransformer (C++) for serving a single LLM.
• The authors list several limitations:
  • The system depends on the grace period.
  • It targets single-type GPU instances.
  • It optimizes only for latency, leaving strict-SLO or throughput goals, mixture-of-experts models, and multi-cloud or heterogeneous setups for future work.

Interesting ideas

• I1. Token-level checkpointing for autoregressive jobs. Saving progress after each generated token means a short shutdown window still preserves most of the work. The same idea suits any long-running generation job on unreliable hardware, such as large-scale corpus annotation or translation with an LLM.
• I2. Migration as an assignment problem. Treating "which old machine takes which new role" as weighted bipartite matching, solved with Kuhn-Munkres, minimizes how much state must be moved. This is a simple, optimal recipe for any resharding task where nodes already hold partial data.
• I3. Overlapping migration with pipeline order. Because pipeline stages run in sequence, state can be moved stage by stage while other stages keep computing. This hides downtime and could be borrowed for rolling updates of multi-stage processing pipelines.
• I4. Re-planning the configuration as conditions change. Instead of fixing one parallel layout, the system re-chooses it whenever available hardware or incoming load shifts, balancing latency, throughput and cost. This is a useful pattern for budget-constrained NLP serving that relies on cheap but volatile compute.
• I5. Applicability beyond cloud spot instances. The authors argue the same techniques apply whenever a scheduler can pull resources away, for example for urgent jobs on a shared cluster. Shared university or lab GPU clusters could benefit in the same way.