Group Sequence Policy Optimization

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2507.18071

Summary

Reinforcement learning is now central to building reasoning-capable LLMs. Yet GRPO, a widely used algorithm, often becomes unstable on large models and long outputs, and training can collapse irreversibly. The Qwen Team traces this to how GRPO uses importance weights. It computes a separate ratio for every token from just one sample, so the ratio fails to correct for the shift between policies. Instead, it adds noise that grows with response length and is amplified by clipping. Their remedy, GSPO, works on whole responses instead. The importance ratio is the length-normalized likelihood ratio of the full sequence, clipping removes entire off-policy responses, and advantages are group-normalized rewards as in GRPO. A variant called GSPO-token gives the same values but allows per-token advantages, for example in multi-turn RL. In experiments with a cold-start model built on Qwen3-30B-A3B-Base, GSPO trained more efficiently and reached better results than GRPO. It also kept MoE training stable without extra workarounds. The method underlies the gains in the latest Qwen3 models and gives a more robust base for scaling RL.

Key points

• GSPO computes its ratio from the geometric mean of the per-token likelihood ratios across a response, so every token in a response gets the same weight. Because the ratio is defined differently, its clipping range is orders of magnitude smaller than GRPO's.
• GSPO clips about 0.15 of tokens on average, compared with about 0.0013 for GRPO. Even though it learns from far fewer tokens, it is more sample-efficient, which suggests GRPO's token-level gradients are noisy.
• In the 48-layer Qwen3-30B-A3B MoE model, about 10% of the activated experts change after each gradient update. GRPO needs Routing Replay to cope with this, which costs memory and communication. GSPO does not need it.
• Sequence-level likelihoods are less affected by precision differences between training engines (Megatron) and inference engines (vLLM, SGLang). This could let training use the inference engine's likelihoods directly, without recomputing them.
• Evaluation covered training reward and benchmarks such as AIME'24, LiveCodeBench and CodeForces.