Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 4.0 (arXiv)" does not allow redistributing the text itself. Original — arxiv.org/pdf/2406.18629

Summary

Large language models often fail at multi-step math because one slip anywhere in a long derivation ruins the final answer. Standard Direct Preference Optimization (DPO) helps little here. It compares whole answers, so a rejected solution that is correct except for one faulty step gives a noisy training signal. Models trained this way barely improve at telling good outputs from bad ones. Step-DPO changes the unit of preference from the full answer to a single reasoning step. Given a problem and a prefix of correct steps, the model learns to prefer a correct next step over the first wrong one. The authors also built a low-cost three-stage pipeline (error collection, step localization, rectification). It produces about 10K step-level pairs, and the preferred step is sampled from the model itself. After fewer than 500 training steps, Qwen2-72B-Instruct scores 70.8% on MATH and 94.0% on GSM8K, ahead of several proprietary systems. The work shows that fine-grained process supervision is possible without a separate reward model or a full RLHF pipeline.

Key points

• Method: each training example is a math problem, the correct earlier steps, one chosen next step and one rejected next step. The DPO objective is applied only to that next step, not to the whole solution.
• Dataset: about 10K step-wise preference pairs. Using the model's own corrected steps (in-distribution data) worked better than steps written by humans or GPT-4.
• Efficiency: with fewer than 500 Step-DPO training steps, models with more than 70B parameters gain close to 3 points of accuracy on MATH.
• Headline result: Qwen2-72B-Instruct + Step-DPO reaches 70.8% on MATH and 94.0% on GSM8K, beating GPT-4-1106, Claude-3-Opus and Gemini-1.5-Pro. Gains are also reported for Qwen2-7B, Qwen2-57B-A14B, Qwen1.5-32B and Llama3-70B.
• Resources: code, data and models are released at github.com/dvlab-research/Step-DPO. The authors are from CUHK, HIT (Shenzhen) and SmartMore.