Scaling Reasoning Efficiently via Relaxed On-Policy Distillation

Paper · 2026 · Jongwoo Ko, Sara Abdali, Young Jin Kim et al.

Summary

Small language models often fail to gain reasoning skills from direct RL, and the usual alternative, on-policy distillation from a stronger teacher, tends to be unstable. It can collapse entropy early and even make the student worse than its starting checkpoint. The authors treat on-policy distillation as policy optimization in which the log-likelihood ratio between teacher and student serves as a per-token reward. Applying a stop-gradient to this reward leaves the expected gradient unchanged but lowers its variance. The analysis traces instability to two causes: heavy-tailed negative rewards and many uninformative near-zero signals. Their method, REOPOLD (Relaxed On-Policy Distillation), eases strict imitation through three mechanisms: a mixture-based scheme for clipping rewards, dynamic selection of tokens according to their entropy, and a staged schedule that shifts from exploration to refinement. They test it on math, visual, and agentic tool-use reasoning, where it beats vanilla RKL distillation and recent RL pipelines. It needs far fewer training samples and scales better at test time. This makes it practical to distill from a much larger teacher into a compact model.

Key points

Compared with recent RL approaches, REOPOLD reports about 6.7–12× better sample efficiency on AIME-25, using DeepSeek-R1-Distill-Qwen-1.5B as the student.
In visual reasoning, a Qwen2.5-VL-7B-Instruct student reaches the level of a 32B teacher with roughly 3.32× faster inference. After 300 steps its average accuracy is 58.83% versus 56.71% for RKL, across Geo3K, MathVerse, MathVision, MathVista, WeMath and Hallusion.
In the ablation on Qwen2.5-VL-3B-Instruct, each component adds accuracy in turn: stop-gradient, reward clipping, token-level dynamic sampling, then multi-stage training. The average rises from 35.37% (RKL) to 41.12%. A stricter entropy threshold (β = 0.2) outperforms β = 0.5.
For agentic visual tool-use, the setup builds on VerlTool, with Pixel-Reasoner-7B as teacher, an SFTed Qwen2.5-VL-3B-Instruct as student, and 15K training queries. REOPOLD averages 65.75%, ahead of GRPO (64.25%) and RKL (63.27%), on Pixel, V-Star, InfoVQA and TallyQA.
The loss uses only sampled tokens rather than full-vocabulary distributions, which avoids heavy memory costs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields, topics and concepts

📚 Shelves 4Computer Science · 19,503 works Artificial Intelligence · 18,805 works Arts and Humanities · 17,959 works Language and Linguistics · 17,789 works

🧭 Topics 1Reasoning · 1,948 works

Cite

Cite as Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen and Pashmina Cameron. 2026. Scaling Reasoning Efficiently via Relaxed On-Policy Distillation. arXiv:2603.11137.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2603.11137 · arXiv 2603.11137

@misc{ko2026scaling,
  title = {{Scaling Reasoning Efficiently via Relaxed On-Policy Distillation}},
  author = {Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen and Pashmina Cameron},
  year = {2026},
  doi = {10.48550/arxiv.2603.11137},
  eprint = {2603.11137},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2603.11137}
}

References

7 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

supplemented by additional public datasets. Detailed training setup is described in Appendix C. Acharya, M., Kafle, K., and Kanan, C. Tallyqa: Answering complex counting questions. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 8076– 8084, 2019. find
Evaluation. Following Su et al. (2025), we evaluate our model and baselines on four representative multi-modal benchmarks using nucleus sampling with temperature of 1.0 and a top-p of 1.0: test-split of Pixel-Reasoner, V-Star (Wu & Xie, 2024), InfographicVQA (Mathew et al., 2022), and TallyQA (Achar… find
Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Garea, S. R., Geist, M., and Bachem, O. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=3zKtaqxLhW. ↗
Results. As reported in Table 7, R EOPOLD outperforms both vanilla RKL and GRPO, notably surpassing the latter even when it utilizes the complex reward designs proposed by Su et al. (2025). Although GRPO achieves slightly higher accuracy on the Pixel test split, R EOPOLD demonstrates superior perfor… find
Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024. find
Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
Barbiero, P., Ciravegna, G., Giannini, F., Zarlenga, M. E., Magister, L. C., Tonda, A., Lio, P., Precioso, F., Jamnik, M., and Marra, G. Interpretable neural-symbolic concept reasoning. In ICML 2023 Workshop on Differentiable Almost Everything: Differentiable Relaxations, Algorithms, Operators, and… find

Updated 3 November 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ko-2026-scaling-reasoning-efficiently-relaxed-policy/