📄 Self-Distilled Policy Gradient

Paper · 2026 · Yifeng Liu, Shiyuan Zhang, Yifan Zhang et al.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

🏛️ Institutions

🎓Princeton UniversityPrinceton · United States137 works · 53,849 citations🎓University of California, Los AngelesLos Angeles · United States110 works · 58,035 citations

🧭 Fields, topics and concepts

Cite

Cite as Yifeng Liu, Shiyuan Zhang, Yifan Zhang and Quanquan Gu. 2026. Self-Distilled Policy Gradient. arXiv:2606.04036.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2606.04036 · arXiv 2606.04036

@misc{liu2026self,
  title = {{Self-Distilled Policy Gradient}},
  author = {Yifeng Liu, Shiyuan Zhang, Yifan Zhang and Quanquan Gu},
  year = {2026},
  doi = {10.48550/arxiv.2606.04036},
  eprint = {2606.04036},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.04036}
}

References

51 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Garea, S. R., Geist, M. and Bachem, O. (2024). On-policy distillation of language models: Learning from self-generated mistakes. In The twelfth international conference on learning representations.
2. Chen, G., Liao, M., Li, C. and Fan, K. (2024). Step-level value preference optimization for mathematical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024. find
3. Chen, M., Chen, G., Wang, W. and Yang, Y. (2025). Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization. arXiv preprint arXiv:2505.12346 . — [arXiv:2505.12346](https://arxiv.org/abs/2505.12346) find
4. Cheng, D., Huang, S., Zhu, X., Dai, B., Zhao, X., Zhang, Z. and Wei, F. (2026). Reasoning with exploration: An entropy perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40.
5. Comanici, G., Bieber, E., Schaekermann, M. et al. (2025). Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 .
6. Dai, M., Yang, C. and Si, Q. (2025). S-grpo: Early exit via reinforcement learning in reasoning models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
7. Fu, Y., Huang, H., Jiang, K., Zhu, Y. and Zhao, D. (2026). Revisiting on-policy distillation: Empirical failure modes and simple fixes. arXiv preprint arXiv:2603.25562 .
8. Gu, Y., Dong, L., Wei, F. and Huang, M. (2024). Minillm: Knowledge distillation of large language models. In The twelfth international conference on learning representations. find
9. Hübotter, J., Lübeck, F., Behric, L., Baumann, A., Bagatella, M., Marta, D., Hakimi, I., Shenfeld, I., Buening, T. K., Guestrin, C. et al. (2026). Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802 .
10. Kazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A. and Roux, N. L. (2024). Vineppo: Refining credit assignment in rl training of llms. arXiv preprint arXiv:2410.01679 .
11. Kwon, W. (2025). vLLM: An Efficient Inference Engine for Large Language Models. Ph.D. thesis, UC Berkeley. find
12. Lee, B. W., Cho, H. and Yoo, K. M. (2024). Instruction tuning with human curriculum. In Findings of the Association for Computational Linguistics: NAACL 2024. find
13. Li, Z., Kang, L., Xiao, F., Xing, L., Si, Q., Li, Z., Gong, W., Yang, D., Xiao, Y. and Guo, H. (2026). Outcome-grounded advantage reshaping for fine-grained credit assignment in mathematical reasoning. arXiv preprint arXiv:2601.07408 . find
14. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I. and Cobbe, K. (2023). Let’s verify step by step. In The twelfth international conference on learning representations.
15. Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S. and Lin, M. (2025). Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783 .
16. Loshchilov, I. and Hutter, F. (2018). Decoupled weight decay regularization. In International Conference on Learning Representations. find
17. Lu, K. and Lab, T. M. (2025). On-policy distillation. Thinking Machines Lab: Connectionism . find
18. Luo, L., Liu, Y., Liu, R., Phatale, S., Guo, M., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L. et al. (2024). Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592 .
19. MAA, M. A. o. A. A. M. C. (2024a). 2024 AIME-I. Accessed: 2025-05-08. find
20. MAA, M. A. o. A. A. M. C. (2024b). 2024 AIME-II. Accessed: 2025-05-08. find
21. MAA, M. A. o. A. A. M. C. (2025a). 2025 AIME-I. find
22. MAA, M. A. o. A. A. M. C. (2025b). 2025 AIME-II. find
23. Minka, T. (2005). Divergence measures and message passing . find
24. Penaloza, E., Vattikonda, D., Gontier, N., Lacoste, A., Charlin, L. and Caccia, M. (2026). Privileged information distillation for language models. arXiv preprint arXiv:2602.04942 . — [arXiv:2602.04942](https://arxiv.org/abs/2602.04942) find
25. Schulman, J. (2020). Approximating kl divergence. http://joschu.net/blog/kl-approx.html. Accessed on June 4, 2026. ↗
26. Schulman, J., Wolski, F., Dhariwal, P., Radford, A. and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 . find
27. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y. et al. (2024). Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 .
28. Shenfeld, I., Damani, M., Hübotter, J. and Agrawal, P. (2026). Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897 .
29. Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H. and Wu, C. (2025). Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems.
30. Shi, T., Wu, Y., Song, L., Zhou, T. and Zhao, J. (2025). Efficient reinforcement finetuning via adaptive curriculum learning. In NeurIPS 2025 Workshop on Efficient Reasoning. find
31. Srivastava, S. S. and Aggarwal, V. (2025). A technical survey of reinforcement learning techniques for large language models. arXiv preprint arXiv:2507.04136 . — [arXiv:2507.04136](https://arxiv.org/abs/2507.04136) find
32. Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y. and Sui, Z. (2024). Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
33. Wang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X.-H., Yang, J., Zhang, Z. et al. (2025a). Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Syste…
34. Wang, X., Chen, Y. and Zhu, W. (2021). A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence 44 4555–4576. find
35. Wang, Y., Yang, Q., Zeng, Z., Ren, L., Liu, L., Peng, B., Cheng, H., He, X., Wang, K., Gao, J. et al. (2025b). Reinforcement learning for reasoning in large language models with one training example. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
36. Wen, L., Cai, Y., Xiao, F., He, X., An, Q., Duan, Z., Du, Y., Liu, J., Tanglifu, T., Lv, X. et al. (2025). Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).
37. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8 229–256. find
38. Xiao, B., Xia, B., Yang, B., Gao, B., Shen, B., Zhang, C., He, C., Lou, C., Luo, F., Wang, G. et al. (2026). Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780 .
39. Xie, C., Pan, R., Wu, X., Zhang, Y., Fu, J., Gao, T. and Zhou, G. (2025). Unlocking exploration in rlvr: Uncertainty-aware advantage shaping for deeper reasoning. arXiv preprint arXiv:2510.10649 .
40. Xu, W., Han, R., Wang, Z., Le, L., Madeka, D., Li, L., Wang, W. Y., Agarwal, R., Lee, C.-Y. and Pfister, T. (2025). Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling. In The Thirteenth International Conference on Learning Representations.
41. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C. et al. (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388 .
42. Yang, C., Qin, C., Si, Q., Chen, M., Gu, N., Yao, D., Lin, Z., Wang, W., Wang, J. and Duan, N. (2026a). Self-distilled rlvr. arXiv preprint arXiv:2604.03128 .
43. Yang, C., Si, Q., Dai, M., Yao, D., Zheng, M., Chen, M., Lin, Z. and Wang, W. (2026b). Test-time prompt intervention. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40. find
44. Ye, T., Dong, L., Wu, X., Huang, S. and Wei, F. (2026). On-policy context distillation for language models. arXiv preprint arXiv:2602.12275 .
45. Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Dai, W., Fan, T., Liu, G., Liu, L. et al. (2025). Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 .
46. Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Kumar, A. and Agarwal, R. (2025). Generative verifiers: Reward modeling as next-token prediction. In The Thirteenth International Conference on Learning Representations.
47. Zhang, Y., Liu, Y., Yuan, H., Yuan, Y., Gu, Q. and Yao, A. C. (2026a). On the design of kl-regularized policy gradient algorithms for llm reasoning. In The Fourteenth International Conference on Learning Representations. find
48. Zhang, Z., Jiang, S., Shen, Y., Zhang, Y., Ram, D., Yang, S., Tu, Z., Xia, W. and Soatto, S. (2026b). Reinforcement-aware knowledge distillation for llm reasoning. arXiv preprint arXiv:2602.22495 .
49. Zhao, S., Xie, Z., Liu, M., Huang, J., Pang, G., Chen, F. and Grover, A. (2026). Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734 .
50. Zheng, C., Liu, S., Li, M., Chen, X.-H., Yu, B., Gao, C., Dang, K., Liu, Y., Men, R., Yang, A. et al. (2025). Group sequence policy optimization. arXiv preprint arXiv:2507.18071 .
51. Zhu, H. and Rohwer, R. (1995). Information geometric measurements of generalisation. Preprint . find

Updated 3 November 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-self-distilled-policy-gradient/