Elamur Research⌕
☰
home / papers / 2026 / GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control

GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control

Paper · 2026 · Marco Simoni, Aleksandar Fontana, Giulio Rossolini et al.

Summary

GRPO, a critic-free reinforcement learning method for aligning LLMs, often trains unstably and converges poorly. The authors trace this to two causes. First, tokens that several completions in a group share, such as formatting tags or common prefixes, can receive opposing advantage signals. This can lower their probability even when they are useful. Second, completions with negative rewards can flatten the token distribution and push the model toward improbable tokens, which leads to policy collapse. The KL penalty responds to this too slowly.

The proposed fix, GTPO, treats each completion as a trajectory. It skips negative updates on conflicting shared tokens and strengthens positive ones. It also adds entropy regularization and discards completions whose average entropy exceeds a provable threshold. Because KL is dropped, GTPO no longer needs a reference model.

Experiments with LLaMA-8B and Qwen2.5-3B trained on GSM8K, MATH and R1 show more stable training than SFT, GRPO and DAPO. GTPO also generalizes better, both in-distribution and out-of-distribution. This matters for cheaper, more reliable RL fine-tuning on reasoning tasks.

Key points

  • The paper identifies two GRPO failure modes: token-level penalization of shared tokens, and policy collapse, which mean output entropy signals more clearly than the KL term.
  • GTPO combines conflict-aware gradient correction with entropy control. Removing KL eliminates the reference model and speeds up training.
  • Models are LLaMA-8B and Qwen2.5-3B, trained on GSM8K, MATH and R1. Out-of-distribution tests use AIME 2024/2025 and AMC 2023, and MMLU shows that non-math knowledge is retained.
  • Entropy filtering thresholds of 0.7, 1.05 and 2.0 correspond roughly to 2, 3 and 4 equiprobable tokens. The two stricter settings keep training stable. The loose 2.0 setting lets entropy drift toward ln 2 and degrades formatting.
  • In ablations, entropy filtering and regularization (γ = 0.1) also stabilize GRPO on their own. GTPO under the same constraints still performs best. Filtering only negative-advantage completions ("OnlyNeg") over-prunes and hurts results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sapienza University of RomeRome · Italy57 works · 9,564 citations🤝National Research CouncilRome · Italy15 works · 15,649 citations🔬Institute of Informatics and TelematicsPisa · Italy3 works · 251 citations

Also on the byline: Scuola Superiore Sant'Anna.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Marco Simoni, Aleksandar Fontana, Giulio Rossolini, Andrea Saracino and Paolo Mori. 2026. GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.800

@article{simoni2026gtpo,
  title = {{GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control}},
  author = {Marco Simoni, Aleksandar Fontana, Giulio Rossolini, Andrea Saracino and Paolo Mori},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.800},
  url = {https://doi.org/10.1162/tacl.a.800}
}

References

40 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. 2021. What matters for on-policy deep actor-critic methods? A large-scale study. In 9th Internati… find
  2. Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap,… — Gemini: A Family of Highly Capable Multimodal Models
  3. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku — model card. Model card, Anthropic. Accessed: 2025-07-22. find
  4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Christopher Berner, Greg Brockman, Brooke Chan, V… find
  5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  6. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. 2025. The entropy mechanism of reinforcement learning for reasoning language models. find
  7. Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
  8. Michael Han Daniel Han and Unsloth team. 2023. Unsloth. find
  9. Shibhansh Dohare, Qingfeng Lan, and A Rupam Mahmood. 2023. Overcoming policy collapse in deep reinforcement learning. In Sixteenth European Workshop on Reinforcement Learning. find
  10. Aleksandar Fontana, Marco Simoni, Giulio Rossolini, Andrea Saracino, and Paolo Mori. 2026. On the hidden objective biases of group-based reinforcement learning. arXiv preprint arXiv:2601.05002. find
  11. Saurabh Garg, Joshua Zhanson, Emilio Parisotto, Adarsh Prasad, J. Zico Kolter, Zachary C. Lipton, Sivaraman Balakrishnan, Ruslan Salakhutdinov, and Pradeep Ravikumar. 2021. On proximal policy optimization’s heavy-tailed gradients. In Proceedings of the 38th International Conference on Machine Learni… find
  12. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  13. Andre He, Daniel Fried, and Sean Welleck. 2025. Rewarding the unlikely: Lifting GRPO beyond distribution sharpening. CoRR, abs/2506.02355.
  14. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  15. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021a. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets a…
  16. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. 2025. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXi…
  17. Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. 2022. The 37 implementation details of proximal policy optimization. In ICLR Blog Track. Https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/. find
  18. Lu Jia, Binglin Su, Du Xu, Yewei Wang, Jing Fang, and Jun Wang. 2024. Policy optimization algorithm with activation likelihood-ratio for multi-agent reinforcement learning. Neural Process. Lett., 56(6):247. find
  19. Xuying Li, Zhuo Li, Yuji Kosuga, and Victor Bian. 2025. Optimizing safe and aligned language generation: A multi-objective GRPO approach. CoRR, abs/2503.21819. — [arXiv:2503.21819](https://arxiv.org/abs/2503.21819) find
  20. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783.
  21. Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY… find
  22. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis H… find
  23. Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. 2024. No representation, no trust: Connecting representation, collapse, and trust issues in PPO. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024,… find
  24. OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
  25. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.… — Training language models to follow instructions with human feedback
  26. David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David R So, Maud Texier, and Jeff Dean. 2022. The carbon footprint of machine learning training will plateau, then shrink. Computer, 55(7):18–28. find
  27. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Sy…
  28. John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages… find
  29. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  30. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  31. Marco Simoni, Aleksandar Fontana, Giulio Rossolini, and Andrea Saracino. 2025. Improving llm reasoning for vulnerability detection via group relative policy optimization. arXiv preprint arXiv:2507.03051. — [arXiv:2507.03051](https://arxiv.org/abs/2507.03051) find
  32. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. "my answer is c": First-token probabilities do not match text answers in instruction-tuned language models. In Findings of the Association for Computational Linguistics, ACL 2024…
  33. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,…
  34. Yuhui Wang, Hao He, Xiaoyang Tan, and Yaozhong Gan. 2019. Trust region-guided proximal policy optimization. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 624–63… find
  35. Haotian Xu, Zheng Yan, Junyu Xuan, Guangquan Zhang, and Jie Lu. 2023. Improving proximal policy optimization with alpha divergence. Neurocomputing, 534:94–105. find
  36. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  37. Zhihe Yang, Xufang Luo, Zilong Wang, Dongqi Han, Zhiyuan He, Dongsheng Li, and Yunjian Xu. 2025b. Do not let low-probability tokens over-dominate in RL for llms. CoRR, abs/2505.12929. — [arXiv:2505.12929](https://arxiv.org/abs/2505.12929) find
  38. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. 2026. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222– 113244.
  39. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models.
  40. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. 2025. Group sequence policy optimization. arXiv preprint arXiv:2507.18071.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/simoni-2026-gtpo-stabilizing-group-relative-policy/