Elamur Research⌕
☰
home / papers / 2026 / GAPO: Robust Advantage Estimation for Real-World Code LLMs

GAPO: Robust Advantage Estimation for Real-World Code LLMs

Paper · 2026 · Jianqing Zhang, Zhezheng Hao, Wei Xia et al.

Summary

Group-relative RL methods such as GRPO and DAPO are popular for post-training code LLMs because they need no critic. They compute each rollout's advantage relative to the group's mean reward. The authors show that this baseline breaks down in realistic code editing. There, per-prompt reward distributions are often skewed and contain unpredictable outliers, which distort the advantage signal. Their fix, GAPO, changes only the advantage formula. For each prompt it first locates the narrowest reward interval covering a fixed share of rollouts, framed as a Highest-Density Interval (HDI) problem and solved with a sort plus sliding-window scan. The median of that interval then becomes an adaptive baseline Q, used in place of the mean. The method drops into frameworks like verl with a few lines of code. Experiments use nine instruction-tuned models (3B–14B) and a newly collected dataset of history-aware code edits. GAPO beats GRPO and DAPO on in-domain and out-of-domain exact match, with fewer clipped updates and better GPU throughput. This makes RL post-training of code models on noisy production data more stable.

Key points

  • Dataset: 51,844 real-world code-editing tasks in 10 programming languages, led by Go (37.71%), Python (22.14%) and Java (21.03%). Each task pairs a prompt (context, edit history, edit range) with the ground-truth edited snippet.
  • Before training, Qwen2.5-Coder-14B rollout rewards were roughly normal for 68.4% of prompts. They were left-skewed for 15.5%, right-skewed for 14.4%, and approximately symmetric for 1.7%.
  • Models evaluated include Mistral-v0.3, Qwen2.5, Qwen3, Qwen2.5-Coder and DeepSeek-Coder. GAPO gains up to 4.35 exact-match points in-domain and 5.30 out-of-domain over GRPO/DAPO.
  • Ablations show that changing only the numerator to the median hurts performance. The best variant uses the median of the HDI in both the numerator and the matching deviation term. HDI computation costs O(n log n) per group.
  • Stated limitation: searching for a single contiguous high-SNR interval may misestimate multi-peak reward distributions. Code is released as verl-GAPO on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🏢TencentShenzhen · China134 works · 19,743 citations🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🎓Shenzhen UniversityShenzhen · China16 works · 95 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 worksComputer Science Applications · 102 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksCode & programming languages · 129 works

Cite

Cite as Jianqing Zhang, Zhezheng Hao, Wei Xia, Hande Dong, Hong Wang, Chenxing Wei, Yuyan Zhou, Yubin Qi, Qiang Lin and Jian Cao. 2026. GAPO: Robust Advantage Estimation for Real-World Code LLMs. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.875

@inproceedings{zhang2026gapo,
  title = {{GAPO: Robust Advantage Estimation for Real-World Code LLMs}},
  author = {Jianqing Zhang, Zhezheng Hao, Wei Xia, Hande Dong, Hong Wang, Chenxing Wei, Yuyan Zhou, Yubin Qi, Qiang Lin and Jian Cao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.875},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.875}
}

References

38 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  2. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems. find
  3. Le Deng, Xiaoxue Ren, Chao Ni, Ming Liang, David Lo, and Zhongxin Liu. 2025. Enhancing project-specific code completion by inferring internal api information. IEEE Transactions on Software Engineering. find
  4. Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi-Sangdeh, Han Liu, and Saleema Amershi. 2023. Aligning offline metrics and human judgments of value for code generation models. In Findings of the Association for Computational Linguistics: ACL 2023. find
  5. Bernd Frauenknecht, Devdutt Subhasish, Friedrich Solowjow, and Sebastian Trimpe. 2025. On rollouts in model-based reinforcement learning. In The Thirteenth International Conference on Learning Representations. find
  6. Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. 2024. Rlef: Grounding code llms in execution feedback with reinforcement learning. arXiv preprint arXiv:2410.02089.
  7. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. — Qwen2.5-Coder Technical Report
  8. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  9. Nan Jiang, Xiaopeng Li, Shiqi Wang, Qiang Zhou, Soneya B Hossain, Baishakhi Ray, Varun Kumar, Xiaofei Ma, and Anoop Deoras. 2024. Ledex: Training llms to better self-debug and explain code. Advances in Neural Information Processing Systems. find
  10. Onyedikachi O John. 2015. Robustness of quantile regression to outliers. American Journal of Applied Mathematics and Statistics, 3(2):86–88. find
  11. Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. 2025. Llm post-training: A deep dive into reasoning large language models. arXiv preprint arXiv:2502.21321. — [arXiv:2502.21321](https://arxiv.org/abs/2502.21321) find
  12. VI Lcvenshtcin. 1966. Binary coors capable or ‘correcting deletions, insertions, and reversals. In Soviet physics-doklady. find
  13. Jia Li, Ge Li, Zhuo Li, Zhi Jin, Xing Hu, Kechi Zhang, and Zhiyi Fu. 2023. Codeeditor: Learning to edit source code with pre-trained models. ACM Transactions on Software Engineering and Methodology, 32(6):1–22. find
  14. Weichen Li, Albert Jan, Baishakhi Ray, Junfeng Yang, Chengzhi Mao, and Kexin Pei. 2025. Editlord: Learning code transformation rules for code editing. arXiv preprint arXiv:2504.15284. — [arXiv:2504.15284](https://arxiv.org/abs/2504.15284) find
  15. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems.
  16. TIOBE Software BV. 2025. Tiobe index. https://www.tiobe.com/tiobe-index/. Accessed: 2025-12-31. ↗
  17. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, and 1 others. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092–1097.
  18. Hongcheng Wang, Yinuo Huang, Sukai Wang, Guanghui Ren, and Hao Dong. 2025a. Grpo-ma: Multi-answer generation in grpo for stable and efficient chain-of-thought training. arXiv preprint arXiv:2509.24494. — [arXiv:2509.24494](https://arxiv.org/abs/2509.24494) find
  19. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025a. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783.
  20. Hu Wang, Congbo Ma, Ian Reid, and Mohammad Yaqub. 2025b. Kalman filter enhanced grpo for reinforcement learning-based language model reasoning. arXiv preprint arXiv:2505.07527. — [arXiv:2505.07527](https://arxiv.org/abs/2505.07527) find
  21. Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, and 1 others. 2025b. Part i: Tricks or traps? a deep dive into rl for llm reasoning. arXiv preprint arXiv:2508.08221. find
  22. Junqiao Wang, Zeng Zhang, Yangfan He, Zihao Zhang, Xinyuan Song, Yuyang Song, Tianyu Shi, Yuchen Li, Hengyuan Xu, Kunyu Wu, and 1 others. 2024. Enhancing code llms with reinforcement learning in code generation: A survey. arXiv preprint arXiv:2412.20367. — [arXiv:2412.20367](https://arxiv.org/abs/2412.20367) find
  23. David S Moore, George P McCabe, and Bruce A Craig. 2009. Introduction to the Practice of Statistics, volume 4. WH Freeman New York. find
  24. Yongyu Mu, Jiali Zeng, Bei Li, Xinyan Guan, Fandong Meng, Jie Zhou, Tong Xiao, and Jingbo Zhu. 2025. Dissecting long reasoning models: An empirical study. arXiv preprint arXiv:2506.04913. — [arXiv:2506.04913](https://arxiv.org/abs/2506.04913) find
  25. Daye Nam, Ahmed Omran, Ambar Murillo, Saksham Thakur, Abner Araujo, Marcel Blistein, Alexander Frömmgen, Vincent Hellendoorn, and Satish Chandra. 2025. Prompting llms for code editing: Struggles and remedies. arXiv preprint arXiv:2504.20196. — [arXiv:2504.20196](https://arxiv.org/abs/2504.20196) find
  26. Ben O’Neill. 2022. Smallest covering regions and highest density regions for discrete distributions. Computational Statistics, 37(3):1229–1254. find
  27. Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590. — [arXiv:2302.06590](https://arxiv.org/abs/2302.06590) find
  28. Qwen Team. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  29. Peter J Rousseeuw and Mia Hubert. 2011. Robust statistics for outlier detection. Wiley interdisciplinary reviews: Data mining and knowledge discovery, 1(1):73–79. find
  30. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  31. Chenxing Wei, Jiarui Yu, Ying Tiffany He, Hande Dong, Yao Shu, and Fei Yu. 2025. Redit: Reward dithering for improved llm policy optimization. arXiv preprint arXiv:2506.18631. — [arXiv:2506.18631](https://arxiv.org/abs/2506.18631) find
  32. Jie Wu, Haoling Li, Xin Zhang, Jianwen Luo, Yangyu Huang, Ruihang Chu, Yujiu Yang, and Scarlett Li. 2025a. Iterpref: Focal preference learning for code generation via iterative debugging. arXiv preprint arXiv:2503.02783. — [arXiv:2503.02783](https://arxiv.org/abs/2503.02783) find
  33. Junkang Wu, Kexin Huang, Jiancan Wu, An Zhang, Xiang Wang, and Xiangnan He. 2025b. Quantile advantage estimation for entropy-safe reasoning. arXiv preprint arXiv:2509.22611. — [arXiv:2509.22611](https://arxiv.org/abs/2509.22611) find
  34. Zifan Wu, Chao Yu, Chen Chen, Jianye Hao, and Hankz Hankui Zhuo. 2022. Plan to predict: Learning an uncertainty-foreseeing model for model-based reinforcement learning. Advances in Neural Information Processing Systems. find
  35. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  36. Burak Yetiştiren, Işık Özsoy, Miray Ayerdem, and Eray Tüzün. 2023. Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt. arXiv preprint arXiv:2304.10778. — [arXiv:2304.10778](https://arxiv.org/abs/2304.10778) find
  37. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2025. Dapo: Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, and 1 others. 2025. Group sequence…
  38. Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. 2025. The surprising effectiveness of negative reinforcement in llm reasoning. arXiv preprint arXiv:2506.01347. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-gapo-robust-advantage-estimation-real/