Elamur Research⌕
☰
home / papers / 2025 / CodePRM: Execution Feedback-enhanced Process Reward Model for Code Generation

CodePRM: Execution Feedback-enhanced Process Reward Model for Code Generation

Paper · 2025 · Qingyao Li, Xinyi Dai, Xiangyang Li et al.

Summary

Process reward models (PRMs) that score intermediate reasoning steps work well in math, but code is harder. Reasoning steps for programs are abstract, covering algorithms, data structures and workflow, so it is unclear which step caused a bug. The authors show that a code PRM trained the usual math way is no better than random scoring in Best-of-N selection. Their CodePRM addresses this by judging each thought step together with the code it leads to and that code's execution feedback. Tree search produces the training data: each step is labeled with the average private-test pass rate of all code derived from it, and the code and error messages are stored alongside. Qwen2.5-Coder-7B-Instruct is fine-tuned with a value head and MSE loss on these labels. At inference, a Generate-Verify-Refine (GVR) loop uses CodePRM to flag weak steps, which the generator then rewrites. On APPS, CodePRM beats open-source PRMs by a large margin. This makes it the first systematic study of process supervision for code generation.

Key points

  • Step labels are average pass rates on private tests, backpropagated through the search tree. Each sample stores the problem, the partial thoughts, the code, the verbal execution feedback and the label.
  • The backbone is Qwen2.5-Coder-7B-Instruct. Generators include GPT-4o-mini and Deepseek-Coder-V2-Lite-Instruct.
  • In Best-of-N on APPS with GPT-4o-mini and 16 rollouts, CodePRM-7B reaches pass rates of 72.07 (Intro.), 72.72 (Inter.) and 44.83 (Comp.), with pass@1 of 54, 52 and 23. Skywork-PRM-1.5B/7B and Math-Shepherd-Mistral-7B stay around 59–62 (Intro./Inter.) and 35–39 (Comp.).
  • Ablations show that adding execution feedback to the input clearly improves step evaluation, and that step granularity also matters.
  • Code, data and model weights are released at github.com/SIMONLQY/CodePRM. Use in RL training and with other base LLMs remains unexplored.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China232 works · 20,305 citations🏢Huawei TechnologiesShenzhen · China105 works · 15,705 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

🧭 Topics 1Research subjects and languages the work is about.Code & programming languages · 194 works

Cite

Cite as Qingyao Li, Xinyi Dai, Xiangyang Li, Weinan Zhang, Yasheng Wang, Ruiming Tang and Yong Yu. 2025. CodePRM: Execution Feedback-enhanced Process Reward Model for Code Generation. Findings of the Association for Computational Linguistics: ACL 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-acl.428

@inproceedings{li2025codeprm,
  title = {{CodePRM: Execution Feedback-enhanced Process Reward Model for Code Generation}},
  author = {Qingyao Li, Xinyi Dai, Xiangyang Li, Weinan Zhang, Yasheng Wang, Ruiming Tang and Yong Yu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-acl.428},
  url = {https://doi.org/10.18653/v1/2025.findings-acl.428}
}

References

44 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eshwar Ram Arunachaleswaran, Natalie Collina, and Jon Schneider. 2024. Pareto-optimal algorithms for learning in games. In Proceedings of the 25th ACM Conference on Economics and Computation, pages 490–510. find
  2. Jingchang Chen, Hongxuan Tang, Zheng Chu, Qianglong Chen, Zekun Wang, Ming Liu, and Bing Qin. 2024. Divide-and-conquer meets consensus: Unleashing the power of functions in code generation. arXiv preprint arXiv:2405.20092. — [arXiv:2405.20092](https://arxiv.org/abs/2405.20092) find
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  4. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128. find
  5. Ning Dai, Zheng Wu, Renjie Zheng, Ziyun Wei, Wenlei Shi, Xing Jin, Guanlin Liu, Chen Dun, Liang Huang, and Lin Yan. 2024. Process supervision-guided policy optimization for code generation. arXiv preprint arXiv:2410.17621.
  6. Jianqiao Lu, Zhiyang Dou, WANG Hongru, Zeyu Cao, Jianbo Dai, Yunlong Feng, and Zhijiang Guo. 2024. Autopsv: Automated process-supervised verifier. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  7. Lingyue Fu, Huacan Chai, Shuang Luo, Kounianhua Du, Weiming Zhang, Longteng Fan, Jiayi Lei, Renting Rui, Jianghao Lin, Yuchen Fang, et al. 2023. Codeapex: A bilingual programming evaluation benchmark for large language models. arXiv preprint arXiv:2309.01940. — [arXiv:2309.01940](https://arxiv.org/abs/2309.01940) find
  8. Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. 2024. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592.
  9. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992. find
  10. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568. find
  11. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938.
  12. Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. 2023. Let’s reward step by step: Step-level reward model as the navigators for reasoning. arXiv preprint arXiv:2310.10080.
  13. Nan Jiang, Xiaopeng Li, Shiqi Wang, Qiang Zhou, Soneya Binta Hossain, Baishakhi Ray, Varun Kumar, Xiaofei Ma, and Anoop Deoras. 2024. Training llms to better self-debug and explain code. arXiv preprint arXiv:2405.18649. — [arXiv:2405.18649](https://arxiv.org/abs/2405.18649) find
  14. MatrixStudio. 2024. Codeforces-python-submissions. find
  15. Cuong Chi Le, Hoang-Chau Truong-Vinh, Huy Nhat Phan, Dung Duy Le, Tien N Nguyen, and Nghi DQ Bui. 2024. Visualcoder: Guiding large language models in code execution with fine-grained multimodal chain-of-thought reasoning. arXiv preprint arXiv:2410.23402. find
  16. Hung Le, Hailin Chen, Amrita Saha, Akash Gokul, Doyen Sahoo, and Shafiq Joty. 2023. Codechain: Towards modular code generation through chain of self-revisions with representative sub-modules. arXiv preprint arXiv:2310.08992. — [arXiv:2310.08992](https://arxiv.org/abs/2310.08992) find
  17. Jierui Li, Hung Le, Yinbo Zhou, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. 2024a. Code-tree: Agent-guided tree search for code generation with large language models. arXiv preprint arXiv:2411.04329. — [arXiv:2411.04329](https://arxiv.org/abs/2411.04329) find
  18. Qingyao Li, Wei Xia, Kounianhua Du, Xinyi Dai, Ruiming Tang, Yasheng Wang, Yong Yu, and Weinan Zhang. 2024b. Rethinkmcts: Refining erroneous thoughts in monte carlo tree search for code generation. arXiv preprint arXiv:2409.09584.
  19. Wendi Li and Yixuan Li. 2024. model with q-value rankings. arXiv:2410.11287. — [arXiv:2410.11287](https://arxiv.org/abs/2410.11287) find
  20. Process reward arXiv preprint Zhenwen Liang, Ye Liu, Tong Niu, Xiangliang Zhang, Yingbo Zhou, and Semih Yavuz. 2024. Improving llm reasoning through scaling inference computation with collaborative verification. arXiv preprint arXiv:2410.05318. — [arXiv:2410.05318](https://arxiv.org/abs/2410.05318) find
  21. Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2024. Next: Teaching large language models to reason about code execution. arXiv preprint arXiv:2404.14662.
  22. Skywork o1 Team. 2024a. Skywork-prm 1.5b. https://huggingface.co/Skywork/Skywork-o1-Ope n-PRM-Qwen-2.5-1.5B. ↗
  23. Skywork o1 Team. 2024b. Skywork-prm 7b. https://huggingface.co/Skywork/Skywork-o1-Ope n-PRM-Qwen-2.5-7B. ↗
  24. OpenAI. 2024a. Gpt-4o-mini. https://openai.com/index/gpt-4o-mini-advancing-cost-efficie nt-intelligence/. Accessed: 2024-07-18. ↗
  25. OpenAI. 2024b. Openai o1 system card. Accessed: 2025-02-07.
  26. Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. 2024. Rewarding progress: Scaling automated process verifiers for llm reasoning. arXiv preprint arXiv:2410.08146.
  27. Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, et al. 2023. Pangu-coder2: Boosting large language models for code with ranking feedback. arXiv preprint arXiv:2307.14936. — [arXiv:2307.14936](https://arxiv.org/abs/2307.14936) find
  28. Yuling Shi, Songsong Wang, Chengcheng Wan, and Xiaodong Gu. 2024. From code to correctness: Closing the last mile of code generation with hierarchical debugging. arXiv preprint arXiv:2410.01215.
  29. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36.
  30. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. 2017. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815.
  31. Codeforces Team. 2024. Codeforces. https://code forces.com/. find
  32. Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. 2024a. Planning in natural language improves llm search for code generation. arXiv preprint arXiv:2409.03733. find
  33. Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al. 2024b. Openr: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.09671.
  34. Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024c. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape…
  35. Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. 2024d. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. arXiv preprint arXiv:2402.02658.
  36. Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. 2024. Watch every step! llm agent learning via iterative step-level process refinement. arXiv preprint arXiv:2406.11176. find
  37. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36. find
  38. Zhuohao Yu, Weizheng Gu, Yidong Wang, Zhengran Zeng, Jindong Wang, Wei Ye, and Shikun Zhang. 2024. Outcome-refining process supervision for code generation. arXiv preprint arXiv:2412.15118. — [arXiv:2412.15118](https://arxiv.org/abs/2412.15118) find
  39. Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, and Tong Zhang. 2024. Entropy-regularized process reward model. arXiv preprint arXiv:2412.11006.
  40. Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B Tenenbaum, and Chuang Gan. 2023. Planning with large language models for code generation. arXiv preprint arXiv:2303.05510.
  41. Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301.
  42. Li Zhong, Zilong Wang, and Jingbo Shang. 2024. Ldb: A large language model debugger via verifying runtime execution step-by-step. arXiv preprint arXiv:2402.16906.
  43. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2023. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406. find
  44. Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2025-codeprm-execution-feedback-enhanced-process/