Elamur Research⌕
☰
home / papers / 2024 / Process Supervision-Guided Policy Optimization for Code Generation

Process Supervision-Guided Policy Optimization for Code Generation

Paper · 2024 · Ning Dai, Zheng Wu, Renjie Zheng et al.

Summary

RL from unit-test feedback helps LLMs write better code, but the reward is sparse. A single pass/fail signal arrives only after the whole program runs, and a program that fails every test gives the model nothing to learn from. The authors (ByteDance and Oregon State University) propose Process Supervision-Guided Policy Optimization (PSGPO), which adds a Process Reward Model (PRM) that scores each generated line of code.

PRM training labels are produced automatically. A policy response is split into line-level prefixes, best-of-K completions are sampled from each prefix and executed against unit tests, and a binary search finds the point after which no completion succeeds. In RL, the PRM plays two roles at once: it supplies dense per-line rewards, and it initializes the value function. This combination worked best and raised pass rates on HumanEval, MBPP and LiveCodeBench for two base LLMs, with the clearest gains on long outputs. The work matters because it brings process supervision, previously explored mostly for math reasoning, to code generation and offers a practical recipe for using it.

Key points

  • PRM labels need no human annotation. A prefix counts as correct (+1) if any of K sampled completions passes all unit tests, and incorrect (−1) otherwise. Binary search over lines keeps the number of checks low.
  • Using the PRM both as a dense reward and as the value-model initialization gave a stronger synergy than either use alone. The authors also report better exploration and learning efficiency.
  • In the response-length analysis, PRM-trained models improved Pass@1 by about 9% over the baseline. Gains were consistent for responses longer than 100 tokens, while shorter responses saw neutral or slightly negative effects.
  • PRM data selection affects results. On LiveCodeBench with Doubao-Lite-RL, the scores were:
    • Full data: 26.9
    • Remove Hard: 27.8
    • Medium Only: 26.9
    • Revised Only: 29.8 (best)
  • Policy quality kept improving as more responses per prompt were collected for PRM data. The main limitations are:
    • noisy automatic labels;
    • high compute cost;
    • reliance on unit tests, which limits use in tasks without clear correctness checks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

🧭 Topics 1Research subjects and languages the work is about.Code & programming languages · 194 works

Cite

Cite as Ning Dai, Zheng Wu, Renjie Zheng, Ziyun Wei, Wenlei Shi, Xing Jin, Guanlin Liu, Chen Dun, Liang Huang and Lin Yan. 2024. Process Supervision-Guided Policy Optimization for Code Generation. arXiv:2410.17621.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.17621 · arXiv 2410.17621

@misc{dai2024process,
  title = {{Process Supervision-Guided Policy Optimization for Code Generation}},
  author = {Ning Dai, Zheng Wu, Renjie Zheng, Ziyun Wei, Wenlei Shi, Xing Jin, Guanlin Liu, Chen Dun, Liang Huang and Lin Yan},
  year = {2024},
  doi = {10.48550/arxiv.2410.17621},
  eprint = {2410.17621},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.17621}
}

References

34 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. trained on code. CoRR, abs/2107.03374, 2021b. URL https://arxiv.org/abs/2107.03374. ↗
  2. Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., Umapathi, L. K., Anderson, C. J., Zi, Y., Poirier, J. L., Schoelkopf, H., Troshin, S., Abulkhanov, D., Romero, M., Lappert, M., Toni, F. D., del Rı́o, B. G., Liu, Q., Bose, S., Bha… — SantaCoder: don't reach for the stars!
  3. Christopoulou, F., Lampouras, G., Gritta, M., Zhang, G., Guo, Y., Li, Z., Zhang, Q., Xiao, M., Shen, B., Li, L., Yu, H., Yan, L., Zhou, P., Wang, X., Ma, Y., Iacobacci, I., Wang, Y., Liang, G., Wei, J., Jiang, X., Wang, Q., and Liu, Q. Pangu-coder: Program synthesis with function-level language mode… — PanGu-Coder: Program Synthesis with Function-Level Language Modeling
  4. Dou, S., Liu, Y., Jia, H., Xiong, L., Zhou, E., Shan, J., Huang, C., Shen, W., Fan, X., Xi, Z., et al. Stepcoder: Improve code generation with reinforcement learning from compiler feedback. arXiv preprint arXiv:2402.01391, 2024. — [arXiv:2402.01391](https://arxiv.org/abs/2402.01391) find
  5. Austin, J., Odena, A., Nye, M. I., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C. J., Terry, M., Le, Q. V., and Sutton, C. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URL https://arxiv.org/abs/2108.07732. ↗
  6. Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Giorno, A. D., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y. Textbooks are all you need. CoRR, abs/2306.11644, 2023. d… find
  7. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023.
  8. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C.… — Evaluating Large Language Models Trained on Code
  9. Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y. K., Luo, F., Xiong, Y., and Liang, W. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024. URL https://arxiv.org/abs/ 2401.14196. ↗
  10. Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
  11. Jiao, F., Qin, C., Liu, Z., Chen, N. F., and Joty, S. Learning planning-based reasoning by trajectories collection and process reward synthesizing. CoRR, abs/2402.00658, 2024. doi: 10.48550/ARXIV.2402.00658. URL https://doi.org/10.48550/arXiv.2402.00658. ↗
  12. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C.… — SPoC: Search-based Pseudocode to Code
  13. Le, H., La Cava, W., and Moore, J. H. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. arXiv preprint arXiv:2207.01780, 2022. 2024. doi: 10.48550/ARXIV.2406.06592. URL https://doi.org/10.48550/arXiv.2406.06592. ↗
  14. Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering code large language models with evol-instruct, 2023. URL https://arxiv.org/abs/2306.08568. ↗
  15. Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T. Y., Wang, T., Dehaene, O., Davaadorj, M., Lamy-Poirier, J., Monteiro, J., Shliazhko, O., Gontier, N., Meade, N., Zebaze, A., Yee, M.-H., Umapathi, L. K… find
  16. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774. ↗
  17. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions w… — Training language models to follow instructions with human feedback
  18. QwenTeam. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/. ↗
  19. Li, Y., Choi, D., Chung, J., Guo, A., Yang, T.-Y., et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022.
  20. Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, B., Finn, C., and Niekum, S. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv preprint arXiv:2406.02900, 2024. — [arXiv:2406.02900](https://arxiv.org/abs/2406.02900) find
  21. Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. Textbooks are all you need II: phi-1.5 technical report. CoRR, abs/2309.05463, 2023b. doi: 10.48550/ARXIV.2309.05463. URL https://doi.org/10.48550/arXiv.2309.05463. ↗
  22. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. CoRR, abs/2305.20050, 2023. doi: 10.48550/ ARXIV.2305.20050. URL https://doi.org/ 10.48550/arXiv.2305.20050.
  23. Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N.,… — Code Llama: Open Foundation Models for Code
  24. Liu, J., Zhu, Y., Xiao, K., Fu, Q., Han, X., Yang, W., and Ye, D. RLTF: reinforcement learning from unit test feedback. Trans. Mach. Learn. Res., 2023, 2023. URL https://openreview.net/forum?id=hjYmsV6nXZ. ↗
  25. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347. ↗
  26. Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., Gong, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S. K., Fu, S., and Liu, S. Codexglue: A machine learning benchmark dataset for code un… find
  27. Shojaee, P., Jain, A., Tipirneni, S., and Reddy, C. K. Execution-based code generation using deep reinforcement learning. arXiv preprint arXiv:2301.13816, 2023.
  28. Skalse, J., Howe, N., Krasheninnikov, D., and Krueger, D. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460–9471, 2022. find
  29. Luo, L., Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L., Sun, J., and Rastogi, A. Improve mathematical reasoning in language models by automated process supervision. CoRR, abs/2406.06592, Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z.…
  30. Wang, Z., Li, Y., Wu, Y., Luo, L., Hou, L., Yu, H., and Shang, J. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. CoRR, abs/2402.02658, 2024b. doi: 10.48550/ARXIV.2402.02658. URL https://doi.org/10.48550/arXiv.2402.02658. ↗
  31. Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992. find
  32. Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wan… find
  33. Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Wang, Z., Shen, L., Wang, A., Li, Y., Su, T., Yang, Z., and Tang, J. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x, 2024. URL https://arxiv.org/ abs/2303.17568. ↗
  34. A. A Typical Example of the Learned Line-wise Rewards find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dai-2024-process-supervision-guided-policy-optimization/