Elamur Research⌕
☰
home / papers / 2026 / A Comprehensive Survey of Process Reward Models: Data Generation, Model Construction, and Usage

A Comprehensive Survey of Process Reward Models: Data Generation, Model Construction, and Usage

Paper · 2026 · Congmin Zheng, Jiachen Zhu, Zhuoying Ou et al.

Summary

Reasoning alignment for LLMs has mostly relied on outcome reward models (ORMs). These judge only the final answer, so an extended chain of thought receives just one coarse signal. This survey covers Process Reward Models (PRMs), which score individual steps or whole trajectories. It frames the field as a cycle: produce process-level data, train PRMs, apply them, and use the improved policies to create fresh data.

The survey sorts data sources into three groups, each trading fidelity against scale:

  • human labeling,
  • automated pipelines (symbolic or formal checks, MCTS, execution feedback),
  • semi-automated hybrids.

PRM designs are split into discriminative, generative, implicit and other architectures. Usage covers test-time scaling (re-ranking, verifier-guided decoding, search) and RL with dense step-level rewards. The authors review applications in math, code, multimodal reasoning, robotics and agents, summarize benchmarks, and compare PRMs with ORMs and rule-based rewards. In that comparison, PRMs lead on granularity, interpretability and versatility but are weaker on reward-hacking resistance and cross-domain transfer.

Note: the PDF header shows a different title, with a subtitle about moving from outcome signals to process supervision.

Key points

  • Representative data resources:
    • PRM800K: human step-level labels.
    • Math-Shepherd, OmegaPRM, FOVER (Z3/Isabelle) and CodePRM: automated labels.
    • MedS3: seeded with about 8,000 curated medical examples, then expanded via MCTS.
  • ProcessBench and PRMBench solutions average 7.1 and 13.4 steps respectively. This makes PRM annotation several times costlier than ORM annotation.
  • On PRM800K, step-level token-length SD is 71.7 versus 50.6 at trajectory level. The authors read this as a sign that step signals are noisier and easier to hack through verbose steps.
  • Generative PRMs such as ThinkPRM and GenRM output natural-language justifications. Socratic-PRMBench categorizes types of reasoning errors.
  • The authors maintain an updated companion list at github.com/despzcm/Survey-of-Process-Reward-Model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,225 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🎓University College LondonLondon · United Kingdom107 works · 22,923 citations🎓University of BristolBristol · United Kingdom18 works · 822 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksBusiness, Management and Accounting · 40 worksManagement Information Systems · 9 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Congmin Zheng, Jiachen Zhu, Zhuoying Ou, Yuxiang Chen, Kangning Zhang, Rong Shan, Zeyu Zheng, Mengyue Yang, Jianghao Lin, Yong Yu and Weinan Zhang. 2026. A Comprehensive Survey of Process Reward Models: Data Generation, Model Construction, and Usage. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.163

@inproceedings{zheng2026comprehensive,
  title = {{A Comprehensive Survey of Process Reward Models: Data Generation, Model Construction, and Usage}},
  author = {Congmin Zheng, Jiachen Zhu, Zhuoying Ou, Yuxiang Chen, Kangning Zhang, Rong Shan, Zeyu Zheng, Mengyue Yang, Jianghao Lin, Yong Yu and Weinan Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.163},
  url = {https://doi.org/10.18653/v1/2026.acl-long.163}
}

References

109 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  2. Lang Cao, Renhong Chen, Yingtian Zou, Chao Peng, Wu Ning, Huacong Xu, Qian Chen, Yuxian Wang, Peishuo Su, Mofan Peng, Zijie Chen, and Yitong Li. 2025a. More bang for the buck: Process reward modeling with entropy-driven uncertainty. Preprint, arXiv:2503.22233. — [arXiv:2503.22233](https://arxiv.org/abs/2503.22233) find
  3. Qi Cao, Ruiyi Wang, Ruiyi Zhang, Sai Ashish Somayajula, and Pengtao Xie. 2025b. Dreamprm: Domain-reweighted process reward model for multimodal reasoning. arXiv preprint arXiv:2505.20241. — [arXiv:2505.20241](https://arxiv.org/abs/2505.20241) find
  4. Hyungjoo Chae, Sunghwan Kim, Junhee Cho, Seungone Kim, Seungjun Moon, Gyeom Hwangbo, Dongha Lim, Minjin Kim, Yeonjun Hwang, Minju Gwak, and 1 others. 2025. Web-shepherd: Advancing prms for reinforcing web agents. arXiv preprint arXiv:2505.15277. — [arXiv:2505.15277](https://arxiv.org/abs/2505.15277) find
  5. Huacan Chai, Zijie Cao, Maolin Ran, Yingxuan Yang, Jianghao Lin, Xin Peng, Hairui Wang, Renjie Ding, Ziyu Wan, Muning Wen, and 1 others. 2025. Parl-mt: Learning to call functions in multi-turn conversation with progress awareness. arXiv preprint arXiv:2509.23206. — [arXiv:2509.23206](https://arxiv.org/abs/2509.23206) find
  6. Chi-Min Chan, Chunpu Xu, Junqi Zhu, Jiaming Ji, Donghai Hong, Pengcheng Wen, Chunyang Jiang, Zhen Ye, Yaodong Yang, Wei Xue, and 1 others. 2025. Boosting policy and process reward models with monte carlo tree search in open-domain qa. In Findings of the Association for Computational Linguistics: ACL… find
  7. Cong Chen, Kaixiang Ji, Hao Zhong, Muzhi Zhu, Anzhou Li, Guo Gan, Ziyuan Huang, Cheng Zou, Jiajia Liu, Jingdong Chen, and 1 others. 2025a. Gui-shepherd: Reliable process reward and verification for long-sequence gui tasks. arXiv preprint arXiv:2509.23738. — [arXiv:2509.23738](https://arxiv.org/abs/2509.23738) find
  8. Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024. Alphamath almost zero: process supervision without process. Advances in Neural Information Processing Systems, 37:27689–27724.
  9. Hongzhan Chen, Tao Yang, Shiping Gao, Ruijun Chen, Xiaojun Quan, Hongtao Tian, and Ting Yao. 2025b. Discriminative policy optimization for token-level reward models. Preprint, arXiv:2505.23363. — [arXiv:2505.23363](https://arxiv.org/abs/2505.23363) find
  10. Hongzhan Chen, Tao Yang, Shiping Gao, Ruijun Chen, Xiaojun Quan, Hongtao Tian, and Ting Yao. 2025c. Discriminative policy optimization for token-level reward models. arXiv preprint arXiv:2505.23363. — [arXiv:2505.23363](https://arxiv.org/abs/2505.23363) find
  11. Jiabei Chen, Guang Liu, Shizhu He, Kun Luo, Yao Xu, Jun Zhao, and Kang Liu. 2025d. Search-in-context: Efficient multi-hop qa over long contexts via monte carlo tree search with dynamic kv retrieval. In Findings of the Association for Computational Linguistics: ACL 2025, pages 26443–26455. find
  12. Wenxiang Chen, Wei He, Zhiheng Xi, Honglin Guo, Boyang Hong, Jiazheng Zhang, Rui Zheng, Nijun Li, Tao Gui, Yun Li, and 1 others. 2025e. Better process supervision with bi-directional rewarding signals. arXiv preprint arXiv:2503.04618. find
  13. Xinquan Chen, Bangwei Liu, Xuhong Wang, Yingchun Wang, and Chaochao Lu. 2025f. Vrprm: Process reward modeling via visual reasoning. arXiv preprint arXiv:2508.03556. — [arXiv:2508.03556](https://arxiv.org/abs/2508.03556) find
  14. Jie Cheng, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Gang Xiong, Yisheng Lv, and Fei-Yue Wang. 2025. Stop summation: Min-form credit assignment is all process reward model needs for reasoning. arXiv preprint arXiv:2504.15275. — [arXiv:2504.15275](https://arxiv.org/abs/2504.15275) find
  15. Sanjiban Choudhury. 2025. Process reward models for llm agents: Practical framework and directions. arXiv preprint arXiv:2502.10325. — [arXiv:2502.10325](https://arxiv.org/abs/2502.10325) find
  16. Ning Dai, Zheng Wu, Renjie Zheng, Ziyun Wei, Wenlei Shi, Xing Jin, Guanlin Liu, Chen Dun, Liang Huang, and Lin Yan. 2024. Process supervision-guided policy optimization for code generation. arXiv preprint arXiv:2410.17621. find
  17. Yuyang Ding, Xinyu Shi, Juntao Li, Xiaobo Liang, Zhaopeng Tu, and Min Zhang. 2025. Scan: Self-denoising monte carlo annotation for robust process reward learning. arXiv preprint arXiv:2509.16548. — [arXiv:2509.16548](https://arxiv.org/abs/2509.16548) find
  18. Lingxiao Du, Fanqing Meng, Zongkai Liu, Zhixiang Zhou, Ping Luo, Qiaosheng Zhang, and Wenqi Shao. 2025. Mm-prm: Enhancing multimodal mathematical reasoning with scalable step-level supervision. arXiv preprint arXiv:2505.13427. find
  19. Keyu Duan, Zichen Liu, Xin Mao, Tianyu Pang, Changyu Chen, Qiguang Chen, Michael Qizhe Shieh, and Longxu Dou. 2025. Efficient process reward model training via active learning. arXiv preprint arXiv:2504.10559. — [arXiv:2504.10559](https://arxiv.org/abs/2504.10559) find
  20. Zhangying Feng, Qianglong Chen, Ning Lu, Yongqian Li, Siqi Cheng, Shuangmu Peng, Duyu Tang, Shengcai Liu, and Zhirui Zhang. 2025a. Is prm necessary? problem-solving rl implicitly induces prm capability in llms. Preprint, arXiv:2505.11227. — [arXiv:2505.11227](https://arxiv.org/abs/2505.11227) find
  21. Zhaopeng Feng, Jiahan Ren, Jiayuan Su, Jiamei Zheng, Zhihang Tang, Hongwei Wang, and Zuozhu Liu. 2025b. Mt-rewardtree: A comprehensive framework for advancing llm-based machine translation via reward modeling. arXiv preprint arXiv:2503.12123. find
  22. Shubham Gandhi, Jason Tsay, Jatin Ganhotra, Kiran Kate, and Yara Rizk. 2025. When agents go astray: Course-correcting swe agents with prms. arXiv preprint arXiv:2509.02360. — [arXiv:2509.02360](https://arxiv.org/abs/2509.02360) find
  23. Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. 2025. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519.
  24. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. 2025. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.17746.
  25. Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, and 1 others. 2025a. Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.22312. find
  26. Mingqian He, Yongliang Shen, Wenqi Zhang, Zeqi Tan, and Weiming Lu. 2024. Advancing process verification for large language models via tree-based preference learning. arXiv preprint arXiv:2407.00390. find
  27. Tao He, Rongchuan Mu, Lizi Liao, Yixin Cao, Ming Liu, and Bing Qin. 2025b. Good learners think their thinking: Generative prm makes large reasoning model more efficient math learner. arXiv preprint arXiv:2507.23317. — [arXiv:2507.23317](https://arxiv.org/abs/2507.23317) find
  28. Pengfei Hu, Zhenrong Zhang, Qikai Chang, Shuhang Liu, Jiefeng Ma, Jun Du, Jianshu Zhang, Quan Liu, Jianqing Gao, Feng Ma, and 1 others. 2025a. find
  29. Prm-bas: Enhancing multimodal reasoning through prm-guided beam annealing search. arXiv preprint arXiv:2504.10222. — [arXiv:2504.10222](https://arxiv.org/abs/2504.10222) find
  30. Zhiyuan Hu, Shiyun Xiong, Yifan Zhang, See-Kiong Ng, Anh Tuan Luu, Bo An, Shuicheng Yan, and Bryan Hooi. 2025b. Guiding vlm agents with process rewards at inference time for gui navigation. arXiv preprint arXiv:2504.16073. — [arXiv:2504.16073](https://arxiv.org/abs/2504.16073) find
  31. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
  32. Shuyang Jiang, Yusheng Liao, Zhe Chen, Ya Zhang, Yanfeng Wang, and Yu Wang. 2025. Meds 3 : Towards medical slow thinking with self-evolved soft dual-sided process supervision. arXiv preprint arXiv:2501.12051. — [arXiv:2501.12051](https://arxiv.org/abs/2501.12051) find
  33. Ryo Kamoi, Yusen Zhang, Nan Zhang, Sarkar Snigdha Sarathi Das, and Rui Zhang. 2025. Generalizable process reward models via formally verified training data. arXiv preprint arXiv:2505.15960. — [arXiv:2505.15960](https://arxiv.org/abs/2505.15960) find
  34. Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Mingyeong Moon, Kiril Gashteovski, Carolin Lawrence, Julia Hockenmaier, Graham Neubig, and 1 others. 2025. Scaling evaluation-time compute with reasoning models as process evaluators. arXiv preprint arXiv:2503.19877. — [arXiv:2503.19877](https://arxiv.org/abs/2503.19877) find
  35. Dong Bok Lee, Seanie Lee, Sangwoo Park, Minki Kang, Jinheon Baek, Dongki Kim, Dominik Wagner, Jiong-dao Jin, Heejun Lee, Tobias Bocklet, and 1 others. 2025. Rethinking reward models for multi-domain test-time scaling. arXiv preprint arXiv:2510.00492. — [arXiv:2510.00492](https://arxiv.org/abs/2510.00492) find
  36. Mukai Li, Qingcheng Zeng, Tianqing Fang, Zhenwen Liang, Linfeng Song, Qi Liu, Haitao Mi, and Dong Yu. 2026. Verified critical step optimization for llm agents. arXiv preprint arXiv:2602.03412. find
  37. Qingyao Li, Xinyi Dai, Xiangyang Li, Weinan Zhang, Yasheng Wang, Ruiming Tang, and Yong Yu. 2025a. Codeprm: Execution feedback-enhanced process reward model for code generation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8169–8182. find
  38. Ruosen Li, Ziming Luo, and Xinya Du. 2024. Fine-grained hallucination detection and mitigation in language model mathematical reasoning. find
  39. Wendi Li and Yixuan Li. 2024. model with q-value rankings. arXiv:2410.11287. — [arXiv:2410.11287](https://arxiv.org/abs/2410.11287) find
  40. Process reward arXiv preprint Xiang Li, Haiyang Yu, Xinghua Zhang, Ziyang Huang, Shizhu He, Kang Liu, Jun Zhao, Fei Huang, and Yongbin Li. 2025b. Socratic-prmbench: Benchmarking process reward models with systematic reasoning patterns. arXiv preprint arXiv:2505.23474. — [arXiv:2505.23474](https://arxiv.org/abs/2505.23474) find
  41. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  42. Jianghao Lin, Yuanyuan Shi, Xin Peng, Renjie Ding, Hairui Wang, Yuxuan Peng, Bizhe Bai, Weixi Song, Fengshuo Bai, Huacan Chai, and 1 others. 2025. Toolprm: Fine-grained inference scaling of structured outputs for function calling. arXiv preprint arXiv:2510.14703. find
  43. Yuliang Liu, Junjie Lu, Zhaoling Chen, Chaofeng Qu, Jason Klein Liu, Chonghan Liu, Zefan Cai, Yunhui Xia, Li Zhao, Jiang Bian, and 1 others. 2025. Adaptivestep: Automatically dividing reasoning step through model confidence. arXiv preprint arXiv:2502.13943. — [arXiv:2502.13943](https://arxiv.org/abs/2502.13943) find
  44. Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. 2025. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv:2505.18719. find
  45. Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, and 1 others. 2024. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592.
  46. Ruilin Luo, Zhuofan Zheng, Yifan Wang, Xinzhe Ni, Zicheng Lin, Songtao Jiang, Yiyao Yu, Chufan Shi, Ruihang Chu, Jin Zeng, and 1 others. 2025. Ursa: Understanding and verifying chain-of-thought reasoning in multimodal mathematics. arXiv preprint arXiv:2501.04686. — [arXiv:2501.04686](https://arxiv.org/abs/2501.04686) find
  47. Jie Ma, Shihao Qi, Rui Xing, Ziang Yin, Bifan Wei, Jun Liu, and Tongliang Liu. 2025. From static to dynamic: Adaptive monte carlo search for mathematical process supervision. arXiv preprint arXiv:2509.24351. — [arXiv:2509.24351](https://arxiv.org/abs/2509.24351) find
  48. Roy Miles, Aysim Toker, Andreea-Maria Oncescu, Songcen Xu, Jiankang Deng, and Ismail Elezi. 2026. Test-time scaling with diffusion language models via reward-guided stitching. arXiv preprint arXiv:2602.22871. — [arXiv:2602.22871](https://arxiv.org/abs/2602.22871) find
  49. Shuo Nie, Hexuan Deng, Chao Wang, Ruiyu Fang, Xuebo Liu, Shuangyong Song, Yu Li, Min Zhang, and Xuelong Li. 2026. Stop rewarding hallucinated steps: Faithfulness-aware step-level reinforcement learning for small reasoning models. arXiv preprint arXiv:2602.05897. — [arXiv:2602.05897](https://arxiv.org/abs/2602.05897) find
  50. Tej Deep Pala, Panshul Sharma, Amir Zadeh, Chuan Li, and Soujanya Poria. 2025a. Error typing for smarter rewards: Improving process reward models with error-aware hierarchical supervision. Preprint, arXiv:2505.19706. find
  51. Tej Deep Pala, Panshul Sharma, Amir Zadeh, Chuan Li, and Soujanya Poria. 2025b. Error typing for smarter rewards: Improving process reward models with error-aware hierarchical supervision. arXiv preprint arXiv:2505.19706. find
  52. Miao Peng, Nuo Chen, Zongrui Suo, and Jia Li. 2025. Rewarding graph reasoning process makes llms more generalized reasoners. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 2257–2268. find
  53. Md Imbesat Hassan Rizvi, Xiaodan Zhu, and Iryna Gurevych. 2025. Spare: Single-pass annotation with reference-guided evaluation for automatic process supervision and reward modelling. arXiv preprint arXiv:2506.15498. — [arXiv:2506.15498](https://arxiv.org/abs/2506.15498) find
  54. Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. 2024. Rewarding progress: Scaling automated process verifiers for llm reasoning. arXiv preprint arXiv:2410.08146.
  55. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  56. Rituraj Sharma, Weiyuan Chen, Noah Provenzano, and Tu Vu. 2026. Prism: Pushing the frontier of deep think via process reward model-guided inference. arXiv preprint arXiv:2603.02479. — [arXiv:2603.02479](https://arxiv.org/abs/2603.02479) find
  57. Shuaijie She, Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang, and Shujian Huang. 2025. R-prm: Reasoning-driven process reward modeling. Preprint, arXiv:2503.21295.
  58. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.
  59. Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. 2025. Prmbench: A fine-grained and challenging benchmark for process-level reward models. arXiv preprint arXiv:2501.03124.
  60. Lin Sun, Chuang Liu, Xiaofeng Ma, Tao Yang, Weijia Lu, and Ning Wu. 2025a. Freeprm: Training process reward models without ground truth process labels. arXiv preprint arXiv:2506.03570. — [arXiv:2506.03570](https://arxiv.org/abs/2506.03570) find
  61. Wei Sun, Qianlong Du, Fuwei Cui, and Jiajun Zhang. 2025b. An efficient and precise training data construction framework for process-supervised reward model in mathematical reasoning. arXiv preprint arXiv:2503.02382.
  62. Xiaoyu Tan, Tianchu Yao, Chao Qu, Bin Li, Minghao Yang, Dakuan Lu, Haozhe Wang, Xihe Qiu, Wei Chu, Yinghui Xu, and 1 others. 2025. Aurora: Automated training framework of universal process reward models via ensemble prompting and reverse verification. arXiv preprint arXiv:2502.11520. — [arXiv:2502.11520](https://arxiv.org/abs/2502.11520) find
  63. Haoqin Tu, Weitao Feng, Hardy Chen, Hui Liu, Xianfeng Tang, and Cihang Xie. 2025. Vilbench: A suite for vision-language process reward modeling. arXiv preprint arXiv:2503.20271.
  64. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275.
  65. Chenglong Wang, Yongyu Mu, Hang Zhou, Yifu Huo, Ziming Zhu, Jiali Zeng, Murun Yang, Bei Li, Tong Xiao, Xiaoyang Hao, Chunliang Zhang, Fandong Meng, and Jingbo Zhu. 2025a. Gram-r 2 : Self-training generative foundation reward models for reward reasoning. Preprint, arXiv:2509.02492. — [arXiv:2509.02492](https://arxiv.org/abs/2509.02492) find
  66. Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, and 1 others. 2024. Openr: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.09671. find
  67. Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2023. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935.
  68. Shuai Wang, Zhenhua Liu, Jiaheng Wei, Xuanwu Yin, Dong Li, and Emad Barsoum. 2025b. Athena: Enhancing multimodal reasoning with data-efficient process reward models. arXiv preprint arXiv:2506.09532. — [arXiv:2506.09532](https://arxiv.org/abs/2506.09532) find
  69. Teng Wang, Zhangyi Jiang, Zhenqi He, Shenyang Tong, Wenhan Yang, Yanan Zheng, Zeyu Li, Zifan He, and Hailei Gong. 2025c. Towards hierarchical multi-step reward models for enhanced reasoning in large language models. arXiv preprint arXiv:2503.13551.
  70. Teng Wang, Zhangyi Jiang, Zhenqi He, Shenyang Tong, Wenhan Yang, Yanan Zheng, Zeyu Li, Zifan He, and Hailei Gong. 2025d. Towards hierarchical multi-step reward models for enhanced reasoning in large language models. Preprint, arXiv:2503.13551.
  71. Weixuan Wang, Minghao Wu, Barry Haddow, and Alexandra Birch. 2025e. Demystifying multilingual chain-of-thought in process reward modeling. arXiv preprint arXiv:2502.12663. — [arXiv:2502.12663](https://arxiv.org/abs/2502.12663) find
  72. Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, and 1 others. 2025f. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503.10291. — [arXiv:2503.10291](https://arxiv.org/abs/2503.10291) find
  73. Zhao Wang, Ziliang Zhao, and Zhicheng Dou. 2026. Prorag: Process-supervised reinforcement learning for retrieval-augmented generation. arXiv preprint arXiv:2601.21912. — [arXiv:2601.21912](https://arxiv.org/abs/2601.21912) find
  74. Zixiao Wang, Yuxin Wang, Xiaorui Wang, Mengting Xing, Jie Gao, Jianjun Xu, Guangcan Liu, Chenhui Jin, Zhuo Wang, Shengzhuo Zhang, and 1 others. 2025g. Test-time scaling with reflective generative model. arXiv preprint arXiv:2507.01951. — [arXiv:2507.01951](https://arxiv.org/abs/2507.01951) find
  75. Yunjia Xi, Jianghao Lin, Yongzhao Xiao, Zheli Zhou, Rong Shan, Te Gao, Jiachen Zhu, Weiwen Liu, Yong Yu, and Weinan Zhang. 2025. A survey of llm-based deep search agents: Paradigm, optimization, evaluation, and challenges. arXiv preprint arXiv:2508.05668. — [arXiv:2508.05668](https://arxiv.org/abs/2508.05668) find
  76. Bin Xie, Bingbing Xu, Yige Yuan, Shengmao Zhu, and Huawei Shen. 2025a. From outcomes to processes: Guiding PRM learning from ORM for inference-time alignment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 19291–19307, Vienna… find
  77. Bin Xie, Bingbing Xu, Yige Yuan, Shengmao Zhu, and Huawei Shen. 2025b. From outcomes to processes: Guiding prm learning from orm for inference-time alignment. arXiv preprint arXiv:2506.12446. find
  78. Guofu Xie, Yunsheng Shi, Hongtao Tian, Ting Yao, and Xiao Zhang. 2025c. Capo: Towards enhancing llm reasoning through verifiable generative credit assignment. arXiv preprint arXiv:2508.02298. — [arXiv:2508.02298](https://arxiv.org/abs/2508.02298) find
  79. Zhaopan Xu, Pengfei Zhou, Jiaxin Ai, Wangbo Zhao, Kai Wang, Xiaojiang Peng, Wenqi Shao, Hongxun Yao, and Kaipeng Zhang. 2025. Mpbench: A comprehensive multimodal reasoning benchmark for process errors identification. arXiv preprint arXiv:2503.12505. find
  80. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  81. Minglai Yang, Ethan Huang, Liang Zhang, Mihai Surdeanu, William Wang, and Liangming Pan. 2025b. How is llm reasoning distracted by irrelevant context? an analysis using a controlled benchmark. arXiv preprint arXiv:2505.18761.
  82. Yingxuan Yang, Huacan Chai, Yuanyi Song, Siyuan Qi, Muning Wen, Ning Li, Junwei Liao, Haoyi Hu, Jianghao Lin, Gaowei Chang, and 1 others. 2025c. A survey of ai agent protocols. arXiv preprint arXiv:2504.16736. find
  83. Chenlu Ye, Zhou Yu, Ziji Zhang, Hao Chen, Narayanan Sadagopan, Jing Huang, Tong Zhang, and Anurag Beniwal. 2025. Beyond correctness: Harmonizing process and outcome rewards through rl training. arXiv preprint arXiv:2509.03403. — [arXiv:2509.03403](https://arxiv.org/abs/2509.03403) find
  84. Rui Yu, Shenghua Wan, Yucen Wang, Chen-Xiao Gao, Le Gan, Zongzhang Zhang, and De-Chuan Zhan. 2025. Reward models in deep reinforcement learning: A survey. arXiv preprint arXiv:2506.15421. — [arXiv:2506.15421](https://arxiv.org/abs/2506.15421) find
  85. Weichen Yu, Ravi Mangal, Yinyi Luo, Kai Hu, Jingxuan He, Corina S Pasareanu, and Matt Fredrikson. 2026. Seccodeprm: A process reward model for code security. arXiv preprint arXiv:2602.10418. — [arXiv:2602.10418](https://arxiv.org/abs/2602.10418) find
  86. Thomas Zeng, Shuibai Zhang, Shutong Wu, Christian Classen, Daewon Chae, Ethan Ewer, Minjae Lee, Heeju Kim, Wonjun Kang, Jackson Kunde, and 1 others. 2025. Versaprm: Multi-domain process reward model via synthetic reasoning data. arXiv preprint arXiv:2502.06737. — [arXiv:2502.06737](https://arxiv.org/abs/2502.06737) find
  87. Danyang Zhang, Situo Zhang, Ziyue Yang, Zichen Zhu, Zihan Zhao, Ruisheng Cao, Lu Chen, and Kai Yu. 2025a. Progrm: Build better gui agents with progress rewards. arXiv preprint arXiv:2505.18121. — [arXiv:2505.18121](https://arxiv.org/abs/2505.18121) find
  88. Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, and Tong Zhang. 2024. Entropy-regularized process reward model. arXiv preprint arXiv:2412.11006. find
  89. Jianghangfan Zhang, Yibo Yan, Kening Zheng, Xin Zou, Song Dai, and Xuming Hu. 2025b. Gm-prm: A generative multimodal process reward model for multimodal mathematical reasoning. arXiv preprint arXiv:2508.04088.
  90. Kaiyan Zhang, Jiayuan Zhang, Haoxin Li, Xuekai Zhu, Ermo Hua, Xingtai Lv, Ning Ding, Biqing Qi, and Bowen Zhou. 2025c. Openprm: Building open-domain process-based reward models with preference trees. In The Thirteenth International Conference on Learning Representations. find
  91. Kangning Zhang, Wenxiang Jiao, Kounianhua Du, Yuan Lu, Weiwen Liu, Weinan Zhang, and Yong Yu. 2025d. Looptool: Closing the data-training loop for robust llm tool calls. Preprint, arXiv:2511.09148.
  92. Lingyin Zhang, Jun Gao, Xiaoxue Ren, and Ziqiang Cao. 2025e. The bidirectional process reward model. arXiv preprint arXiv:2508.01682.
  93. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2025f. Generative verifiers: Reward modeling as next-token prediction. Preprint, arXiv:2408.15240.
  94. Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. 2025g. A survey on test-time scaling in large language models: What, how, where, and how well? arXiv preprint arXiv:2503.24235. Version v3.
  95. Ruiyi Zhang, Peijia Qin, Qi Cao, Eric Xue, and Pengtao Xie. 2026a. Funprm: Function-as-step process reward model with meta reward correction for code generation. arXiv preprint arXiv:2601.22249. — [arXiv:2601.22249](https://arxiv.org/abs/2601.22249) find
  96. Shimao Zhang, Xiao Liu, Xin Zhang, Junxiao Liu, Zheheng Luo, Shujian Huang, and Yeyun Gong. 2025h. Process-based self-rewarding language models. arXiv preprint arXiv:2503.03746. find
  97. Wenlin Zhang, Xiangyang Li, Kuicai Dong, Yichao Wang, Pengyue Jia, Xiaopeng Li, Yingyi Zhang, Derong Xu, Zhaocheng Du, Huifeng Guo, and 1 others. 2025i. Process vs. outcome reward: Which is better for agentic rag reinforcement learning. arXiv preprint arXiv:2505.14069. — [arXiv:2505.14069](https://arxiv.org/abs/2505.14069) find
  98. Yuanchen Zhou, Shuo Jiang, Jie Zhu, Junhui Li, Lifan Guo, Feng Chen, and Chi Zhang. 2025b. Fin-prm: A domain-specialized process reward model for financial reasoning in large language models. arXiv preprint arXiv:2508.15202. — [arXiv:2508.15202](https://arxiv.org/abs/2508.15202) find
  99. Yao Zhang, Shijie Tang, Zeyu Li, Zhen Han, and Volker Tresp. 2026b. Webarbiter: A principle-guided reasoning process reward model for web agents. arXiv preprint arXiv:2601.21872. — [arXiv:2601.21872](https://arxiv.org/abs/2601.21872) find
  100. Jiachen Zhu, Congmin Zheng, Jianghao Lin, Kounianhua Du, Ying Wen, Yong Yu, Jun Wang, and Weinan Zhang. 2025. Retrieval-augmented process reward model for generalizable mathematical reasoning. arXiv preprint arXiv:2502.14361. find
  101. Yuxin Zhang, Meihao Fan, Ju Fan, Mingyang Yi, Yuyu Luo, Jian Tan, and Guoliang Li. 2025j. Reward-sql: Boosting text-to-sql via stepwise reasoning and process-supervised rewards. arXiv preprint arXiv:2505.04671. — [arXiv:2505.04671](https://arxiv.org/abs/2505.04671) find
  102. A Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025k. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301.
  103. Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, and 1 others. 2025. Genprm: Scaling test-time compute of process reward models via generative reasoning. arXiv preprint arXiv:2504.00891. — [arXiv:2504.00891](https://arxiv.org/abs/2504.00891) find
  104. Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2024. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412.06559.
  105. Congmin Zheng, Xiaoyun Mo, Xinbei Ma, Qiqiang Lin, Yin Zhao, Jiachen Zhu, Xingyu Lou, Jun Wang, Zhaoxiang Wang, Weiwen Liu, and 1 others. 2026. Adaptive milestone reward for gui agents. arXiv preprint arXiv:2602.11524. — [arXiv:2602.11524](https://arxiv.org/abs/2602.11524) find
  106. Congmin Zheng, Jiachen Zhu, Jianghao Lin, Xinyi Dai, Yong Yu, Weinan Zhang, and Mengyue Yang. 2025. Cold: Counterfactually-guided length debiasing for process reward models. arXiv preprint arXiv:2507.15698. — [arXiv:2507.15698](https://arxiv.org/abs/2507.15698) find
  107. Jialun Zhong, Wei Shen, Yanzeng Li, Songyang Gao, Hua Lu, Yicheng Chen, Yang Zhang, Wei Zhou, Jinjie Gu, and Lei Zou. 2025. A comprehensive survey of reward models: Taxonomy, applications, challenges, and future. arXiv preprint arXiv:2504.12328. — [arXiv:2504.12328](https://arxiv.org/abs/2504.12328) find
  108. Chenyu Zhou, Huacan Chai, Wenteng Chen, Zihan Guo, Rong Shan, Yuanyi Song, Tianyi Xu, Yingxuan Yang, Aofan Yu, Weiming Zhang, and 1 others. 2026. Externalization in llm agents: A unified review of memory, skills, protocols and harness engineering. arXiv preprint arXiv:2604.08224. — [arXiv:2604.08224](https://arxiv.org/abs/2604.08224) find
  109. Chenyu Zhou, Tianyi Xu, Jianghao Lin, and Dongdong Ge. 2025a. Steporlm: A self-evolving framework with generative process supervision for operations research language models. arXiv preprint arXiv:2509.22558. — [arXiv:2509.22558](https://arxiv.org/abs/2509.22558) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2026-comprehensive-survey-process-reward-models/