Elamur Research⌕
☰
home / papers / 2026 / Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning

Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning

Paper · 2026 · Massimiliano Pronesti, Anya Belz and Yufang Hou

Summary

Reinforcement learning with verifiable rewards (RLVR) has improved LLMs on code and mathematics, but it only checks the final answer. Existing process-supervision methods do score intermediate steps, but they rely on neural judges that are opaque, biased and open to reward hacking. The authors propose Verifiable Process Reward Models (VPRMs), an RL framework in which deterministic, rule-based verifiers derived from task guidelines check every reasoning step and its label. This gives dense step-level feedback on top of outcome verification. They also prove that, under mild reward-separation assumptions, the resulting gradient updates favour correct trajectories and penalise inconsistent ones. The testbed is risk-of-bias (RoB) assessment in medical systematic reviews. There, the Cochrane RoB tool fixes the order of assessment questions and uses decision trees that map step answers to a risk level. Models trained with VPRMs follow domain rules more faithfully and link step decisions to final labels more coherently. They beat state-of-the-art models by up to 20% F1 and outcome-only verifiable rewards by 6.5%. The approach makes the reasoning process auditable, not just the final answer.

Key points

  • Each step receives two bounded scores in [0, 1]: one for choosing the correct step identifier and one for the correct step label. These are combined with format and accuracy rewards.
  • Qwen models were trained with both GRPO and DAPO. Baselines were outcome-only RLVR, neural process-reward models and CoT-prompted pretrained LLMs. Coherence results are reported on CochraneForest.
  • Out-of-distribution tests on CRiskEval (AI safety risks) and the Gretel financial risk dataset showed VPRM models outperforming the original instruct model as well as a version of it trained with RL on outcome rewards alone.
  • During training, the format reward levels off early. Process and accuracy rewards rise slowly and in step with each other, suggesting better intermediate reasoning drives final correctness.
  • Limitations:
    • The method needs explicit deterministic rules.
    • Evaluation is mostly limited to RoB.
    • Output that does not match the verifier's expected format can weaken the reward, especially for smaller models.
    • The authors describe this as the first RL-based approach to RoB assessment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States139 works · 70,691 citations🎓Dublin City UniversityDublin · Ireland70 works · 9,447 citations🎓Interdisciplinary Transformation University AustriaLinz · Austria5 works · 35 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Massimiliano Pronesti, Anya Belz and Yufang Hou. 2026. Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1611

@inproceedings{pronesti2026beyond,
  title = {{Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning}},
  author = {Massimiliano Pronesti, Anya Belz and Yufang Hou},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1611},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1611}
}

References

41 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565.
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  3. Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J. Page, and Vivian A. Welch. 2019. Cochrane handbook for systematic reviews of interventions. Hoboken: Wiley, 4. find
  4. Abel Corrêa Dias, Viviane Pereira Moreira, and João Luiz Dihl Comba. 2025. RoBIn: A transformer-based model for risk of bias inference with machine reading comprehension. Journal of Biomedical Informatics, 166:104819. find
  5. IBM Granite Team. 2024. Granite 3.0 language models. URL: https://github.com/ibm-granite/granite-3.0-language-models. ↗
  6. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  7. GretelAI. 2024. Synthetic financial risk analysis dataset. https://huggingface.co/gretelai/gretel-financial-risk-analysis-v1. ↗
  8. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 180 others. 2025. Deepseek-R1: Incentivizing reasoning capability in LLMs…
  9. Jiajie Huang, Honghao Lai, Weilong Zhao, Danni Xia, Chunyang Bai, Mingyao Sun, Jianing Liu, Jiayi Liu, Bei Pan, Jinhui Tian, and Long Ge. 2025. Large language model–assisted risk-of-bias assessment in randomized controlled trials using the revised risk-of-bias tool: Usability study. Journal of Medic… find
  10. Hugging Face. 2025. Open R1: A fully open reproduction of DeepSeek-R1. find
  11. Changkai Ji, Bowen Zhao, Zhuoyao Wang, Yingwen Wang, Yuejie Zhang, Ying Cheng, Rui Feng, and Xiaobo Zhang. 2025. RoBGuard: Enhancing LLMs to assess risk of bias in clinical trial documents. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1258–1277. find
  12. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626.
  13. Honghao Lai, Jiayi Liu, Chunyang Bai, Hui Liu, Bei Pan, Xufei Luo, Liangying Hou, Weilong Zhao, Danni Xia, Jinhui Tian, Yaolong Chen, Lu Zhang, Janne Estill, Jie Liu, Xing Liao, Nannan Shi, Xin Sun, Hongcai Shang, Zhaoxiang Bian, and 17 others. 2025. Language models for data extraction and risk of b… find
  14. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Christopher Wilhelm, Luca Soldaini,… — Tulu 3: Pushing Frontiers in Open Language Model Post-Training
  15. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  16. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  17. OpenAI. 2024. Introducing openai o1. openai.com/it-IT/o1/. https:// find
  18. OpenAI. 2025. Introducing gpt-5. https://openai.com/it-IT/gpt-5/. ↗
  19. OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, and 107 others. 2025. gpt-oss-120b & gpt-oss-2…
  20. Massimiliano Pronesti, Joao H Bettencourt-Silva, Paul Flanagan, Alessandra Pascale, Oisín Redmond, Anya Belz, and Yufang Hou. 2025a. Query-driven document-level scientific evidence extraction from biomedical studies. In Proceedings of the 63rd Annual Meeting of the Association for Computational Ling… find
  21. Massimiliano Pronesti, Michela Lorandi, Paul Flanagan, Oisín Redmond, Anya Belz, and Yufang Hou. 2025b. Enhancing study-level inference from clinical trial papers via reinforcement learning-based numeric reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Proces… find
  22. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  23. Ling Shi and Deyi Xiong. 2025. CRiskEval: A Chinese multi-level risk evaluation benchmark dataset for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13638–13659, Vienna, Austria. Association for Computa… find
  24. Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward gaming. In Advances in Neural Information Processing Systems, volume 35, pages 9460– 9471. find
  25. Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. 2025. PRMBench: A fine-grained and challenging benchmark for process-level reward models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25299–25346, Vienna, A… find
  26. Jonathan AC Sterne, Jelena Savović, Matthew J. Page, Roy G. Elbers, Natalie S. Blencowe, Isabelle Boutron, Christopher J. Cates, He Cheng, Mark S. Corbett, Sandra M. Eldridge, Miguel A. Hernán, Sally Hopewell, Asbjørn Hróbjartsson, Diana R. Junqueira, Peter Jüni, Jamie J. Kirkham, Toby Lasserson, T… find
  27. Simon Šuster, Timothy Baldwin, and Karin Verspoor. 2024. Zero-and few-shot prompting of generative large language models provides weak assessment of risk of bias in clinical trials. Research Synthesis Methods, 15(6):988–1000. find
  28. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12. MIT Press. find
  29. Jianyou Wang, Weili Cao, Longtian Bao, Youze Zheng, Gil Pasternak, Kaicheng Wang, Xiaoyue Wang, Ramamohan Paturi, and Leon Bergen. 2025a. Measuring risk of bias in biomedical reports: The RoBBR benchmark. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, page… find
  30. Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and yelong shen. 2025b. Reinforcement learning for reasoning in large language models with one training example. In The Thirty-nint… find
  31. Zhen Wang, Zhifeng Gao, and Guolin Ke. 2025c. Masked-and-reordered self-supervision for reinforcement learning from verifiable rewards. arXiv preprint arXiv:2511.17473. — [arXiv:2511.17473](https://arxiv.org/abs/2511.17473) find
  32. Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. 2026. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLMs. In The Fourteenth International Conference on L… find
  33. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388.
  34. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025b. Qwen2.5 technical report. Preprint, arXiv:24… find
  35. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, YuYue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, and 17 others. 2025. DAPO: An open-source LLM reinforcement learning sy…
  36. Jaehoon Yun, Jiwoong Sohn, Jungwoo Park, Hyunjae Kim, Xiangru Tang, Daniel Shao, Yong Hoe Koo, Ko Minhyeok, Qingyu Chen, Mark Gerstein, Michael Moor, and Jaewoo Kang. 2025. Med-PRM: Medical reasoning models with stepwise, guideline-verified process rewards. In Proceedings of the 2025 Conference on E… find
  37. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. STaR: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems.
  38. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun MA, and Junxian He. 2025. SimpleRL-Zoo: Investigating and taming zero reinforcement learning for open base models in the wild. In Second Conference on Language Modeling.
  39. Jixiao Zhang and Chunsheng Zuo. 2025. GRPO-LEAD: A difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5642–5665, Suzhou, China. Association for Comput…
  40. Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The lessons of developing process reward models in mathematical reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10495–10516, Vienna…
  41. Jiaru Zou, Ling Yang, Jingwen Gu, Jiahao Qiu, Ke Shen, Jingrui He, and Mengdi Wang. 2025. ReasonFlux-PRM: Trajectory-aware PRMs for long chain-of-thought reasoning in LLMs. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pronesti-2026-beyond-outcome-verification-verifiable-process/