Elamur Research⌕
☰
home / papers / 2026 / Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards

Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards

Paper · 2026 · Raffaele Pisano and Roberto Navigli

Summary

Process Reward Models (PRMs) score each intermediate step of a chain of thought rather than only the final answer. They are limited by their training data. PRM800k depends on costly human labels of uneven reliability. Math-Shepherd uses expensive LLM rollouts that only approximate whether a step is correct. Both are restricted to mathematics and offer coarse labels.

The authors propose generating step-level supervision from classical planning problems written in PDDL. Each plan action is treated as a reasoning step. Partially solved trajectories are verbalized into natural language, and rule-based criteria assign precise, graded rewards to each step. With this pipeline they build a synthetic corpus and add it to PRM800k and Math-Shepherd. They then fine-tune PRMs on Qwen2.5-Math-7B and Llama-3.1-8B backbones.

PRMs trained with the PDDL data consistently beat counterparts trained without it. The gains are especially large on non-mathematical benchmarks such as MR-Ben. The PDDL-trained models are also better at locating the first erroneous step. Because planning problems can be generated cheaply and checked exactly, they offer a scalable source of supervision beyond mathematics.

Key points

  • Corpus scale: about one million reasoning steps from roughly 15,000 planning problems across eleven PDDL domains. Rewards are graded (0.0, 0.25, 0.5, 0.75, 1.0) rather than binary.
  • MR-Ben results: adding PDDL data raised math F1 from 47.5 to 53.6 for Qwen2.5-Math-7B-PRM800k-r. It also improved the Math-Shepherd-based model from 25.8 to 43.2. Scores rose in biology, physics, medicine, chemistry and logic as well.
  • Regression vs. classification: a regression objective on PRM800k (Qwen2.5-Math-7B-PRM800k-r) outperformed the classification-trained Qwen2.5-Math-7B-PRM800k on every benchmark, given the same base model and data.
  • Error types: in qualitative analysis, PDDL-trained PRMs were better at catching modeling errors, such as invalid assumptions or unjustified constraints, not just execution slips.
  • Limitations: intermediate reward levels, especially 0.25, are underrepresented, and the verbalized traces sound less natural than LLM-written CoTs.
  • Release: datasets and trained PRMs are available at github.com/Babelscape/prm-meets-planning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sapienza University of RomeRome · Italy60 works · 10,105 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Raffaele Pisano and Roberto Navigli. 2026. Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1292

@inproceedings{pisano2026process,
  title = {{Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards}},
  author = {Raffaele Pisano and Roberto Navigli},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1292},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1292}
}

References

45 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ella M. Atkins, Edmund H. Durfee, and Kang G. Shin. 1997. Detecting and reacting to unplanned-for world states. In Proceedings of the Fourteenth National Conference on Artificial Intelligence and Ninth Innovative Applications of Artificial Intelligence Conference, AAAI 97, IAAI 97, July 27-31, 1997,… find
  2. Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. 2025. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. CoRR, abs/2503.09567.
  3. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, and 4 others. 2025. Process reinforcement through implicit rewards. CoRR, abs/2502.…
  4. Rina Dechter and Judea Pearl. 1985. Generalized best-first search strategies and the optimality of a*. J. ACM, 32(3):505–536. find
  5. Maria Fox and Derek Long. 2002. The third international planning competition: Temporal and metric planning. In Proceedings of the Sixth International Conference on Artificial Intelligence Planning Systems, April 23-27, 2002, Toulouse, France, page 333. AAAI. find
  6. Malik Ghallab, Dana S. Nau, and Paolo Traverso. 2004. Automated planning - theory and practice. Elsevier. find
  7. Kai Goebel and Patrik Zips. 2025. Can llm-reasoning models replace classical planning? A benchmark study. CoRR, abs/2507.23589. — [arXiv:2507.23589](https://arxiv.org/abs/2507.23589) find
  8. Google DeepMind. 2025. Gemini 3. deepmind.google/models/gemini/. https:// find
  9. Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. 2023. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processin…
  10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. Deepseek-r1 incentivizes reasoning in llms through rein… find
  11. Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. 2025. Skywork open reasoner 1 technical report. CoRR, abs/2505.22312. find
  12. Malte Helmert. 2006. The fast downward planning system. J. Artif. Intell. Res., 26:191–246. find
  13. Malte Helmert and Carmel Domshlak. 2009. Landmarks, critical paths and abstractions: What’s the difference anyway? In Proceedings of the 19th International Conference on Automated Planning and Scheduling, ICAPS 2009, Thessaloniki, Greece, September 19-23, 2009. AAAI. find
  14. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  15. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open-Rev…
  16. Llama Team. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.
  17. Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. 2024. Improve mathematical reasoning in language models by automated process supervision. CoRR, abs/2406.06592.
  18. Meta AI. 2024. Llama 3.1 8B Instruct. https://huggingface.co/meta-llama/Llama-3. 1-8B-Instruct. ↗
  19. Andrea Micheli, Arthur Bit-Monnot, Gabriele Röger, Enrico Scala, Alessandro Valentini, Luca Framba, Alberto Rovetta, Alessandro Trapasso, Luigi Bonassi, Alfonso Emilio Gerevini, Luca Iocchi, Félix Ingrand, Uwe Köckemann, Fabio Patrizi, Alessandro Saetti, Ivan Serina, and Sebastian Stock. 2025. Unifi… find
  20. Mistral AI. 2025. Magistral. CoRR, abs/2506.10910. find
  21. Francesco Maria Molfese, Luca Moroni, Ciro Porcaro, Simone Conia, and Roberto Navigli. 2026. Re-traceqa: Evaluating reasoning traces of small language models in commonsense question answering. In Association for Computational Linguistics: ACL 2026, San Diego, California. Association for Computationa… find
  22. OpenAI. 2024. Openai o1 system card. abs/2412.16720.
  23. CoRR, James T. Oswald, Kavitha Srinivas, Harsha Kokel, Junkyu Lee, Michael Katz, and Shirin Sohrabi. 2024. Large language models as planning domain generators (student abstract). In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Application… find
  24. Qwen Team. 2025a. Qwen2.5 Math 7B Instruct. https://huggingface.co/Qwen/Qwen2. 5-Math-7B-Instruct. ↗
  25. Qwen Team. 2025b. Qwen3 Technical Report. CoRR, abs/2505.09388.
  26. Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. 2025. Prmbench: A fine-grained and challenging benchmark for process-level reward models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, J…
  27. Kaya Stechly, Karthik Valmeekam, Atharva Gundawar, Vardhan Palod, and Subbarao Kambhampati. 2025a. Beyond semantics: The unreasonable effectiveness of reasonless intermediate tokens. CoRR, abs/2505.13775. — [arXiv:2505.13775](https://arxiv.org/abs/2505.13775) find
  28. Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2024. Chain of thoughtlessness? an analysis of cot in planning. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15,… find
  29. Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2025b. On the self-verification limitations of large language models on reasoning and planning tasks. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  30. Wei Sun, Qianlong Du, Fuwei Cui, and Jiajun Zhang. 2025. An efficient and precise training data construction framework for process-supervised reward model in mathematical reasoning. CoRR, abs/2503.02382.
  31. Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. 2024. Easy-to-hard generalization: Scalable alignment beyond human supervision. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, Neur… find
  32. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, Ne…
  33. Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process- and outcome-based feedback. CoRR, abs/2211.14275.
  34. Karthik Valmeekam, Matthew Marquez, Alberto Olmo Hernandez, Sarath Sreedharan, and Subbarao Kambhampati. 2023a. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Advances in Neural Information Processing Systems 36: Annual Conference o…
  35. Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. 2023b. On the planning abilities of large language models - A critical investigation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 202…
  36. Pulkit Verma, Ngoc La, Anthony Favier, Swaroop Mishra, and Julie A. Shah. 2025. Teaching llms to plan: Logical chain-of-thought instruction tuning for symbolic planning. CoRR, abs/2509.13351. — [arXiv:2509.13351](https://arxiv.org/abs/2509.13351) find
  37. Kaustubh Vyas, Damien Graux, Sébastien Montella, Pavlos Vougiouklis, Ruofei Lai, Keshuang Li, Yang Ren, and Jeff Z. Pan. 2025. An extensive evaluation of PDDL capabilities in off-the-shelf llms. CoRR, abs/2502.20175. — [arXiv:2502.20175](https://arxiv.org/abs/2502.20175) find
  38. Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M. Ni, Linyi Yang, Ying Wen, and Weinan Zhang. 2024a. Openr: An open source framework for advanced reasoning with large language models. CoRR, abs/2410.09671. find
  39. Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024b. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape…
  40. Zhongshen Zeng, Yinhong Liu, Yingjia Wan, Jingyao Li, Pengguang Chen, Jianbo Dai, Yuxuan Yao, Rongwu Xu, Zehan Qi, Wanru Zhao, Linling Shen, Jianqiao Lu, Haochen Tan, Yukang Chen, Hao Zhang, Zhan Shi, Bailin Wang, Zhijiang Guo, and Jiaya Jia. 2024. Mr-ben: A meta-reasoning benchmark for evaluating s… find
  41. Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The lessons of developing process reward models in mathematical reasoning. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27…
  42. Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. Processbench: Identifying process errors in mathematical reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
  43. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. C…
  44. Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. 2025. Free process rewards without process labels. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, Proceedings of Machine Lea…
  45. Thomas Zeng, Shuibai Zhang, Shutong Wu, Christian Classen, Daewon Chae, Ethan Ewer, Minjae Lee, Heeju Kim, Wonjun Kang, Jackson Kunde, Ying Fan, Jungtaek Kim, Hyung Il Koo, Kannan Ramchandran, Dimitris Papailiopoulos, and Kangwook Lee. 2025. Versaprm: Multi-domain process reward model via synthetic… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pisano-2026-process-reward-models-meet-planning/