Elamur Research⌕
☰
home / papers / 2026 / Policy-Guided Stepwise Action Planning for Controllable LLM Reasoning

Policy-Guided Stepwise Action Planning for Controllable LLM Reasoning

Paper · 2026 · Jianpeng Zhou, Qisheng Hu, Jiahai Wang et al.

Summary

LLM reasoning can be steered by explicit high-level actions such as analysis, decomposition, knowledge recall or verification. Existing ways of choosing these actions often do no better than plain Chain-of-Thought. The authors trace this to two failure modes. Prompt-only planners loop or skip steps, while retrieval-based and supervised planners collapse onto one or two fixed templates. PG-HAP addresses both with a small, separate planner that picks one action at a time, based on the reasoning produced so far. A frozen executor LLM then writes the content for each step. The planner is trained with reinforcement learning on final-answer correctness, using two regularizers:

  • a logit mask that blocks invalid or redundant action transitions;
  • a batch-level reward that penalizes overuse of identical action sequences.

On mathematical and commonsense benchmarks, PG-HAP beats training-free and instance-level planning baselines across several executor sizes. Its trajectories are less repetitive and adapt to the task. The learned policy also transfers to unseen domains without retraining. This matters because better reasoning comes from training only a lightweight, interpretable planner, with no costly tuning of the large model.

Key points

  • The planner pairs a frozen Qwen2.5-1.5B backbone with a trainable head, and the executor LLM is never updated. Code is released at github.com/john1226966735/PG-HAP.
  • The baselines are CoM (prompt-based), HiAR-ICL (template retrieval) and DOTS (supervised full-plan prediction). On GSM8K, HiAR-ICL and DOTS send about 75–78% of problems through their single most common action sequence.
  • PG-HAP's strategies differ by task:
    • On GSM8K, Analysis → Continue Reasoning → Final Answer covers 59.1% of trajectories, and 9.6% include a Verification step.
    • On StrategyQA, direct reasoning covers 53.1%, and knowledge recall followed by reasoning covers 38.4%.
  • A failure analysis on MATH with a 7B executor found 117 errors, 76.1% of them on Level 4–5 problems. PG-HAP still solved 43 problems that CoT missed, mostly in Intermediate Algebra.
  • The authors list two limitations: performance is capped by what the frozen executor can do, and the action set is fixed and coarse. Cross-domain transfer from CSQA to GPQA suggests the actions are reusable across domains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations🎓Sun Yat-sen UniversityGuangzhou · China72 works · 9,502 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 works

Cite

Cite as Jianpeng Zhou, Qisheng Hu, Jiahai Wang and Wenya Wang. 2026. Policy-Guided Stepwise Action Planning for Controllable LLM Reasoning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2024

@inproceedings{zhou2026policy,
  title = {{Policy-Guided Stepwise Action Planning for Controllable LLM Reasoning}},
  author = {Jianpeng Zhou, Qisheng Hu, Jiahai Wang and Wenya Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2024},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2024}
}

References

25 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. find
  2. Trans. Mach. Learn. Res., 2023. find
  3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  4. Zhihao Dou, Yueming Feng, Jia Liu, and Zhigang Tan. 2025. Plan then action: High-level planning guidance reinforcement learning for llm reasoning. arXiv preprint arXiv:2510.01833. — [arXiv:2510.01833](https://arxiv.org/abs/2510.01833) find
  5. Brown Ebouky, Andrea Bartezzaghi, and Mattia Rigotti. 2025. Eliciting reasoning in language models with cognitive tools. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  6. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. PAL: program-aided language models. In International Conference on Machine Learning, ICML 2023, volume 202 of Proceedings of Machine Learning Research, pages 10764–10799. PMLR. find
  7. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. find
  8. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  9. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  10. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
  11. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
  12. Cong Liu, Jie Wu, Weigang Wu, Xu Chen, Liang Lin, and Wei-Shi Zheng. 2025. Chain of methodologies: Scaling test time computation without training. In Findings of the Association for Computational Linguistics: ACL 2025, pages 5298–5312, Vienna, Austria. Association for Computational Linguistics. find
  13. Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. 2025. S 2 R: Teaching LLMs to self-verify and self-correct via reinforcement learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa…
  14. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… find
  15. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094.
  16. Zhenting Qi, Mingyuan MA, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. 2025. Mutual reasoning makes smaller LLMs stronger problem-solver. In The Thirteenth International Conference on Learning Representations. dynamically in LLMs via optimal reasoning trajectories search. In The Thirteenth Int… find
  17. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representation…
  18. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022.
  19. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  20. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  21. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… find
  22. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
  23. Jinyang Wu, Mingkuan Feng, Shuai Zhang, Feihu Che, Zengqi Wen, and Jianhua Tao. 2024. Beyond examples: High-level automated reasoning paradigm in in-context learning via mcts. arXiv preprint arXiv:2411.18478. — [arXiv:2411.18478](https://arxiv.org/abs/2411.18478) find
  24. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, YuYue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, and 17 others. 2025. DAPO: An open-source LLM reinforcement learning sy…
  25. Murong Yue, Wenlin Yao, Haitao Mi, Dian Yu, Ziyu Yao, and Dong Yu. 2025. DOTS: Learning to reason *Table 6: Statistics of the datasets used in experiments. The training and validation sets are sampled from the original training splits, while the test sets remain consistent with the official benchmar… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-policy-guided-stepwise-action-planning/