Elamur Research⌕
☰
home / papers / 2025 / PlanGenLLMs: A Modern Survey of LLM Planning Capabilities

PlanGenLLMs: A Modern Survey of LLM Planning Capabilities

Paper · 2025 · Hui Wei, Zihao Zhang, Shenghua He et al.

Summary

Planners built on large language models produce action sequences that move from a starting world state to a target one. They have spread across many domains, including web navigation, itinerary building and database querying. However, most systems are designed for one narrow problem and judged with inconsistent yardsticks, which makes them hard to compare or to choose between for a new task. This survey, by authors from UC Merced, Emory and PAII Inc., organizes the field by updating the evaluation framework that Kartam and Wilkins (1990) proposed for early AI planners.

It first explains four foundational paradigms: task decomposition, LLMs paired with classical planners, search algorithms, and fine-tuning. It then weighs the strengths and weaknesses of representative systems along six dimensions: whether their plans are complete and executable, how close to optimal they are, how they represent problems, how well they generalize, and how efficient they are. The paper also catalogues benchmarks, evaluation methods and metrics. It closes with open problems such as hallucination, preference alignment, cost and multi-agent settings. The result is a structured entry point for anyone building agentic workflows.

Key points

  • The taxonomy groups benchmarks into two kinds. Planning-focused sets include BlocksWorld, PlanBench, ALFWorld, VirtualHome, TravelPlanner and Natural Plan. Downstream sets include GSM8K, ToolBench, Spider, WEBARENA and Mind2Web.
  • Evaluation approaches covered are automatic verifiers (VAL, BUTLER), ground-truth comparison, human judgment and LLM-as-a-Judge. Metrics include Success Rate, Optimality Rate, Executability Rate, Constraint Pass Rate, token counts and the number of LLM/world-model calls.
  • In hybrid systems such as LLM+P and LLM-DP, the LLM translates the problem into PDDL, which Fast Downward or BFS(f) then solves. Search-based methods include Tree of Thought, RAP and LATS, which use BFS/DFS, MCTS or greedy best-first search.
  • LLM planners frequently produce paths longer than necessary, which the authors link to length bias in LLMs. They suggest RLHF or DPO as possible fixes.
  • The scope is limited to single-agent domains (robotics, household tasks, computer tasks), and security and personalization are left out. A curated paper list is maintained at the GitHub repo Awesome-LLM-Planning-Capability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Emory UniversityAtlanta · United States36 works · 39,779 citations🎓University of California, MercedMerced · United States11 works · 425 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Hui Wei, Zihao Zhang, Shenghua He, Tian Xia, Shijia Pan and Fei Liu. 2025. PlanGenLLMs: A Modern Survey of LLM Planning Capabilities. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.958

@inproceedings{wei2025plangenllms,
  title = {{PlanGenLLMs: A Modern Survey of LLM Planning Capabilities}},
  author = {Hui Wei, Zihao Zhang, Shenghua He, Tian Xia, Shijia Pan and Fei Liu},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.958},
  url = {https://doi.org/10.18653/v1/2025.acl-long.958}
}

References

139 references, 38 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Mohamed Aghzal, Erion Plaku, and Ziyu Yao. 2023. Can large language models be good path planners? a benchmark and investigation on spatial-temporal reasoning. arXiv preprint arXiv:2310.03249.
  3. Mohamed Aghzal, Erion Plaku, and Ziyu Yao. 2024. Look further ahead: Testing the limits of gpt-4 in path planning. arXiv preprint arXiv:2406.12000. — [arXiv:2406.12000](https://arxiv.org/abs/2406.12000) find
  4. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. 2022. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691.
  5. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. find
  6. Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D C, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B. Ashok, and Shashank Shet. 2023. Codeplan: Repository-level coding using llms and planning. Preprint, arXiv:2309.12499. — [arXiv:2309.12499](https://arxiv.org/abs/2309.12499) find
  7. Ksenia Bestuzheva, Mathieu Besançon, Wei-Kun Chen, Antonia Chmiela, Tim Donkiewicz, Jasper van Doornmalen, Leon Eifler, Oliver Gaul, Gerald Gam-rath, Ambros Gleixner, et al. 2021. The scip optimization suite 8.0. arXiv preprint arXiv:2112.08872. — [arXiv:2112.08872](https://arxiv.org/abs/2112.08872) find
  8. Vineet Bhat, Ali Umut Kaypak, Prashanth Krishnamurthy, Ramesh Karri, and Farshad Khorrami. 2024. Grounding llms for robot task planning using closed-loop state feedback. arXiv preprint arXiv:2402.08546. — [arXiv:2402.08546](https://arxiv.org/abs/2402.08546) find
  9. Stephen Boyd and Lieven Vandenberghe. 2004. Convex Optimization. Cambridge University Press. find
  10. Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D Hwang, Xiang Lorraine Li, Hirona Jacqueline Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Yejin Choi. 2024. Plasma: Procedural knowledge models for language-based planning and re-planning. In The Twelfth International Conference… find
  11. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  12. Daniel Cao, Michael Katz, Harsha Kokel, Kavitha Srinivas, and Shirin Sohrabi. 2024. Automating thought of search: A journey towards soundness and completeness. arXiv preprint arXiv:2408.11326. — [arXiv:2408.11326](https://arxiv.org/abs/2408.11326) find
  13. Georgia Chalvatzaki, Ali Younes, Daljeet Nandha, An Thai Le, Leonardo FR Ribeiro, and Iryna Gurevych. 2023. Learning to reason over scene graphs: a case study of finetuning gpt-2 into a robot language model for grounded task planning. Frontiers in Robotics and AI, 10:1221739. find
  14. Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, et al. 2024. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. arXiv preprint arXiv:2411.00081. — [arXiv:2411.00081](https://arxiv.org/abs/2411.00081) find
  15. Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023a. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915.
  16. Jiangjie Chen, Siyu Yuan, Rong Ye, Bodhisattwa Prasad Majumder, and Kyle Richardson. 2023b. Put your money where your mouth is: Evaluating strategic planning and execution of llm agents in an auction arena. arXiv preprint arXiv:2310.05746. — [arXiv:2310.05746](https://arxiv.org/abs/2310.05746) find
  17. Yongchao Chen, Jacob Arkin, Charles Dawson, Yang Zhang, Nicholas Roy, and Chuchu Fan. 2024a. Autotamp: Autoregressive task and motion planning with llms as translators and checkers. In 2024 IEEE International conference on robotics and automation (ICRA), pages 6695–6702. IEEE. find
  18. Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. 2024b. Agent-flan: Designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403.12881.
  19. Zhenfang Chen, Delin Chen, Wenjun Sun, Rui abd Liu, and Chuang Gan. 2025. Autonomous agents from automatic reward modeling and planning. In The Thirteenth International Conference on Learning Representations. find
  20. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  21. Hongwei Cui, Yuyang Du, Qun Yang, Yulin Shao, and Soung Chang Liew. 2024. Llmind: Orchestrating ai and iot with llm for complex task execution. IEEE Communications Magazine. find
  22. Aidan Curtis, Nishanth Kumar, Jing Cao, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. 2024. Trust the proc3s: Solving long-horizon robotics problems with llms and constraint satisfaction. arXiv preprint arXiv:2406.05572. — [arXiv:2406.05572](https://arxiv.org/abs/2406.05572) find
  23. Gautier Dagan, Frank Keller, and Alex Lascarides. 2023. Dynamic planning with a llm. arXiv preprint arXiv:2308.06391.
  24. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114.
  25. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36.
  26. Alfonso Gerevini, Ivan Serina, et al. 2002. Lpg: A planner based on local search for planning graphs with action costs. In Aips, volume 2, pages 281–290. find
  27. Gonzalo Gonzalez-Pumariega, Wayne Chen, Kushal Kedia, and Sanjiban Choudhury. 2024. Query-efficient planning with language models. arXiv preprint arXiv:2412.06162. — [arXiv:2412.06162](https://arxiv.org/abs/2412.06162) find
  28. Gonzalo Gonzalez-Pumariega, Leong Su Yean, Neha Sunkara, and Sanjiban Choudhury. 2025. Robotouille: An asynchronous planning benchmark for llm agents. arXiv preprint arXiv:2502.05227. — [arXiv:2502.05227](https://arxiv.org/abs/2502.05227) find
  29. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. find
  30. Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. 2023. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. Advances in Neural Information Processing Systems, 36:79081–79094. find
  31. Shiguang Guo, Ziliang Deng, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. 2024. Open grounded planning: Challenges and benchmark construction. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-… find
  32. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992. find
  33. Yilun Hao, Yongchao Chen, Yang Zhang, and Chuchu Fan. 2024a. Large language models can plan your travels rigorously with formal verification tools. arXiv preprint arXiv:2404.11891. — [arXiv:2404.11891](https://arxiv.org/abs/2404.11891) find
  34. Yilun Hao, Yang Zhang, and Chuchu Fan. 2024b. Planning anything with rigor: General-purpose zero-shot planning with llm-based formalized programming. arXiv preprint arXiv:2410.12112. — [arXiv:2410.12112](https://arxiv.org/abs/2410.12112) find
  35. Rishi Hazra, Pedro Zuidberg Dos Martires, and Luc De Raedt. 2024. Saycanpay: Heuristic planning with large language models using learnable domain knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20123–20133. find
  36. Malte Helmert. 2006. The fast downward planning system. Journal of Artificial Intelligence Research, 26:191–246. find
  37. Eran Hirsch, Guy Uziel, and Ateret Anaby-Tavor. 2024. What’s the plan? evaluating and developing planning-aware techniques for llms. arXiv preprint arXiv:2402.11489. — [arXiv:2402.11489](https://arxiv.org/abs/2402.11489) find
  38. Richard Howey, Derek Long, and Maria Fox. 2004. Val: Automatic plan validation, continuous effects and mixed initiative planning using pddl. In 16th IEEE International Conference on Tools with Artificial Intelligence, pages 294–301. IEEE. find
  39. Hanxu Hu, Hongyuan Lu, Huajian Zhang, Yun-Ze Song, Wai Lam, and Yue Zhang. 2023a. Chain-of-symbol prompting elicits planning in large langauge models. arXiv preprint arXiv:2305.10276. — [arXiv:2305.10276](https://arxiv.org/abs/2305.10276) find
  40. Hanxu Hu, Hongyuan Lu, Huajian Zhang, Yun-Ze Song, Wai Lam, and Yue Zhang. 2024. Chain-of-symbol prompting for spatial reasoning in large language models. In First Conference on Language Modeling. find
  41. Mengkang Hu, Yao Mu, Xinmiao Yu, Mingyu Ding, Shiguang Wu, Wenqi Shao, Qiguang Chen, Bin Wang, Yu Qiao, and Ping Luo. 2023b. Tree-planner: Efficient close-loop task planning with large language models. arXiv preprint arXiv:2310.08582. — [arXiv:2310.08582](https://arxiv.org/abs/2310.08582) find
  42. Hsiu-Yuan Huang, Yutong Yang, Zhaoxi Zhang, Sanwoo Lee, and Yunfang Wu. 2024a. A survey of uncertainty estimation in llms: Theory meets practice. arXiv preprint arXiv:2410.15326. — [arXiv:2410.15326](https://arxiv.org/abs/2410.15326) find
  43. Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael R Lyu. 2024b. How far are we on the decision-making of llms? evaluating llms’ gaming ability in multi-agent environments. arXiv preprint arXiv:2403.11807. — [arXiv:2403.11807](https://arxiv.org/abs/2403.11807) find
  44. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232.
  45. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022a. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pages 9118–9147. PMLR.
  46. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2022b. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608. find
  47. Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024c. Understanding the planning of llm agents: A survey. arXiv preprint arXiv:2402.02716. find
  48. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
  49. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024b. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579.
  50. Peter A Jansen. 2020. Visually-grounded planning without vision: Language models infer detailed plans from high-level instructions. arXiv preprint arXiv:2009.14259. find
  51. Haoming Li, Zhaoliang Chen, Songyuan Liu, Yiming Lu, and Fei Liu. 2024c. Systematic analysis of llm contributions to planning: Solver, verifier, heuristic. Preprint, arXiv:2412.09666. — [arXiv:2412.09666](https://arxiv.org/abs/2412.09666) find
  52. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.
  53. Haoming Li, Zhaoliang Chen, Jonathan Zhang, and Fei Liu. 2024d. Lasp: Surveying the state-of-the-art in large language model-assisted ai planning. arXiv preprint arXiv:2409.01806. — [arXiv:2409.01806](https://arxiv.org/abs/2409.01806) find
  54. Da Ju, Song Jiang, Andrew Cohen, Aaron Foss, Sasha Mitts, Arman Zharmagambetov, Brandon Amos, Xian Li, Justine T Kao, Maryam Fazel-Zarandi, et al. 2024. To the globe (ttg): Towards language-driven guaranteed travel planning. arXiv preprint arXiv:2410.16456. find
  55. Haoming Li, Zhaoliang Chen, Jonathan Zhang, and Fei Liu. 2025. Planet: A collection of benchmarks for evaluating llms’ planning capabilities. Preprint, arXiv:2504.14773. — [arXiv:2504.14773](https://arxiv.org/abs/2504.14773) find
  56. Subbarao Kambhampati. 2024. Can large language models reason and plan? Annals of the New York Academy of Sciences, 1534(1):15–18. find
  57. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. 2024e. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36.
  58. Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. 2024. Llms can’t plan, but can help planning in llm-modulo frameworks. arXiv preprint arXiv:2402.01817. find
  59. Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, et al. 2024f. Embodied agent interface: Benchmarking llms for embodied decision making. arXiv preprint arXiv:2410.07166. find
  60. Nabil A Kartam and David E Wilkins. 1990. Towards a foundation for evaluating ai planners. AI EDAM, 4(1):1–13. find
  61. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244.
  62. Michael Katz, Harsha Kokel, Kavitha Srinivas, and Shirin Sohrabi. 2024. Thought of search: Planning with language models through the lens of efficiency. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  63. Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. 2024. Tree search for language model agents. arXiv preprint arXiv:2407.01476. find
  64. Kurt Konolige and Nils J Nilsson. 1980. Multiple-agent planning systems. In AAAI, volume 80, pages 138– 142. find
  65. Lucas Lehnert, Sainbayar Sukhbaatar, DiJia Su, Qinqing Zheng, Paul Mcvay, Michael Rabbat, and Yuandong Tian. 2024. Beyond a*: Better planning with transformers via search dynamics bootstrapping. arXiv preprint arXiv:2402.14083. find
  66. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2024a. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594. find
  67. Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. 2022. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31199–31212. find
  68. Bill Yuchen Lin, Chengsong Huang, Qian Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. 2023a. On grounded planning for embodied tasks with language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13192– 13200. find
  69. Fangru Lin, Emanuele La Malfa, Valentin Hofmann, Elle Michelle Yang, Anthony Cohn, and Janet B Pierrehumbert. 2024. Graph-enhanced large language models in asynchronous plan reasoning. arXiv preprint arXiv:2402.02805. — [arXiv:2402.02805](https://arxiv.org/abs/2402.02805) find
  70. Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. 2023b. Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning. arXiv preprint arXiv:2309.15091. find
  71. Nir Lipovetzky, Miquel Ramirez, Christian Muise, and Hector Geffner. 2014. Width and inference based planners: Siw, bfs (f), and probe. Proceedings of the 8th International Planning Competition (IPC-2014), page 43. find
  72. Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. 2023. Adapt: As-needed decomposition and planning with language models. arXiv preprint arXiv:2311.05772.
  73. Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. 2023a. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477.
  74. Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. 2018. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8494–8502. find
  75. Gang Liu, Michael Sun, Wojciech Matusik, Meng Jiang, and Jie Chen. 2024. Multimodal large language models for inverse molecular design with retrosynthetic planning. arXiv preprint arXiv:2410.04223. — [arXiv:2410.04223](https://arxiv.org/abs/2410.04223) find
  76. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023b. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688.
  77. Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. 2023. Multimodal procedural planning via dual text-image prompting. arXiv preprint arXiv:2305.01795. find
  78. Drew McDermott, Malik Ghallab, Adele E. Howe, Craig A. Knoblock, Ashwin Ram, Manuela M. Veloso, Daniel S. Weld, and David E. Wilkins. 1998. Pddl-the planning domain definition language. find
  79. Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019. Step-by-step: Separating planning from realization in neural data-to-text generation. arXiv preprint arXiv:1904.03396.
  80. Allen Newell, John Calman Shaw, and Herbert A Simon. 1958. Elements of a theory of human problem solving. Psychological review, 65(3):151. find
  81. Odhran O’Donoghue, Aleksandar Shtedritski, John Ginger, Ralph Abboud, Ali Essa Ghareeb, Justin Booth, and Samuel G Rodriques. 2023. Bioplanner: automatic evaluation of llms on protocol planning in biology. arXiv preprint arXiv:2310.10632. find
  82. Jiefu Ou, Arda Uzunoglu, Benjamin Van Durme, and Daniel Khashabi. 2024. Worldapis: The world is worth how many apis? a thought experiment. arXiv preprint arXiv:2407.07778. — [arXiv:2407.07778](https://arxiv.org/abs/2407.07778) find
  83. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  84. Jiayi Pan, Glen Chou, and Dmitry Berenson. 2023. Data-efficient learning of natural language to linear temporal logic translators for robot task specification. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11554–11561. IEEE. find
  85. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  86. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.
  87. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.
  88. Shreyas Sundara Raman, Vanya Cohen, Ifrah Idrees, Eric Rosen, Raymond Mooney, Stefanie Tellex, and David Paulius. 2024. Cape: Corrective actions from precondition errors using large language models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 14070–14077. IEEE. find
  89. Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. 2022. Planning with large language models via corrective re-prompting. In NeurIPS 2022 Foundation Models for Decision Making Workshop. find
  90. Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, and Ebrahim Bagheri. 2025. Benchmarking prompt sensitivity in large language models. Preprint, arXiv:2502.06065. — [arXiv:2502.06065](https://arxiv.org/abs/2502.06065) find
  91. Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. 2023. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint arXiv:2307.01928. find
  92. Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Ziyue Li, Xingyu Zeng, and Rui Zhao. 2023. Tptu: Large language model-based ai agents for task planning and tool usage. Preprint, arXiv:2308.03427. — [arXiv:2308.03427](https://arxiv.org/abs/2308.03427) find
  93. Stuart J Russell and Peter Norvig. 2016. Artificial intelligence: a modern approach. pearson. find
  94. Abulhair Saparov and He He. 2022. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. arXiv preprint arXiv:2210.01240.
  95. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. Preprint, arXiv:2310.11324. find
  96. Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. 2021. Skill induction and planning with latent language. arXiv preprint arXiv:2110.01517. find
  97. Zijing Shi, Meng Fang, and Ling Chen. 2025. Monte carlo planning with large language model for text-based games. In The Thirteenth International Conference on Learning Representations. find
  98. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020a. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pag…
  99. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020b. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768.
  100. Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B Tenenbaum, Leslie Kaelbling, and Michael Katz. 2024. Generalized planning in pddl domains with pretrained large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20256–20264. find
  101. Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2023. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1…
  102. Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998–3009.
  103. Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and error: Exploration-based trajectory optimization for llm agents. arXiv preprint arXiv:2403.02502. — [arXiv:2403.02502](https://arxiv.org/abs/2403.02502) find
  104. Siddharth Srivastava, Neil Immerman, and Shlomo Zilberstein. 2011. A new representation and associated algorithms for generalized planning. Artificial Intelligence, 175(2):615–647. find
  105. Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. 2024. Adaplanner: Adaptive planning from feedback with language models. Advances in Neural Information Processing Systems, 36. find
  106. Marcus Tantakoun, Xiaodan Zhu, and Christian Muise. 2025. Llms as planning modelers: A survey for leveraging large language models to construct automated planning models. Preprint, arXiv:2503.18971. — [arXiv:2503.18971](https://arxiv.org/abs/2503.18971) find
  107. Alejandro Torreno, Eva Onaindia, Antonín Komenda, and Michal Štolba. 2017. Cooperative multi-agent planning: A survey. ACM Computing Surveys (CSUR), 50(6):1–32. find
  108. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  109. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023a. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems, 36:38975–38987.
  110. Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. 2023b. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36:75993–76005. find
  111. Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. 2024. Llms still can’t plan; can lrms? a preliminary evaluation of openai’s o1 on planbench. arXiv preprint arXiv:2409.13373. — [arXiv:2409.13373](https://arxiv.org/abs/2409.13373) find
  112. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023a. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. find
  113. Renxi Wang, Haonan Li, Xudong Han, Yixuan Zhang, and Timothy Baldwin. 2024. Learning from failure: Integrating negative examples when fine-tuning large language models as agents. arXiv preprint arXiv:2402.11651. — [arXiv:2402.11651](https://arxiv.org/abs/2402.11651) find
  114. Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023b. Describe, explain, plan and select: Interactive planning with large language models enables open-worldzhao2024large multi-task agents. arXiv preprint arXiv:2302.01560.
  115. Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, and Mei Han. 2024. Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates. arXiv preprint arXiv:2408.13006. — [arXiv:2408.13006](https://arxiv.org/abs/2408.13006) find
  116. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  117. Xixi Wu, Yifei Shen, Caihua Shan, Kaitao Song, Siwei Wang, Bohang Zhang, Jiarui Feng, Hong Cheng, Wei Chen, Yun Xiong, et al. 2024. Can graph learning improve planning in llm-based agents? In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  118. Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li. 2023a. Smartplay: A benchmark for llms as intelligent agents. arXiv preprint arXiv:2310.01557. find
  119. Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. 2023b. Embodied task planning with large language models. arXiv preprint arXiv:2307.01848. — [arXiv:2307.01848](https://arxiv.org/abs/2307.01848) find
  120. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. 2024. Travelplanner: A benchmark for real-world planning with language agents. arXiv preprint arXiv:2402.01622. find
  121. Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. 2023. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128. — [arXiv:2302.05128](https://arxiv.org/abs/2302.05128) find
  122. Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv:2305.16504.
  123. Ruihan Yang, Jiangjie Chen, Yikai Zhang, Siyu Yuan, Aili Chen, Kyle Richardson, Yanghua Xiao, and Deqing Yang. 2024. Selfgoal: Your language agents already know how to achieve high-level goals. arXiv preprint arXiv:2406.04784. find
  124. Quan Yuan, Mehran Kazemi, Xin Xu, Isaac Noble, Vaiva Imbrasaite, and Deepak Ramachandran. 2024. Tasklama: probing the complex task understanding of language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19468–19476. find
  125. Siyu Yuan, Jiangjie Chen, Ziquan Fu, Xuyang Ge, Soham Shah, Charles Robert Jankowski, Yanghua Xiao, and Deqing Yang. 2023. Distilling script knowledge from large language models for constrained language planning. arXiv preprint arXiv:2305.05252. find
  126. Abhay Zala, Han Lin, Jaemin Cho, and Mohit Bansal. 2023. Diagrammergpt: Generating open-domain, open-platform diagrams via llm planning. arXiv preprint arXiv:2310.12128. find
  127. Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang, Liangwei Yang, Yihao Feng, Zuxin Liu, et al. 2024. Agentohana: Design unified data and training pipeline for effective agent learning. arXiv preprint arXiv:2402.15506. — [arXiv:2402.15506](https://arxiv.org/abs/2402.15506) find
  128. Zirui Zhao, Wee Sun Lee, and David Hsu. 2024. Large language models as commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36. find
  129. Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V Le, Ed H Chi, et al. 2024. Natural plan: Benchmarking llms on natural language planning. arXiv preprint arXiv:2406.04520. find
  130. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
  131. Ryan Yang, Tom Silver, Aidan Curtis, Tomas Lozano-Perez, and Leslie Pack Kaelbling. 2022. Pg3: Policy-guided planning for generalized policy generation. arXiv preprint arXiv:2204.10420. — [arXiv:2204.10420](https://arxiv.org/abs/2204.10420) find
  132. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2023a. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406. find
  133. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36. find
  134. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2023b. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.
  135. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736.
  136. Zhehua Zhou, Jiayang Song, Kunpeng Yao, Zhan Shu, and Lei Ma. 2024. Isr-llm: Iterative self-refined large language model for long-horizon sequential task planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 2081–2088. IEEE. find
  137. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887.
  138. Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A Rossi, Somdeb Sarkhel, and Chao Zhang. 2023. Toolchain*: Efficient action space navigation in large language models with a* search. arXiv preprint arXiv:2310.13227. — [arXiv:2310.13227](https://arxiv.org/abs/2310.13227) find
  139. Max Zuo, Francisco Piedrahita Velez, Xiaochen Li, Michael L Littman, and Stephen H Bach. 2024. Planetarium: A rigorous benchmark for translating text to structured planning languages. arXiv preprint arXiv:2407.03321. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wei-2025-plangenllms-modern-survey-llm-planning/