Elamur Research⌕
☰
home / papers / 2023 / Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning

Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning

Paper · 2023 · Lin Guan, Karthik Valmeekam, Sarath Sreedharan et al.

Summary

The paper argues that LLMs should not act as planners themselves. Instead, they should build the world model that a classical planner then uses. The authors give GPT-4 a list of actions with short natural-language descriptions and have it write PDDL action models, meaning predicates, preconditions and effects. Early drafts contain mistakes, so the LLM also acts as a go-between with feedback sources. It turns PDDL into plain language so that non-expert users can review it, and it applies their remarks, along with error reports from the VAL validator, back into the model.

The corrected models are used in two ways:

  • Classical planning: goals are translated into PDDL and solved with Fast Downward.
  • Plan checking: VAL checks plans proposed by an LLM planner and returns any unmet conditions as feedback.

Tests cover two IPC domains and a Household domain whose constraints are richer than those of ALFWorld. The classical-planner route solved nearly all tasks, while LLM-only planning mostly failed. Users correct the model once at the start instead of reviewing every plan, and the external planner keeps plans sound.

Key points

  • GPT-4 produced PDDL models for 41 actions, totalling more than 400 literals. Natural-language feedback from humans and VAL was enough to repair all remaining errors. The corrected models then solved 48 challenging planning tasks.
  • In the Household and Logistics domains, success rates were as follows:
    • LLM planner alone: 15% and 0%.
    • Fast Downward with the LLM-built PDDL model: 95% and 100%. The few failures came only from goal-translation errors.
    • ReAct-style GPT-4 planner back-prompted with VAL feedback: 48% and 33%, with at most 8 feedback rounds per task.
  • With validator feedback, GPT-4 often got stuck re-proposing the same plan, and its fixes sometimes introduced new errors.
  • LLM plans respected ordering constraints that the existing predicates could not express. They also avoided odd but valid actions, such as putting a knife on a toaster, which hints at implicit knowledge of human preferences.
  • Stated limitations include domains simpler than many classical-planning benchmarks, a fixed set of object types, an assumption of full observability, and an assumption of perfect predicate grounding. Code and resources are available at guansuns.github.io/pages/llm-dm.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksEngineering · 188 worksControl and Systems Engineering · 62 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Lin Guan, Karthik Valmeekam, Sarath Sreedharan and Subbarao Kambhampati. 2023. Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning. arXiv:2305.14909.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.14909 · arXiv 2305.14909

@misc{guan2023leveraging,
  title = {{Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning}},
  author = {Lin Guan, Karthik Valmeekam, Sarath Sreedharan and Subbarao Kambhampati},
  year = {2023},
  doi = {10.48550/arxiv.2305.14909},
  eprint = {2305.14909},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.14909}
}

References

62 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
  2. Ankuj Arora, Humbert Fiorino, Damien Pellier, Marc Métivier, and Sylvie Pesty. A review of learning planning action models. The Knowledge Engineering Review, 33:e20, 2018. find
  3. Pratyay Banerjee, Chitta Baral, Man Luo, Arindam Mitra, Kuntal Pal, Tran C Son, and Neeraj Varshney. Can transformers reason about effects of actions? arXiv preprint arXiv:2012.09938, 2020. — [arXiv:2012.09938](https://arxiv.org/abs/2012.09938) find
  4. Blai Bonet and Hector Geffner. Learning first-order symbolic representations for planning from the structure of the state space. arXiv preprint arXiv:1909.05546, 2019. — [arXiv:1909.05546](https://arxiv.org/abs/1909.05546) find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  6. Ethan Callanan, Rebecca De Venezia, Victoria Armstrong, Alison Paredes, Tathagata Chakraborti, and Christian Muise. Macq: A holistic view of model acquisition techniques. In The ICAPS Workshop on Knowledge Engineering for Planning and Scheduling (KEPS), 2022. find
  7. Hongyi Chen, Yilun Du, Yiye Chen, Joshua B. Tenenbaum, and Patricio A. Vela. Planning with sequence models through iterative energy minimization. In The Eleventh International Conference on Learning Representations, 2023. find
  8. Shuo Cheng and Danfei Xu. Guided skill learning and abstraction for long-horizon manipulation. arXiv preprint arXiv:2210.12631, 2022. — [arXiv:2210.12631](https://arxiv.org/abs/2210.12631) find
  9. Stephen N Cresswell, Thomas L McCluskey, and Margaret M West. Acquiring planning domain models using locm. The Knowledge Engineering Review, 28(2):195–213, 2013. find
  10. Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. find
  11. Richard E Fikes and Nils J Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, 2(3-4):189–208, 1971.
  12. Alfonso Gerevini and Ivan Serina. Lpg: A planner based on local search for planning graphs with action costs. In AIPS, volume 2, pages 281–290, 2002. find
  13. Malik Ghallab, Dana Nau, and Paolo Traverso. Automated Planning: theory and practice. Elsevier, 2004. find
  14. Lin Guan, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging approximate symbolic models for reinforcement learning via skill diversity. In International Conference on Machine Learning, pages 7949–7967. PMLR, 2022. find
  15. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023. find
  16. Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26:191–246, 2006. find
  17. Jörg Hoffmann and Bernhard Nebel. The ff planning system: Fast plan generation through heuristic search. Journal of Artificial Intelligence Research, 14:253–302, 2001. find
  18. Richard Howey, Derek Long, and Maria Fox. Val: Automatic plan validation, continuous effects and mixed initiative planning using pddl. In 16th IEEE International Conference on Tools with Artificial Intelligence, pages 294–301. IEEE, 2004. find
  19. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118–9147. PMLR, 2022.
  20. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. find
  21. León Illanes, Xi Yan, Rodrigo Toro Icarte, and Sheila A McIlraith. Symbolic plans as high-level instructions for reinforcement learning. In Proceedings of the international conference on automated planning and scheduling, volume 30, pages 540–550, 2020. find
  22. IPC. International planning competition, 1998. find
  23. Subbarao Kambhampati, Sarath Sreedharan, Mudit Verma, Yantian Zha, and Lin Guan. Symbols as a lingua franca for bridging human-ai chasm for explainable and advisable ai systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12262–12267, 2022. find
  24. Subbarao Kambhampati, Karthik Valmeekam, Matthew Marquez, and Lin Guan. On the role of large language models in planning, July 2023. Tutorial presented at the International Conference on Automated Planning and Scheduling (ICAPS), Prague. https://yochan-lab.github.io/tutorial/ICAPS-2023/. ↗
  25. George Konidaris, Leslie Pack Kaelbling, and Tomas Lozano-Perez. From skills to symbols: Learning symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61:215–289, 2018. find
  26. Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations, 2023. find
  27. Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31199–31212, 2022.
  28. Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2022. find
  29. Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303.12153, 2023. — [arXiv:2303.12153](https://arxiv.org/abs/2303.12153) find
  30. Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477, 2023.
  31. Man Luo, Shrinidhi Kumbhar, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya, Chitta Baral, et al. Towards logiglue: A brief survey and a benchmark for analyzing logical reasoning capabilities of language models. arXiv preprint arXiv:2310.00836, 2023.
  32. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. Faithful chain-of-thought reasoning. arXiv preprint arXiv:2301.13379, 2023.
  33. Drew McDermott, Malik Ghallab, Adele E. Howe, Craig A. Knoblock, Ashwin Ram, Manuela M. Veloso, Daniel S. Weld, and David E. Wilkins. Pddl-the planning domain definition language. 1998. find
  34. Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, and Roy Fox. Do embodied agents dream of pixelated sheep?: Embodied decision making using language guided world modelling. arXiv preprint arXiv:2301.12050, 2023. — [arXiv:2301.12050](https://arxiv.org/abs/2301.12050) find
  35. Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Gpt3-to-plan: Extracting plans from text using gpt-3. arXiv preprint arXiv:2106.07131, 2021. — [arXiv:2106.07131](https://arxiv.org/abs/2106.07131) find
  36. OpenAI. Introducing chatgpt by openai, 2022. find
  37. OpenAI. Gpt-4 technical report, 2023.
  38. Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Lior Horesh, Biplav Srivastava, Francesco Fabiano, and Andrea Loreggia. Plansformer: Generating symbolic plans using transformers. arXiv preprint arXiv:2212.08681, 2022. — [arXiv:2212.08681](https://arxiv.org/abs/2212.08681) find
  39. Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv:2305.12295, 2023.
  40. Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494–8502, 2018. find
  41. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 20…
  42. Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re-prompting. arXiv preprint arXiv:2211.09935, 2022. — [arXiv:2211.09935](https://arxiv.org/abs/2211.09935) find
  43. Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022. find
  44. Stuart Jonathan Russell. Norvig (2003). Artificial intelligence: a modern approach, 25:26, 2003. find
  45. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
  46. Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.
  47. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  48. Tom Silver, Varun Hariprasad, Reece S Shuttleworth, Nishanth Kumar, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Pddl planning with pretrained large language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022. find
  49. Ron M Simpson, Diane E Kitchin, and Thomas Leo McCluskey. Planning domain definition using gipo. The Knowledge Engineering Review, 22(2):117–134, 2007. find
  50. Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302, 2022.
  51. Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. arXiv preprint arXiv:2212.04088, 2022.
  52. Sarath Sreedharan, Tathagata Chakraborti, Christian Muise, Yasaman Khazaeni, and Subbarao Kambhampati. –d3wa+–a case study of xaip in a model acquisition task for dialogue planning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pages 488–497, 2020. find
  53. Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. Gpt-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems. arXiv preprint arXiv:2310.12397, 2023. — [arXiv:2310.12397](https://arxiv.org/abs/2310.12397) find
  54. Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. Can large language models really improve by self-critiquing their own plans? arXiv preprint arXiv:2310.08118, 2023. — [arXiv:2310.08118](https://arxiv.org/abs/2310.08118) find
  55. Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models–a critical investigation. arXiv preprint arXiv:2305.15771, 2023.
  56. Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560, 2023.
  57. Lionel Wong, Gabriel Grand, Alexander K Lew, Noah D Goodman, Vikash K Mansinghka, Jacob Andreas, and Joshua B Tenenbaum. From word models to world models: Translating from natural language to the probabilistic language of thought. arXiv preprint arXiv:2306.12672, 2023. find
  58. Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128, 2023. — [arXiv:2302.05128](https://arxiv.org/abs/2302.05128) find
  59. Qiang Yang, Kangheng Wu, and Yunfei Jiang. Learning action models from plan examples using weighted max-sat. Artificial Intelligence, 171(2-3):107–143, 2007. find
  60. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
  61. Hankz Hankui Zhuo, Qiang Yang, Derek Hao Hu, and Lei Li. Learning complex action models with quantifiers and logical implications. Artificial Intelligence, 174(18):1540–1569, 2010. find
  62. Hankz Hankui Zhuo, Yantian Zha, Subbarao Kambhampati, and Xin Tian. Discovering underlying plans based on shallow models. ACM Transactions on Intelligent Systems and Technology (TIST), 11(2):1–30, 2020. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guan-2023-leveraging-pre-trained-large-language/