Elamur Research⌕
☰
home / papers / 2024 / LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks

LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks

Paper · 2024 · Subbarao Kambhampati, Karthik Valmeekam, Lin Guan et al.

Summary

This position paper responds to two opposing errors in the debate on LLMs and planning. One overstates their reasoning ability. The other treats them as mere text-to-symbol converters for classical solvers. The authors review evidence, much of it from their own group, that autoregressive LLMs cannot reliably produce executable plans or check plans and solutions. PlanBench experiments on IPC-style domains show weak plan generation, and performance falls further when action and object names are obfuscated. This suggests the models approximately retrieve plans rather than reason about them. Graph-coloring studies show that self-critique does not help and can even hurt. As an alternative, the paper proposes LLM-Modulo Frameworks. In these, LLMs generate candidate plans, reformat them, help refine specifications and help acquire domain models, while external sound verifiers, critics and humans check every candidate. The authors argue this design keeps formal correctness guarantees and avoids the expressiveness and search limits of purely symbolic planners. This matters given the current push toward agentic LLM systems.

Key points

  • In autonomous mode, only about 12% of GPT-4's plans were executable and reached the goal. Fine-tuning and the choice of model (GPT-3 through GPT-4) changed this little.
  • PlanBench Table 1 shows uneven results:
    • On Blocksworld, the best score was Claude-3-Opus zero-shot at 356/600 (59.3%).
    • On Mystery Blocksworld, every model scored no higher than 26/600 (4.3%, GPT-4 one-shot).
  • In graph coloring, GPT-4 failed both to solve and to verify instances. When it critiqued its own answers, it sometimes rejected correct colorings and ended with wrong ones.
  • Chain-of-thought and ReAct-style prompting brought little improvement on planning tasks.
  • The framework's name is loosely inspired by SAT Modulo Theories. The authors link it to LLM-assisted model and reward construction in RL, citing SayCan as an example. They include two case studies, one of them on travel planning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Saldyt and Anil Murthy. 2024. LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks. arXiv:2402.01817 · Proceedings of the 41 st International Conference on Machine Learning, Vienna, Austria. PMLR 235, 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.01817 · arXiv 2402.01817

@misc{kambhampati2024llms,
  title = {{LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks}},
  author = {Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Saldyt and Anil Murthy},
  year = {2024},
  doi = {10.48550/arxiv.2402.01817},
  eprint = {2402.01817},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.01817}
}

References

60 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gendron, G., Bao, Q., Witbrock, M., and Dobbie, G. Large language models are not abstract reasoners. arXiv preprint arXiv:2305.19555, 2023.
  2. Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., Sanyal, S., Ren, X., Ettinger, A., Harchaoui, Z., and Choi, Y. Faith and fate: Limits of transformers on compositionality. In Thirty-seventh Conference on Neural Informa…
  3. Clever Hans. https://en.wikipedia.org/wiki/Clever Hans. Ghallab, M., Nau, D., and Traverso, P. Automated Planning: theory and practice. Elsevier, 2004. ↗
  4. Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
  5. Guan, L., Sreedharan, S., and Kambhampati, S. Leveraging approximate symbolic models for reinforcement learning via skill diversity. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 1… find
  6. Arora, D. and Kambhampati, S. Learning and leveraging verifiers to improve planning capabilities of pre-trained language models. ICML Workshop on Knowledge and Logical Reasoning in the Era of Data-driven Learning (arXiv preprint arXiv:2305.17077), 2023.
  7. Guan, L., Valmeekam, K., Sreedharan, S., and Kambhampati, S. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=zDbsSscmuj. ↗
  8. Bairi, R., Sonwane, A., Kanade, A., Iyer, A., Parthasarathy, S., Rajamani, S., Ashok, B., Shet, S., et al. Codeplan: Repository-level coding using llms and planning. arXiv preprint arXiv:2309.12499, 2023. — [arXiv:2309.12499](https://arxiv.org/abs/2309.12499) find
  9. Bhambri, S., Bhattacharjee, A., Liu, H., and Kambhampati, Guan, L., Zhou, Y., Liu, D., Zha, Y., Amor, H. B., and Kambhampati, S. ”task success” is not enough: Investigating the use of video-language models as behavior critics for catching undesirable agent behaviors, 2024. find
  10. Kambhampati, S. Polanyi’s revenge and AI’s new romance with tacit knowledge. Communications of the ACM, 64 (2):31–32, 2021. find
  11. Kambhampati, S. Can LLMs reason and plan? Annals of the New York Academy of Sciences, 2024. find
  12. Gundawar, A., Verma, M., Guan, L., Valmeekam, K., Bhambri, S., and Kambhampati, S. Robust planning with llm-modulo framework: Case study in travel planning. arXiv preprint arxiv:2405.20625, 2024. — [arXiv:2405.20625](https://arxiv.org/abs/2405.20625) find
  13. Kambhampati, S., Valmeekam, K., Marquez, M., and Guan, L. On the role of large language models in planning, July 2023. URL https://yochan-lab.github.io/tutorial/ICAPS-2023/. Tutorial presented at the International Conference on Automated Planning and Scheduling (ICAPS), Prague. ↗
  14. Hao, S., Gu, Y., Ma, H., Hong, J. J., Wang, Z., Wang, D. Z., and Hu, Z. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023. find
  15. Howey, R., Long, D., and Fox, M. VAL: Automatic plan validation, continuous effects and mixed initiative planning using PDDL. In 16th IEEE International Conference on Tools with Artificial Intelligence, pp. 294–301. IEEE, 2004. find
  16. Kugel, S. and Hiltner, S. A new frontier for travel scammers: A.I.-Generated Guidebooks. New York Times, August 2023. URL https://www.nytimes.com/2023/08/05/travel/ amazon-guidebooks-artificial-intelligence. html. ↗
  17. Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023a.
  18. Kwon, M., Xie, S. M., Bullard, K., and Sadigh, D. Reward design with language models. In The Eleventh International Conference on Learning Representations, 2022.
  19. Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Language model programs for embodied control, 2023. find
  20. Huang, J., Gu, S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1051–1068, Singapore, December 2023b. Association for C…
  21. Liu, B., Jiang, Y., Zhang, X., Liu, Q., Zhang, S., Biswas, J., and Stone, P. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477, 2023.
  22. Ma, Y. J., Liang, W., Wang, G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar, A. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.12931, 2023.
  23. Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. find
  24. McCoy, R. T., Yao, S., Friedman, D., Hardy, M., and Griffiths, T. L. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638, 2023. find
  25. IPC. International planning competition, 1998. URL https://www.icaps-conference.org/ competitions/. ↗
  26. McDermott, D., Ghallab, M., Howe, A. E., Knoblock, C. A., Ram, A., Veloso, M. M., Weld, D. S., and Wilkins, D. E. Pddl-the planning domain definition language. 1998. find
  27. Johnston, M. D., Tran, D., Arroyo, B., Sorensen, S., Tay, P., Carruth, B., Coffman, A., and Wallace, M. Automated scheduling for nasa’s deep space network. AI Magazine, 35(4):7–25, 2014. find
  28. Nieuwenhuis, R. and Oliveras, A. On sat modulo theories and optimization problems. In Theory and Applications of Satisfiability Testing-SAT 2006: 9th International Conference, Seattle, WA, USA, August 12-15, 2006. Proceedings 9, pp. 156–169. find
  29. Kahneman, D. Thinking, fast and slow. macmillan, 2011. find
  30. Kambhampati, S. Model-lite planning for the web age masses: The challenges of planning with incomplete and evolving domain models. In Proceedings of the National Conference on Artificial Intelligence, volume 22, pp. 1601. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2007. find
  31. Olmo, A., Sreedharan, S., and Kambhampati, S. Gpt3-to-plan: Extracting plans from text using gpt-3. FinPlan 2021, pp. 24, 2021. find
  32. OpenAI. Introducing chatgpt by openai, 2022. https://openai.com/blog/chatgpt. ↗
  33. URL Stechly, K., Valmeekam, K., and Kambhampati, S. On the self-verification limitations of large language models on reasoning and planning tasks. arXiv preprint arxiv:2402.08115, 2024a. find
  34. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  35. Stechly, K., Valmeekam, K., and Kambhampati, S. Chain of thoughtlessness: An analysis of cot in planning. arXiv preprint arxiv:2405.04776, 2024b. — [arXiv:2405.04776](https://arxiv.org/abs/2405.04776) find
  36. OpenAI. Gpt-4 technical report, 2023.
  37. Trinh, T. H., Wu, Y., Le, Q. V., He, H., and Luong, T. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476–482, 2024.
  38. Pan, L., Albalak, A., Wang, X., and Wang, W. Y. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv:2305.12295, 2023.
  39. Ullman, T. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399, 2023. find
  40. Rajvanshi, A., Sikka, K., Lin, X., Lee, B., Chiu, H.-P., and Velasquez, A. Saynav: Grounding large language models for dynamic planning to navigation in new environments. arXiv preprint arXiv:2309.04077, 2023. — [arXiv:2309.04077](https://arxiv.org/abs/2309.04077) find
  41. Valmeekam, K., Marquez, M., and Kambhampati, S. Can large language models really improve by self-critiquing their own plans? In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023a. find
  42. Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J., Ellenberg, J. S., Wang, P., Fawzi, O., et al. Mathematical discoveries from program search with large language models. Nature, pp. 1–3, 2023.
  43. Valmeekam, K., Marquez, M., Olmo, A., Sreedharan, S., and Kambhampati, S. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023b. URL ht…
  44. Russell, S. J. and Norvig, P. Artificial intelligence a modern approach. London, 2010. find
  45. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  46. Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL https://arxiv.org/abs/2010.03768. ↗
  47. Silver, T., Hariprasad, V., Shuttleworth, R. S., Kumar, N., Lozano-Pérez, T., and Kaelbling, L. P. PDDL planning with pretrained large language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022. URL https://openreview.net/forum? id=1QMMUB4zfl. ↗
  48. Valmeekam, K., Marquez, M., Sreedharan, S., and Kambhampati, S. On the planning abilities of large language models - a critical investigation. In Thirty-seventh Conference on Neural Information Processing Systems (Spotlight), 2023c. URL — On the Planning Abilities of Large Language Models : A Critical Investigation https://openreview.net/ forum?id=X6dEqXIsEW. ↗
  49. Verma, M., Bhambri, S., and Kambhampati, S. On the brittle foundations of react prompting for agentic large language models. arXiv preprint arXiv:2405.13966, 2024a. — [arXiv:2405.13966](https://arxiv.org/abs/2405.13966) find
  50. Verma, M., Bhambri, S., and Kambhampati, S. Theory of mind abilities of large language models in human-robot interaction: An illusion? arXiv preprint arXiv:2401.05302, 2024b. — [arXiv:2401.05302](https://arxiv.org/abs/2401.05302) find
  51. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models, 2023a. find
  52. Simpson, R., McCluskey, T. L., and Zhao, W. Gipo: an integrated graphical tool to support knowledge engineering in ai planning. In ECP-01, pp. 445. Citeseer, 2001. find
  53. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023b. find
  54. Stechly, K., Marquez, M., and Kambhampati, S. GPT-4 Doesn’t Know It’s Wrong: An Analysis of Iterative Prompting for Reasoning Problems. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. find
  55. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. find
  56. Weng, Y., Zhu, M., Xia, F., Li, B., He, S., Liu, S., Sun, B., Liu, K., and Zhao, J. Large language models are better reasoners with self-verification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 2550–2575, 2023.
  57. Xie, J., Zhang, K., Chen, J., Zhu, T., Lou, R., Tian, Y., Xiao, Y., and Su, Y. Travelplanner: A benchmark for real-world planning with language agents. arXiv preprint arxiv:2402.01622, 2024.
  58. Xie, Y., Yu, C., Zhu, T., Bai, J., Gong, Z., and Soh, H. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128, 2023. — [arXiv:2302.05128](https://arxiv.org/abs/2302.05128) find
  59. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. R. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https://openreview.net/ forum?id=5Xc1ecxO1h. ↗
  60. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023b. URL https://openreview.net/forum? id=WE_vluYUL-X. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kambhampati-2024-llms-t-plan-but-help/