Elamur Research⌕
☰
home / papers / 2024 / TravelPlanner: A Benchmark for Real-World Planning with Language Agents

TravelPlanner: A Benchmark for Real-World Planning with Language Agents

Paper · 2024 · Jian Xie, Kai Zhang, Jiangjie Chen et al.

Summary

Planning has long been studied in AI, but mostly in narrow settings with a single objective and a fixed correct answer. This paper asks whether LLM-powered language agents can handle open-ended, realistic planning. The authors introduce TravelPlanner, a benchmark built around multi-day trip planning. Agents work in a sandbox of about four million records collected from the web, which they access through six search tools covering cities, flights, distances, restaurants, attractions and lodging. A valid itinerary must meet the user's explicit requirements, such as budget, room type and pet rules, as well as implicit commonsense rules, such as a sensible route and no conflicting transport. Five LLMs, including GPT-4, Gemini and Mixtral, were tested with four planning strategies, including ReAct and Reflexion. The results are poor: GPT-4 satisfies every constraint in only 0.6% of cases, and the other models never do. The benchmark gives the field a demanding, realistic way to measure progress toward human-like planning with many constraints.

Key points

  • The dataset has 1,225 curated queries with reference plans. They fall into nine groups defined by trip length and the number of hard constraints. The splits are 45 training queries with human-annotated plans, 180 validation queries and 1,000 test queries.
  • The best final pass rate is 0.6%, achieved by GPT-4. All other evaluated LLMs score zero.
  • ReAct and Reflexion fall short on this task. Agents often act against their own reasoning, for example picking expensive options right after noting the need to save money. They also lose track of global constraints such as "Budget" and "Minimum Nights Stay".
  • Common failure modes include:
    • wrong tool arguments, such as dates outside the 2022 sandbox data;
    • dead loops that the agent never corrects;
    • hallucinations caused by mixing up retrieved facts, such as reusing one flight number for both legs of a trip.
  • Trained human annotators need about 12 minutes per plan, while agents produce one in 1–2 minutes. This suggests real potential for automation once agents become more reliable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations🎓The Ohio State UniversityColumbus · United States90 works · 19,112 citations🎓Pennsylvania State UniversityState College · United States67 works · 5,865 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao and Yu Su. 2024. TravelPlanner: A Benchmark for Real-World Planning with Language Agents. arXiv:2402.01622.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.01622 · arXiv 2402.01622

@misc{xie2024travelplanner,
  title = {{TravelPlanner: A Benchmark for Real-World Planning with Language Agents}},
  author = {Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao and Yu Su},
  year = {2024},
  doi = {10.48550/arxiv.2402.01622},
  eprint = {2402.01622},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.01622}
}

References

67 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alterovitz, R., Koenig, S., and Likhachev, M. Robot planning in the real world: Research challenges and opportunities. Ai Magazine, 2016. find
  2. Andreas, J. Language models as agent models. In Findings of EMNLP, 2022.
  3. AutoGPT. Autogpt, 2023. URL https://github.com/Significant-Gravitas/AutoGPT. ↗
  4. Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Gianinazzi, L., Gajda, J., Lehmann, T., Podstawski, M., Niewiadomski, H., Nyczyk, P., et al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687, 2023. find
  5. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Proceedings of NeurIPS, 2020.
  6. Campbell, M., Hoane Jr, A. J., and Hsu, F.-h. Deep blue. Artificial intelligence, 2002. find
  7. Chen, H., Pasunuru, R., Weston, J., and Celikyilmaz, A. Walking down the memory maze: Beyond context limit through interactive reading. arXiv preprint arXiv:2310.05029, 2023. — [arXiv:2310.05029](https://arxiv.org/abs/2310.05029) find
  8. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  9. Cross, S. and Estrada, R. Dart: an example of accelerated evolutionary development. In Proceedings of Workshop on RSP, 1994. find
  10. Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards a generalist agent for the web. In Proceedings of NeurIPS, 2023.
  11. G Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  12. Ge, Y., Hua, W., Ji, J., Tan, J., Xu, S., and Zhang, Y. Openagi: When llm meets domain experts. In Proceedings of NeurIPS, 2023. find
  13. Georgievski, I. and Aiello, M. Htn planning: Overview, comparison, and beyond. Artificial Intelligence, 2015. find
  14. Grafman, J., Spector, L., and Rattermann, M. J. Planning and the brain. In The cognitive psychology of planning. 2004. find
  15. Hayes-Roth, B. and Hayes-Roth, F. A cognitive model of planning. Cognitive science, 1979. find
  16. Ho, M. K., Saxe, R., and Cushman, F. Planning with theory of mind. Trends in Cognitive Sciences, 2022. find
  17. Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Proceedings of ICML, 2022.
  18. Ichter, B., Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., Kalashnikov, D., Levine, S., Lu, Y., Parada, C., Rao, K., Sermanet, P., Toshev, A., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Yan, M., Brown, N., Ahn, M., Cortes, O., Sievers,… — Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
  19. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. find
  20. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., La… find
  21. Karpas, E. and Magazzeni, D. Automated planning for robotics. Annual Review of Control, Robotics, and Autonomous Systems, 2020. find
  22. Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V. V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., and Misra, V. Solving quantitative reasoning problems with language models. In Proceedings of NeurIPS, 2022.
  23. Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., and Li, Y. Api-bank: A comprehensive benchmark for tool-augmented llms. In Proceedings of EMNLP, 2023.
  24. Liang, X., Wang, B., Huang, H., Wu, S., Wu, P., Lu, L., Ma, Z., and Li, Z. Unleashing infinite-length input capacity for large-scale language models with self-controlled memory system. arXiv preprint arXiv:2304.13343, 2023. — [arXiv:2304.13343](https://arxiv.org/abs/2304.13343) find
  25. Lin, J., Tomlin, N., Andreas, J., and Eisner, J. Decision-oriented dialogue for human-ai collaboration. arXiv preprint arXiv:2305.20076, 2023.
  26. Liu, B., Jiang, Y., Zhang, X., Liu, Q., Zhang, S., Biswas, J., and Stone, P. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477, 2023a.
  27. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172, 2023b.
  28. Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. Agentbench: Evaluating llms as agents. In Proceedings of ICLR, 2024.
  29. Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y. N., Zhu, S.-C., and Gao, J. Chameleon: Plug-and-play compositional reasoning with large language models. In Proceedings of NeurIPS, 2023.
  30. Mattar, M. G. and Lengyel, M. Planning in the brain. Neuron, 2022. find
  31. McDermott, D. Robot planning. AI magazine, 1992. find
  32. Nakajima, Y. Task-driven autonomous agent utilizing gpt-4, pinecone, and langchain for diverse applications, 2023. find
  33. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.
  34. OpenAI. Chatgpt, 2022. URL https://openai.com/blog/chatgpt. ↗
  35. OpenAI. Gpt-4 technical report. arXiv:2303.08774, 2023. arXiv preprint Park, J. S., O’Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of UIST, 2023.
  36. Patel, A., Bhattamishra, S., and Goyal, N. Are NLP models really able to solve simple math word problems? In Proceedings of NAACL, 2021.
  37. Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023.
  38. Pinedo, M. Planning and scheduling in manufacturing and services. Springer, 2005. find
  39. Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. In Proceedings of ICLR, 2024.
  40. Roy, S. and Roth, D. Solving general arithmetic word problems. In Proceedings of EMNLP, 2015.
  41. Russell, S. J. and Norvig, P. Artificial intelligence a modern approach. 2010. find
  42. Schick, T., Dwivedi-Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. In Proceedings of NeurIPS, 2023.
  43. Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. HuggingGPT: Solving AI tasks with chatGPT and its friends in hugging face. In Proceedings of NeurIPS, 2023.
  44. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: Language agents with verbal reinforcement learning. In Proceedings of NeurIPS, 2023.
  45. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature, 2016. find
  46. Silver, D., Hasselt, H., Hessel, M., Schaul, T., Guez, A., Harley, T., Dulac-Arnold, G., Reichert, D., Rabinowitz, N., Barreto, A., et al. The predictron: End-to-end learning and planning. In Proceedings of ICML, 2017. find
  47. Song, C. H., Wu, J., Washington, C., Sadler, B. M., Chao, W.-L., and Su, Y. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of ICCV, 2023.
  48. Su, Y. Language agents: a critical evolutionary step of artificial intelligence. 2023. URL https://yusu.substack.com/p/language-agents. ↗
  49. Sumers, T. R., Yao, S., Narasimhan, K., and Griffiths, T. L. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427, 2023. find
  50. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
  51. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  52. Valmeekam, K., Olmo, A., Sreedharan, S., and Kambhampati, S. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Proceedings of NeurIPS, 2023.
  53. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. find
  54. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of NeurIPS, 2022.
  55. Weng, L. Llm-powered autonomous agents. lilian-weng.github.io, Jun 2023. URL https://lilianweng.github.io/posts/2023-06-23-agent/. ↗
  56. Xie, T., Zhou, F., Cheng, Z., Shi, P., Weng, L., Liu, Y., Hua, T. J., Zhao, J., Liu, Q., Liu, C., Liu, L. Z., Xu, Y., Su, H., Shin, D., Xiong, C., and Yu, T. Openagents: An open platform for language agents in the wild. arXiv preprint arXiv:2310.10634, 2023.
  57. Xu, Q., Hong, F., Li, B., Hu, C., Chen, Z., and Zhang, J. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv:2305.16504, 2023.
  58. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In Proceedings of ICLR, 2022.
  59. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. R. Tree of thoughts: Deliberate problem solving with large language models. In Proceedings of NeurIPS, 2023. find
  60. Yuan, S., Chen, J., Fu, Z., Ge, X., Shah, S., Jankowski, C., Xiao, Y., and Yang, D. Distilling script knowledge from large language models for constrained language planning. In Proceedings of ACL, 2023. find
  61. Zhang, Y., Yuan, S., Hu, C., Richardson, K., Xiao, Y., and Chen, J. Timearena: Shaping efficient multitasking language agents in a time-aware simulation. arXiv preprint arXiv:2402.05733, 2024. find
  62. Zheng, B., Gou, B., Kil, J., Sun, H., and Su, Y. Gpt-4v(ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024.
  63. Zhong, W., Guo, L., Gao, Q., and Wang, Y. Memorybank: Enhancing large language models with long-term memory. arXiv preprint arXiv:2305.10250, 2023. find
  64. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., et al. Webarena: A realistic web environment for building autonomous agents. In Proceedings of ICLR, 2024.
  65. Zhou, W., Jiang, Y. E., Cui, P., Wang, T., Xiao, Z., Hou, Y., Cotterell, R., and Sachan, M. Recurrentgpt: Interactive generation of (arbitrarily) long text. arXiv preprint arXiv:2305.13304, 2023.
  66. Zhuang, Y., Yu, Y., Wang, K., Sun, H., and Zhang, C. ToolQA: A dataset for LLM question answering with external tools. In Proceedings of NeurIPS, 2023.
  67. Zhuang, Y., Chen, X., Yu, T., Mitra, S., Bursztyn, V., Rossi, R. A., Sarkhel, S., and Zhang, C. Toolchain*: Efficient action space navigation in large language models with a* search. In Proceedings of ICLR, 2024. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xie-2024-travelplanner-benchmark-real-world-planning/