Elamur Research⌕
☰
home / papers / 2026 / DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints

DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints

Paper · 2026 · Yinger Zhang, Shutong Jiang, Renhao Li et al.

Summary

DeepPlanning is a benchmark from the Qwen Team that tests whether LLM agents can plan over long horizons under verifiable constraints. Many existing agent benchmarks reward only step-level reasoning. Planning benchmarks, for their part, tend to skip active information gathering and fine-grained real-world limits.

The benchmark covers two domains:

  • Travel Planning: the agent builds multi-day itineraries scheduled to the minute.
  • Shopping Planning: the agent assembles a multi-product cart.

Each task runs in an offline sandbox backed by a database that the agent can reach only through Python tools. Outputs are scored by rule-based checkers. Tasks were built through layered generation, starting from a base skeleton and then injecting personalized and environment constraints, followed by manual quality control.

The authors assess three abilities: actively seeking out the information a task requires, reasoning correctly about constraints at the level of individual steps, and optimizing the plan as a whole against constraints that span it. Frontier agentic models, including GPT-5.x, Claude-4.5-Opus, GLM-4.7 and Grok-4.1-Fast, perform poorly, and scores fall as tasks become more complex. Reliable explicit reasoning and parallel tool calls give better trade-offs between quality and cost. The resource is useful for diagnosing where agents break down on realistic, tightly coupled planning tasks.

Key points

  • Travel Planning: 120 tasks, each available in Chinese (ZH) and English (EN). The agent works with 9 APIs over 7 sub-databases, averaging about 7,708 records per task. The required output is a structured itinerary report with itemized costs.
  • Shopping Planning: 120 English tasks with 15 APIs and 3 sub-databases, averaging about 171 records per task. The output is a JSON shopping cart that must account for sizes, shipping times, coupons and budget.
  • Interaction strategy: GPT-5.2-high works step by step and scores 12.7% higher than GPT-5.1-high, which batches tool calls in parallel. However, GPT-5.2-high needs roughly 10× as many turns.
  • Task complexity: performance declines as Travel itineraries grow from 2 to 7 days. It also declines across Shopping Levels 1 to 3, as constraints shift from price ranges to coupon timing.
  • Error analysis: the authors annotated 140 failed Claude-4.5-Opus trajectories (80 Travel, 60 Shopping).
    • Global optimization failures are the most frequent (101 Travel, 52 Shopping).
    • Implicit-constraint violations come next (86 Travel, 21 Shopping).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China168 works · 15,386 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Yinger Zhang, Shutong Jiang, Renhao Li, Jianhong Tu, Yang Su, Lianghao Deng, Xudong Guo, ChenXu Lv and Junyang Lin. 2026. DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.335

@inproceedings{zhang2026deepplanning,
  title = {{DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints}},
  author = {Yinger Zhang, Shutong Jiang, Renhao Li, Jianhong Tu, Yang Su, Lianghao Deng, Xudong Guo, ChenXu Lv and Junyang Lin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.335},
  url = {https://doi.org/10.18653/v1/2026.acl-long.335}
}

References

33 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pierre Andrews, Amine Benhalloum, Gerard Moreno-Torres Bertran, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Romain Froger, Emilien Garreau, Jean-Baptiste Gaya, and 1 others. 2025. ARE: Scaling up agent environments and evaluations. arXiv preprint arXiv:2509.17158. — [arXiv:2509.17158](https://arxiv.org/abs/2509.17158) find
  2. Anthropic. 2025. System card: Claude Opus 4.5. https://assets.anthropic.com/m/64823ba7485345a7/ Claude-Opus-4-5-System-Card.pdf. Accessed: 2025-12-22. ↗
  3. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. τ 2 -Bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982. find
  4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  5. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2Web: Towards a generalist agent for the web. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  6. Zifeng Ding, Sikuan Yan, Moy Yuan, Xianglong Hu, Fangru Lin, and Andreas Vlachos. 2025. TCP: a benchmark for temporal constraint-based planning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 22463–22486, Suzhou, China. Association for Computational… find
  7. Google. 2025. Gemini 3 pro model card. https://storage.googleapis.com/deepmind-media/ Model-Cards/Gemini-3-Pro-Model-Card.pdf. Accessed: 2025-12-22. ↗
  8. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. WebVoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
  9. Wei He, Yueqing Sun, Hongyan Hao, Xueyuan Hao, Zhikang Xia, Qi Gu, Chengcheng Han, Dengchang Zhao, Hui Su, Kefeng Zhang, and 1 others. 2025. VitaBench: Benchmarking llm agents with versatile interactive tasks in real-world applications. arXiv preprint arXiv:2509.26490. — [arXiv:2509.26490](https://arxiv.org/abs/2509.26490) find
  10. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-bank: A comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3102–3116, Singapore. Ass…
  11. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025. DeepSeek-v3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
  12. Yougang Lyu, Xiaoyu Zhang, Lingyong Yan, Maarten de Rijke, Zhaochun Ren, and Xiuying Chen. 2025. DeepShop: A benchmark for deep research shopping agents. arXiv preprint arXiv:2506.02839. — [arXiv:2506.02839](https://arxiv.org/abs/2506.02839) find
  13. OpenAI. 2025a. GPT-5 system card. https://cdn.openai.com/gpt-5-system-card.pdf. Accessed: 2025-12-22. ↗
  14. OpenAI. 2025b. OpenAI o3 and o4-mini system card. https://cdn.openai.com/pdf/ 2221c875-02dc-4789-800b-e7758f3722c1/ o3-and-o4-mini-system-card.pdf. Accessed: 2025-12-22. ↗
  15. Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning. find
  16. Cheng Qian, Zuxin Liu, Akshara Prabhakar, Zhiwei Liu, Jianguo Zhang, Haolin Chen, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, and Huan Wang. 2025. UserBench: An interactive gym environment for user-centric agents. In Workshop on Scaling Environments for Agents. find
  17. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating large language models to master 16000+ real…
  18. Yincen Qu, Huan Xiao, Feng Li, Gregory Li, Hui Zhou, Xiangying Dai, and Xiaoru Dai. 2025. TripScore: Benchmarking and rewarding real-world travel planning with fine-grained evaluation. arXiv preprint arXiv:2510.09011. — [arXiv:2510.09011](https://arxiv.org/abs/2510.09011) find
  19. ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, and 1 others. 2025. Seed1.5-Thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914.
  20. Jie-Jing Shao, Bo-Wen Zhang, Xiao-Wen Yang, Baizhi Chen, Siyu Han, Wen-Da Wei, Guohao Cai, Zhenhua Dong, Lan-Zhe Guo, and Yu-Feng Li. 2025. ChinaTravel: An open-ended benchmark for language agents in chinese travel planning. In NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmar… find
  21. Harmanpreet Singh, Nikhil Verma, Yixiao Wang, Manasa Bharadwaj, Homa Fashandi, Kevin Ferreira, and Chul Lee. 2024. Personal large language model agents: A case study on tailored travel planning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track… find
  22. Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2025. On the self-verification limitations of large language models on reasoning and planning tasks. In The Thirteenth International Conference on Learning Representations. find
  23. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, and 1 others. 2025. Kimi K2: Open agentic intelligence. arXiv preprint arXiv:2507.20534.
  24. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023. PlanBench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benc…
  25. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516. xAI. 2025. Grok 4.1 model card. https://… find
  26. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. 2024. TravelPlanner: A benchmark for real-world planning with language agents. In Forty-first International Conference on Machine Learning. find
  27. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  28. Shunyu Yao, Howard Chen, John Yang, and Karthik R Narasimhan. 2022. WebShop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems.
  29. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. τ -Bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045.
  30. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, and 1 others. 2025. GLM-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471.
  31. Yikai Zhang, Siyu Yuan, Caiyu Hu, Kyle Richardson, Yanghua Xiao, and Jiangjie Chen. 2024. TimeArena: Shaping efficient multitasking language agents in a time-aware simulation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 38… find
  32. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representation…
  33. Kaimin Wang, Yuanzhe Shen, Changze Lv, Xiaoqing Zheng, and Xuanjing Huang. 2025. TripTailor: A real-world benchmark for personalized travel planning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 9705–9723, Vienna, Austria. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-deepplanning-benchmarking-long-horizon-agentic/