Elamur Research⌕
☰
home / papers / 2025 / The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

Paper · 2025 · Junlong Li, Wenshuo Zhao, Jian Zhao et al.

Summary

Most benchmarks for tool-using language agents test narrow domains, short interactions, or artificial environments, so they reveal little about how agents handle real multi-application work. To fill this gap, the authors built TOOLATHLON (Tool Decathlon). It contains 108 hand-built tasks that use 32 software applications and 604 tools, from everyday services like Google Calendar and Notion to professional systems such as Snowflake, Kubernetes, WooCommerce and BigQuery. Most tools come from Model Context Protocol (MCP) servers, which the team fixed, extended, or wrote themselves.

Each task starts from a realistic initial state, such as a Canvas course with many students or genuine financial spreadsheets. Prompts are short and somewhat vague, like real user requests. A deterministic script checks the final environment state to score each task.

Current models perform poorly. The best, Claude-4.5-Sonnet, solves only 38.6% of tasks. The analysis points to weak long-context handling and poor recovery from tool-call errors. The benchmark is fully open-sourced and offers a demanding testbed for practical, long-horizon agents.

Key points

  • Scale: 108 tasks, 32 applications (MCP servers), 604 tools. Most tasks span several apps and take about 20 tool-calling turns.
  • Results: Claude-4.5-Sonnet scores 38.6% with 20.2 turns on average. The strongest open-weights model, DeepSeek-V3.2-Exp, reaches 20.1%.
  • Realistic environments: Some services that are hard to reset run locally in containers, with poste.io replacing Gmail and WooCommerce replacing Shopify. This allows rich initial states that can be restored between runs.
  • Evaluation framework: Each task runs in its own container and is scored by an execution-based script. Ground truth is either static or generated dynamically, for example from live train schedules. A full Claude-4.5-Sonnet run takes about 70 minutes with 10 parallel processes.
  • Availability: The benchmark and environment are released at toolathlon.xyz and github.com/hkust-nlp/toolathlon (HKUST, All Hands AI, CMU, Duke).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Junlong Li, Wenshuo Zhao, Jian Zhao, Weihao Zeng, Haoze Wu, Xiaochen Wang, Rui Ge, Yuxuan Cao, Yuzhen Huang, Wei Liu, Junteng Liu, Zhaochen Su, Yiyang Guo, Fan Zhou, Lueyang Zhang, Juan Michelini, Xingyao Wang, Xiang Yue, Shuyan Zhou, Graham Neubig and Junxian He. 2025. The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution. arXiv:2510.25726.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2510.25726 · arXiv 2510.25726

@misc{li2025tool,
  title = {{The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution}},
  author = {Junlong Li, Wenshuo Zhao, Jian Zhao, Weihao Zeng, Haoze Wu, Xiaochen Wang, Rui Ge, Yuxuan Cao, Yuzhen Huang, Wei Liu, Junteng Liu, Zhaochen Su, Yiyang Guo, Fan Zhou, Lueyang Zhang, Juan Michelini, Xingyao Wang, Xiang Yue, Shuyan Zhou, Graham Neubig and Junxian He},
  year = {2025},
  doi = {10.48550/arxiv.2510.25726},
  eprint = {2510.25726},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.25726}
}

References

36 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pierre Andrews, Amine Benhalloum, Gerard Moreno-Torres Bertran, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Romain Froger, Emilien Garreau, Jean-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Grégoire Mialon, Ulyana Piterbarg,… — [arXiv:2509.17158](https://arxiv.org/abs/2509.17158) find
  2. Anthropic. Introducing the model context protocol. model-context-protocol, 2024. https://www.anthropic.com/news/ ↗
  3. Anthropic. Introducing claude 4. https://www.anthropic.com/news/claude-4, 2025a. ↗
  4. Anthropic. https://www.anthropic.com/news/claude-haiku-4-5. https://www.anthropic.com/ news/claude-haiku-4-5, 2025b. ↗
  5. Anthropic. Introducing claude sonnet 4.5. claude-sonnet-4-5, 2025c. https://www.anthropic.com/news/ ↗
  6. Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang, Wulong Liu, Xinzhi Wang, Defu Lian, Baoqun Yin, Yasheng Wang, and Wu Liu. ACEBench: Who wins the match point in tool usage?, 2025. URL https://arxiv.org/abs/2501.12851. — [arXiv:2501.12851](https://arxiv.org/abs/2501.12851) ↗
  7. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint a…
  8. DeepSeek-AI. Introducing deepseek-v3.2-exp. news250929, 2025. https://api-docs.deepseek.com/news/ ↗
  9. Xuanqi Gao, Siyi Xie, Juan Zhai, Shqing Ma, and Chao Shen. MCP-RADAR: A multi-dimensional benchmark for evaluating tool use capabilities in large language models, 2025. URL https://arxiv.org/abs/2505.16700. — [arXiv:2505.16700](https://arxiv.org/abs/2505.16700) ↗
  10. Zikang Guo, Benfeng Xu, Chiwei Zhu, Wentao Hong, Xiaorui Wang, and Zhendong Mao. Mcp-agentbench: Evaluating real-world language agent performance with mcp-mediated tools, 2025. URL https://arxiv.org/abs/2509.09734. — [arXiv:2509.09734](https://arxiv.org/abs/2509.09734) ↗
  11. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/ forum?id=VTF8yNQM66. ↗
  12. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025.
  13. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating LLMs as age… — AgentBench: Evaluating LLMs as Agents
  14. Zhiwei Liu, Jielin Qiu, Shiyu Wang, Jianguo Zhang, Zuxin Liu, Roshan Ram, Haolin Chen, Weiran Yao, Shelby Heinecke, Silvio Savarese, Huan Wang, and Caiming Xiong. MCPEval: Automatic mcp-based deep evaluation for ai agent models, 2025. URL https://arxiv.org/abs/2507. 12806. ↗
  15. Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. MCP-Universe: Benchmarking large language models with real-world model context protocol servers, 2025. URL https://arxiv.org/abs/2508.14704. — [arXiv:2508.14704](https://arxiv.org/abs/2508.14704) ↗
  16. Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn LLM agents. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL… find
  17. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general ai assistants, 2023. URL https://arxiv.org/abs/2311.12983. ↗
  18. Guozhao Mo, Wenliang Zhong, Jiawei Chen, Xuanang Chen, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, and Le Sun. Livemcpbench: Can agents navigate an ocean of mcp tools?, 2025. URL https://arxiv.org/abs/2508.01780. — [arXiv:2508.01780](https://arxiv.org/abs/2508.01780) ↗
  19. OpenAI. Introducing deep introducing-deep-research/, 2024. research. find
  20. OpenAI. Introducing openai o3 and introducing-o3-and-o4-mini/, 2025a. o4-mini. https://openai.com/index/https://openai.com/index/ ↗
  21. OpenAI. Introducing gpt-5. https://openai.com/index/introducing-gpt-5/, 2025b. ↗
  22. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=tBRNC6YemY. ↗
  23. Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. URL h… find
  24. Qwen Team. Qwen3-coder: Agentic coding in the world. https://qwenlm.github.io/blog/ qwen3-coder/, 2025. ↗
  25. Scale AI. Mcp atlas. https://scale.com/leaderboard/mcp atlas, 2025. ↗
  26. The MCPMark Team. MCPMark: Stress-testing comprehensive mcp use. https://github.com/ eval-sys/mcpmark, 2025. ↗
  27. The Terminal-Bench Team. Terminal-bench: A benchmark for ai agents in terminal environments, Apr 2025. URL https://github.com/laude-institute/terminal-bench. ↗
  28. Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Pr…
  29. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. URL https://arxiv.org/abs/2504.12516. xAI. Grok-4. https://x.ai/news/grok… ↗
  30. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real… — OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
  31. Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. Theagentcompany: Benchma… find
  32. Yunhe Yan, Shihe Wang, Jiajun Du, Yexuan Yang, Yuxuan Shan, Qichen Qiu, Xianqing Jia, Xinge Wang, Xin Yuan, Xu Han, Mao Qin, Yinxiao Chen, Chen Peng, Shangguang Wang, and Mengwei Xu. MCPWorld: A unified benchmarking testbed for api, gui, and hybrid computer use agents, 2025. URL https://arxiv.org/ab… ↗
  33. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. τ -bench: A benchmark for Tool-Agent-User interaction in real-world domains. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=roNSXZpUDN. ↗
  34. Ming Yin, Dinghan Shen, Silei Xu, Jianbing Han, Sixun Dong, Mian Zhang, Yebowen Hu, Shujian Liu, Simin Ma, Song Wang, Sathish Reddy Indurthi, Xun Wang, Yiran Chen, and Kaiqiang Song. Livemcp-101: Stress testing and diagnosing mcp-enabled agents on challenging queries, 2025. URL https://arxiv.org/abs… — [arXiv:2508.15760](https://arxiv.org/abs/2508.15760) ↗
  35. Zhipu AI. Glm-4.6: Advanced agentic, reasoning and coding capabilities. https://z.ai/blog/ glm-4.6, 2025. ↗
  36. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 202…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2025-tool-decathlon-benchmarking-language-agents/