Elamur Research⌕
☰
home / papers / 2025 / Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation

Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation

Paper · 2025 · Sayash Kapoor, Benedikt Stroebl, Peter Kirgis et al.

Summary

Evaluating AI agents is slow, inconsistent and error-prone. Benchmarks can take weeks to run serially, costs are rarely reported, scaffolds are seldom compared, and agents may game tasks or act unsafely without detection. The authors built the Holistic Agent Leaderboard (HAL), an open-source harness. It accepts any agent through a minimal Python interface, logs every model call along with its price, and spreads runs across hundreds of virtual machines, which cuts evaluation time from weeks to hours. Using HAL, they jointly varied models, scaffolds and benchmarks across coding, web navigation, science and customer service. Results are reported as accuracy–cost Pareto frontiers rather than accuracy alone. Automated review of the logs revealed hidden problems: agents fetching benchmark answers online, risky payment actions, and a scaffold with data leakage. One counterintuitive finding was that higher reasoning effort reduced accuracy in most runs. HAL matters because it makes agent comparisons reproducible and aware of both cost and behavior. The team plans to maintain it as a shared community resource.

Key points

  • The scale is 21,730 agent rollouts covering 9 models and 9 benchmarks, at a compute cost of about $40,000. The released logs contain roughly 2.5B tokens of LLM calls.
  • The benchmarks are Online Mind2Web, AssistantBench, GAIA, SWE-bench Verified Mini, USACO, CORE-Bench Hard, ScienceAgentBench, SciCode and TAU-bench Airline. HAL is the official leaderboard for CORE-Bench Hard and ScienceAgentBench.
  • The harness relies on Weave for telemetry and LiteLLM for cross-provider model access. It runs locally, in Docker or on Azure VMs, and scaffolds only need to expose a run(input) → dict function.
  • The authors used Docent to analyze logs for four benchmarks. In AssistantBench, they found that the prompt telling agents not to guess hurt Claude Opus 4.1. They also detected leakage in the TAU-bench Few Shot scaffold and excluded it from the results.
  • On USACO, the USACO Episodic + Semantic scaffold paired with GPT-5 Medium scored highest, at 69.7% accuracy for $64.13. Taken together, all agents solved 240 of the 307 problems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 works

Cite

Cite as Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj Oruganty, Sophie Luskin, Kangheng Liu, Botao Yu, Amit Arora, Dongyoon Hahm, Harsh Trivedi, Huan Sun, Juyong Lee, Tengjun Jin, Yifan Mai, Yifei Zhou, Yuxuan Zhu, Rishi Bommasani, Daniel Kang, Dawn Song, Peter Henderson, Yu Su, Percy Liang and Arvind Narayanan. 2025. Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation. arXiv:2510.11977.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2510.11977 · arXiv 2510.11977

@misc{kapoor2025holistic,
  title = {{Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation}},
  author = {Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj Oruganty, Sophie Luskin, Kangheng Liu, Botao Yu, Amit Arora, Dongyoon Hahm, Harsh Trivedi, Huan Sun, Juyong Lee, Tengjun Jin, Yifan Mai, Yifei Zhou, Yuxuan Zhu, Rishi Bommasani, Daniel Kang, Dawn Song, Peter Henderson, Yu Su, Percy Liang and Arvind Narayanan},
  year = {2025},
  doi = {10.48550/arxiv.2510.11977},
  eprint = {2510.11977},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.11977}
}

References

72 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Enterprise h2oGPTe, a. URL https://h2ogpte.genai.h2o.ai/. Ormind - Knowledge Work at AI speed, but your quality standards, b. URL https://ormind. ai/. ReAct Agent, c. URL https://inspect.aisi.org.uk/react-agent.html. ↗
  2. Artificial Analysis Coding Index, September 2025a. URL https://artificialanalysis. ai/?intelligence-tab=coding\#artificial-analysis-coding-index. find
  3. Claude 3.7 Sonnet and Claude Code, February 2025b. URL https://www.anthropic.com/news/claude-3-7-sonnet. ↗
  4. Claude Opus 4.1, August 2025c. claude-opus-4-1. find
  5. Enhans :: Mind2Web: The Benchmark for AI Agent Evaluation and Performance, September 2025d. URL https://www.enhans.ai/newsroom/mind2web-the-benchmark-for-ai-agent-evaluation-and-performance. ↗
  6. Introducing GPT-4.1 in the API, April 2025e. gpt-4-1/. find
  7. Reyna Abhyankar, Qi Qi, and Yiying Zhang. OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents, June 2025. URL http://arxiv.org/abs/2506.16042. arXiv:2506.16042 [cs]. adyuter. adyuter/xManus, September 2025. URL https://github.com/adyuter/xManus. original-date: 2025-04-24T01:50:42Z. — [arXiv:2506.16042](https://arxiv.org/abs/2506.16042) ↗
  8. URL — Deep Reinforcement Learning at the Edge of the Statistical Precipice https://www.anthropic.com/news/URL https://openai.com/index/Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G. Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice, January 2022. URL http://arxiv.org/abs/2108.13264. arXiv:2108.13264 [cs]… ↗
  9. Anthropic. Claude Code, claude-code. Dzmitry Bahdanau, Nicolas Gontier, Gabriel Huang, Ehsan Kamalloo, Rafael Pardinas, Alex Piché, Torsten Scholak, Oleh Shliazhko, Jordan Prince Tremblay, Karam Ghanem, Soham Parikh, Mitul Tiwari, and Quaizar Vohra. TapeAgents: a Holistic Framework for Agent Develo… find
  10. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. Tau2-Bench: Evaluating Conversational Agents in a Dual-Control Environment, June 2025. URL http://arxiv.org/abs/2506.07982. arXiv:2506.07982 [cs]. ↗
  11. Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47:253–279, June 2013. ISSN 1076-9757. doi: 10.1613/jair.3912. URL http://arxiv.org/abs/1207.4708. arXiv:1207.47… ↗
  12. BerriAI. LiteLLM, 2025. URL https://docs.litellm.ai/docs/. ↗
  13. Pratik Bhavsar and Conor Bronsdon. Introducing Our Agent Leaderboard on Hugging Face, 2025. URL https://galileo.ai/blog/agent-leaderboard. ↗
  14. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles… find
  15. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym, June 2016. URL http://arxiv.org/abs/1606.01540. arXiv:1606.01540 [cs]. ↗
  16. Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E Gonzalez, and Ion Stoica. WHY DO MULTI-AGENT LLM SYSTEMS FAIL? Building Trust Workshop at ICLR 2025, 2025.
  17. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. ScienceAgentBench: Toward Rigorous Assessment of… — ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
  18. Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicol… find
  19. Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Alijubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Kevin Liu, and Aleksander Madry. Introducing SWE-bench Verified, August 2024. URL https://openai. com/index/introducing-swe-benc… find
  20. DeepSeek-AI. DeepSeek-V3 Technical Report, 2024. 2412.19437. eprint: 2412.19437. find
  21. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  22. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a Generalist Agent for the Web. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 28091–28114.…
  23. Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian. TRAIL: Trace Reasoning and Agentic Issue Localization, June 2025. URL http://arxiv.org/abs/2505.08638. arXiv:2505.08638 [cs]. ↗
  24. GLM-4.5 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao D… — GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
  25. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. A Survey on LLM-as-a-Judge, November 2024. URL http://arxiv.org/abs/2411.15594. arXiv:2411.15594 [cs] version: 1. ↗
  26. Dong Guo, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, Jingji Chen, Jingjia Huang, Kang Lei, Liping Yuan, Lishu Luo, Pengfei Liu, Qinghao Ye, Rui Qian, Shen Yan, Shixiong Zhao, Shuai Peng, Shuangye Li, Sihang Yuan, Sijin Wu, Tianheng Cheng, Weiwei… — Seed1.5-VL Technical Report
  27. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep Reinforcement Learning That Matters. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018. ISSN 2374-3468. doi: 10.1609/aaai.v32i1.11694. URL https://ojs.aaai.org/index.php… ↗
  28. Marius Hobbhahn. SWE-Bench Verified Mini, 2025. URL https://huggingface.co/datasets/MariusHobbhahn/swe-bench-verified-mini. ↗
  29. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation, April 2024. URL http://arxiv.org/abs/2310.03302. arXiv:2310.03302 [cs]. ↗
  30. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, October 2023. URL https://arxiv.org/abs/2310.06770v1. ↗
  31. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That Matter. Transactions on Machine Learning Research, February 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=Zy4uFzMviZ. ↗
  32. Khimya Khetarpal, Zafarali Ahmed, Andre Cianflone, Riashat Islam, and Joelle Pineau. RE-EVALUATE: Reproducibility in Evaluating Reinforcement Learning Algorithms. June 2018. URL https://openreview.net/forum?id=HJgAmITcgm. ↗
  33. Shanghai AI Lab, Xiaoyang Chen, Yunhao Chen, Zeren Chen, Zhiyun Chen, Hanyun Cui, Yawen Duan, Jiaxuan Guo, Qi Guo, Xuhao Hu, Hong Huang, Lige Huang, Chunxiao Li, Juncheng Li, Qihao Lin, Dongrui Liu, Xinmin Liu, Zicheng Liu, Chaochao Lu, Xiaoya Lu, Jingjing Qu, Qibing Ren, Jing Shao, Jingwei Shi, Jin… — [arXiv:2507.16534](https://arxiv.org/abs/2507.16534) find
  34. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… find
  35. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as Age… — AgentBench: Evaluating LLMs as Agents
  36. Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents, December 2024. URL http://arxiv.org/abs/2401.13178. arXiv:2401.13178 [cs]. ↗
  37. Kevin Meng, Vincent Huang, Jacob Steinhardt, and Sarah Schwettmann. Introducing docent. https://transluce.org/introducing-docent, March 2025. ↗
  38. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for General AI Assistants, November 2023. URL http://arxiv.org/abs/2311.12983. arXiv:2311.12983 [cs]. ↗
  39. Magnus Müller and Gregor Žunič. Browser use: Enable ai to control your browser, 2024. URL https://github.com/browser-use/browser-use. ↗
  40. Noah MacCallum and Julian Lee. GPT-4.1 Prompting Guide \| OpenAI Cookbook, April 2025. URL https://cookbook.openai.com/examples/gpt4-1_prompting_guide. ↗
  41. OpenAI. ChatGPT agent: bridging research and action, September 2025a. //openai.com/index/introducing-chatgpt-agent/. find
  42. OpenAI. o3 and o4-mini, 2025b. introducing-o3-and-o4-mini/. find
  43. Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical Design in Reinforcement Learning, October 2024. URL http://arxiv.org/abs/2304.01315. arXiv:2304.01315 [cs]. — [arXiv:2304.01315](https://arxiv.org/abs/2304.01315) ↗
  44. URL https: URL https://openai.com/index/Jiahao Qiu, Xuan Qi, Tongcheng Zhang, Xinzhe Juan, Jiacheng Guo, Yifu Lu, Yimin Wang, Zixin Yao, Qihan Ren, Xun Jiang, Xing Zhou, Dongrui Liu, Ling Yang, Yue Wu, Kaixuan Huang, Shilong Liu, Hongru Wang, and Mengdi Wang. Alita: Generalist Agent Enabling Scalabl… — [arXiv:2505.20286](https://arxiv.org/abs/2505.20286) ↗
  45. Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kau-nismäki. ‘smolagents‘: a smol library to build great agentic systems. https://github.com/huggingface/smolagents, 2025a. ↗
  46. Aymeric Roucher, Albert Villanova del Moral, merve, Thomas Wolf, and Clémentine Fourrier. Open-source DeepResearch – Freeing our search agents, February 2025b. URL https://huggingface.co/blog/open-deep-research. ↗
  47. Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G. Parameswaran, and Eugene Wu. Do-cETL: Agentic Query Rewriting and Evaluation for Complex Document Processing, April 2025. URL http://arxiv.org/abs/2410.12189. arXiv:2410.12189 [cs]. — [arXiv:2410.12189](https://arxiv.org/abs/2410.12189) ↗
  48. Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can Language Models Solve Olympiad Programming?, April 2024. URL http://arxiv.org/abs/2404.10952. arXiv:2404.10952 [cs] version: 1. ↗
  49. Zachary S. Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark, September 2024. URL http://arxiv.org/abs/2409.11363. arXiv:2409.11363 [cs]. ↗
  50. Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind Control Suite, January 2018. URL http://arxiv.org/abs/1801.00690. arXiv:1801.00690 [cs]. ↗
  51. MiroMind AI Team. MiroFlow: A High-Performance Open-Source Research Agent Framework, 2025. URL https://github.com/MiroMindAI/MiroFlow. ↗
  52. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yuf… — SciCode: A Research Coding Benchmark Curated by Scientists
  53. UK AISI. Inspect Evals, 2025. URL https://inspect.ai-safety-institute.org.uk. ↗
  54. Wandb.ai. Wandb Weave, 2024. URL https://wandb.ai/site/weave. ↗
  55. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, a… — OpenHands: An Open Platform for AI Software Developers as Generalist Agents
  56. Zihan Wang, Jiaze Chen, Zhicheng Liu, Markus Mak, Yidi Du, Geonsik Moon, Luoqi Xu, Aaron Tua, Kunshuo Peng, Jiayi Lu, Mingfei Xia, Boqian Zou, Chenyang Ran, Guang Tian, Shoutai Zhu, Yeheng Duan, Zhenghui Kang, Zhenxing Lin, Shangshu Li, Qiang Luo, Qingshen Long, Zhiyong Chen, Yihan Xiao, Yurong Wu,… — [arXiv:2508.16402](https://arxiv.org/abs/2508.16402) find
  57. Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. AgentGym: Evolving Large Language Model-based Agents… — AgentGym: Evolving Large Language Model-based Agents across Diverse Environments
  58. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying LLM-based Software Engineering Agents, October 2024. URL http://arxiv.org/abs/2407.01489. arXiv:2407.01489 [cs]. ↗
  59. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real… — OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
  60. Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An Illusion of Progress? Assessing the Current State of Web Agents, May 2025. URL http://arxiv.org/abs/2504.01382. arXiv:2504.01382 [cs]. ↗
  61. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-AGENT: Agent-Computer Interfaces Enable Automated Software Engineering. 2024. URL https://swe-agent.com/paper.pdf. ↗
  62. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models, March 2023. URL http://arxiv.org/abs/2210.03629. arXiv:2210.03629 [cs]. ↗
  63. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. $\\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, June 2024. URL http://arxiv.org/abs/2406.12045. arXiv:2406.12045 [cs]. ↗
  64. Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?, July 2024. URL http://arxiv.org/abs/2407.15711. arXiv:2407.15711 [cs]. ↗
  65. Chengyue Yu, Siyuan Lu, Chenyi Zhuang, Dong Wang, Qintong Wu, Zongyue Li, Runsheng Gan, Chunfeng Wang, Siqi Hou, Gaochi Huang, Wenlong Yan, Lifeng Hong, Aohui Xue, Yanfeng Wang, Jinjie Gu, David Tsai, and Tao Lin. AWorld: Orchestrating the Training Recipe for Agentic AI, 2025. URL https://arxiv.org/… — [arXiv:2508.20404](https://arxiv.org/abs/2508.20404) ↗
  66. Andy K. Zhang, Neil Perry, Riya Dulepet, Eliot Jones, Justin W. Lin, Joey Ji, Celeste Menders, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Alluri, Nathan Tran, Rinnar… find
  67. Wentao Zhang, Liang Zeng, Yuzhen Xiao, Yongcong Li, Ce Cui, Yilei Zhao, Rui Hu, Yang Liu, Yahui Zhou, and Bo An. AgentOrchestra: A Hierarchical Multi-Agent Framework for General-Purpose Task Solving, August 2025. URL http://arxiv.org/abs/2506.12508. arXiv:2506.12508 [cs]. — [arXiv:2506.12508](https://arxiv.org/abs/2506.12508) ↗
  68. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) is a Generalist Web Agent, if Grounded, March 2024. URL http://arxiv.org/abs/2401.01614. arXiv:2401.01614 [cs]. ↗
  69. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents. January 2024. URL https://openreview.net/forum?id=oKn9c6ytLx. ↗
  70. Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellermann, Sarah S… find
  71. Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, Jet Geronimo, Avi Dhir, Sudhit Rao, Kaicheng Yu, Twm Stone, and Daniel Kang. CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulne… — [arXiv:2503.17332](https://arxiv.org/abs/2503.17332) find
  72. A PPENDIX Details on LLM use. We used LLMs to edit the text and for proofreading the paper’s contents, as well as for coding assistance. The authors take full responsibility for all text and results in the paper. A1 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kapoor-2025-holistic-agent-leaderboard-missing/