Elamur Research⌕
☰
home / papers / 2026 / AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts

AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts

Paper · 2026 · Keyu Li, Junhao Shi, Yang Xiao et al.

Summary

Most agent benchmarks test one skill, such as tool use, coding, or web research, through short interactions. Realistic multi-step work also tends to need a human giving feedback along the way, which makes large-scale evaluation hard to automate. AgencyBench addresses both problems. Its tasks were collected from AI researchers, practitioners, and software developers and reflect everyday AI use. It contains 32 scenarios split into 138 tasks, and each task specifies a query, the expected deliverables, and grading rubrics. A typical scenario takes around 90 tool calls, about 1M tokens, and hours of runtime. To remove the human from the loop, a simulated user agent gives the working agent iterative feedback. Deliverables are then run in a Docker-based remote sandbox that mimics UI rendering and mouse or keyboard actions, and rule-based or LLM judges score the results against the rubrics. Proprietary models clearly beat open-source ones, yet none masters these long-horizon tasks. The benchmark therefore gives a scalable, demanding test of how useful agents are in practice.

Key points

  • Coverage: the benchmark spans 6 capabilities: game development, front-end, back-end, code generation, research, and MCP tool use. Tasks build on each other step by step, for example extending a Gomoku game over five successive queries. Judges score each task from 0 to 10.
  • Main results: closed-source models average 48.4% and open-source models 32.1%. Closed-source scores range from GPT-5.2 (56.5%) down to Grok-4.1-Fast (44.3%). Open-source scores range from GLM-4.6 (38.6%) down to Qwen-3-235B-A22B-Thinking (27.0%).
  • Behavioral differences:
    • GPT-5.2 is strongest at correcting itself after feedback.
    • Grok-4.1-Fast uses tokens most efficiently.
    • Claude-4.5-Opus prefers shell tools.
    • Gemini-3-Pro favors file and memory-management tools.
  • Scaffold sensitivity: the same model can score very differently depending on the agent framework it runs in.
    • Claude-4.5-Opus gains 20.5% inside Anthropic's own SDK.
    • GLM-4.6 gains 10.6% and Minimax-M2 gains 8.6% with the Claude-Agent-SDK.
    • Kimi-K2-Thinking loses 12.8% on the OpenAI-Agents-SDK.
  • Availability and scope: the benchmark is released on GitHub and HuggingFace. It covers only software agents working at a computer, not embodied or robotic settings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations🎓Hong Kong Polytechnic UniversityHong Kong · China84 works · 4,956 citations🎓Shanghai Innovation InstituteShanghai · China24 works · 262 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Keyu Li, Junhao Shi, Yang Xiao, Mohan Jiang, Jie Sun, Yunze Wu, Dayuan Fu, Shijie Xia, Xiaojie Cai, Tianze Xu, Weiye Si, Wenjie Li, Dequan Wang and Pengfei Liu. 2026. AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.337

@inproceedings{li2026agencybench,
  title = {{AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts}},
  author = {Keyu Li, Junhao Shi, Yang Xiao, Mohan Jiang, Jie Sun, Yunze Wu, Dayuan Fu, Shijie Xia, Xiaojie Cai, Tianze Xu, Weiye Si, Wenjie Li, Dequan Wang and Pengfei Liu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.337},
  url = {https://doi.org/10.18653/v1/2026.acl-long.337}
}

References

37 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pierre Andrews, Amine Benhalloum, Gerard Moreno-Torres Bertran, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Romain Froger, Emilien Garreau, Jean-Baptiste Gaya, and 1 others. 2025. Are: Scaling up agent environments and evaluations. arXiv preprint arXiv:2509.17158. — [arXiv:2509.17158](https://arxiv.org/abs/2509.17158) find
  2. Anthropic. 2025a. claude-4.5. https://www.anthropic.com/news/claude-opus-4-5. ↗
  3. Anthropic. 2025b. Claude-agent-sdk-python. https://github.com/anthropics/ claude-agent-sdk-python. ↗
  4. Anthropic. 2025c. claude code. https://claude.com/product/claude-code. ↗
  5. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. τ 2 -bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982. find
  6. Cursor. 2025. cursor-cli. https://cursor.com/cn/ cli. ↗
  7. DeepMind. 2025. gemini-3. https://deepmind.google/models/gemini/pro/. ↗
  8. DesignArena Team. 2025. Designarena. https://www.designarena.ai/. ↗
  9. GLM Team. 2025. glm-4.6. https://x.ai/news/ grok-4-1. ↗
  10. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.
  11. Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, and 1 others. 2025. Measuring ai ability to complete long tasks. arXiv preprint arXiv:2503.14499. find
  12. Junlong Li, Wenshuo Zhao, Jian Zhao, Weihao Zeng, Haoze Wu, Xiaochen Wang, Rui Ge, Yuxuan Cao, Yuzhen Huang, Wei Liu, and 1 others. 2025a. The tool decathlon: Benchmarking language agents for diverse, realistic, and long-horizon task execution. arXiv preprint arXiv:2510.25726.
  13. Keyu Li, Jin Gao, and Dequan Wang. Aligned agents, biased swarm: Measuring bias amplification in multi-agent systems. In The Fourteenth International Conference on Learning Representations. find
  14. Keyu Li, Mohan Jiang, Dayuan Fu, Yunze Wu, Xiangkun Hu, Dequan Wang, and Pengfei Liu. 2025b. Datasetresearch: Benchmarking agent systems for demand-driven dataset discovery. arXiv preprint arXiv:2508.06960. — [arXiv:2508.06960](https://arxiv.org/abs/2508.06960) find
  15. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
  16. Haotian Luo, Huaisong Zhang, Xuelin Zhang, Haoyu Wang, Zeyu Qin, Wenjie Lu, Guozheng Ma, Haiying He, Yingsha Xie, Qiyang Zhou, and 1 others. 2025. Ultrahorizon: Benchmarking agent capabilities in ultra long-horizon scenarios. arXiv preprint arXiv:2509.21766. — [arXiv:2509.21766](https://arxiv.org/abs/2509.21766) find
  17. Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. Swe-lancer: Can frontier llms earn 1 million from real-world freelance software engineering?, 2025. URL https://arxiv.org/abs/2502.12115. ↗
  18. OpenAI. 2025a. codex. zh-Hans-CN/codex/. https://openai.com/ ↗
  19. OpenAI. 2025b. gpt-5.2. https://cdn.openai.com/ pdf/3a4153c8-c748-4b71-8e31-aecbde944f8d/ oai_5_2_system-card.pdf. ↗
  20. OpenAI. 2025c. Openai-agents-sdk. https://github.com/openai/openai-agents-python. ↗
  21. OpenRouter. 2025. State of ai: An empirical 100 trillion token study with openrouter. https://openrouter.ai/state-of-ai. ↗
  22. Melissa Z Pan, Negar Arabzadeh, Riccardo Cogo, Yuxuan Zhu, Alexander Xiong, Lakshya A Agrawal, Huanzhi Mao, Emma Shen, Sid Pallerla, Liana Patel, and 1 others. 2025. Measuring agents in production. arXiv preprint arXiv:2512.04123. — [arXiv:2512.04123](https://arxiv.org/abs/2512.04123) find
  23. Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, and 1 others. 2025. Gdpval: Evaluating ai model performance on real-world economically valuable tasks. arXiv preprint arXiv:2510.04374. find
  24. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, and 1 others. 2025. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534.
  25. Terminal-bench Team. 2025. terminal-bench. https://www.tbench.ai/. ↗
  26. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, and 1 others. 2024. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741.
  27. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516. find
  28. Yunze Wu, Dayuan Fu, Weiye Si, Zhen Huang, Mohan Jiang, Keyu Li, Shijie Xia, Jie Sun, Tianze Xu, Xiangkun Hu, and 1 others. 2025a. Innovatorbench: Evaluating agents’ ability to conduct innovative llm research. arXiv preprint arXiv:2510.27598. find
  29. Zijian Wu, Xiangyan Liu, Xinyuan Zhang, Lingjun Chen, Fanqing Meng, Lingxiao Du, Yiran Zhao, Fanshi Zhang, Yaoqi Ye, Jiawei Wang, and 1 others. 2025b. Mcpmark: A benchmark for stress-testing realistic and comprehensive mcp use. arXiv preprint arXiv:2509.24002. xAI. 2025. grok-4.1. https://x.ai/news/… — [arXiv:2509.24002](https://arxiv.org/abs/2509.24002) ↗
  30. Yang Xiao, Mohan Jiang, Jie Sun, Keyu Li, Jifan Lin, Yumin Zhuang, Ji Zeng, Shijie Xia, Qishuo Hua, Xuefeng Li, and 1 others. 2025a. Limi: Less is more for agency. arXiv preprint arXiv:2509.17567. — [arXiv:2509.17567](https://arxiv.org/abs/2509.17567) find
  31. Yang Xiao, Jiashuo Wang, Qiancheng Xu, Changhe Song, Chunpu Xu, Yi Cheng, Wenjie Li, and Pengfei Liu. 2025b. Towards dynamic theory of mind: Evaluating llm adaptation to temporal evolution of human states. arXiv preprint arXiv:2505.17663.
  32. Yang Xiao, Jiashuo Wang, Ruifeng Yuan, Chunpu Xu, Kaishuai Xu, Wenjie Li, and Pengfei Liu. 2025c. Limopro: Reasoning refinement for efficient and effective test-time scaling. arXiv preprint arXiv:2505.19187. — [arXiv:2505.19187](https://arxiv.org/abs/2505.19187) find
  33. Yang Xiao, Chunpu Xu, Ruifeng Yuan, Jiashuo Wang, Wenjie Li, and Pengfei Liu. 2025d. Scale: Selective resource allocation for overcoming performance bottlenecks in mathematical test-time scaling. arXiv preprint arXiv:2512.00466. — [arXiv:2512.00466](https://arxiv.org/abs/2512.00466) find
  34. Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. 2025. Researcherbench: Evaluating deep ai research systems on the frontiers of scientific inquiry. arXiv preprint arXiv:2507.16280. — [arXiv:2507.16280](https://arxiv.org/abs/2507.16280) find
  35. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  36. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528– 50652.
  37. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2026-agencybench-benchmarking-frontiers-autonomous-agents/