Elamur Research⌕
☰
home / papers / 2024 / TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

Paper · 2024 · Frank F. Xu, Yufan Song, Boxuan Li et al.

Summary

Can LLM-driven agents actually take over real office work, and how much of it? Existing agent benchmarks are often detached from professional practice or limited to one job type, and they rarely test communication with coworkers. TheAgentCompany addresses this by simulating a small software firm inside a fully self-hosted, resettable environment built from open-source tools and filled with realistic mock data. Agents operate from a sandboxed Docker workspace with a browser, code editor and terminal. They must complete 175 tasks spanning engineering, project management, HR, finance, data science and administration, sometimes by chatting with LLM-simulated colleagues. Scoring is checkpoint-based, so partially completed tasks earn partial credit. The authors tested twelve model backbones, both proprietary and open-weights, mainly within the OpenHands framework. The strongest, Gemini 2.5 Pro, fully finished about 30% of tasks. Agents can handle many simpler, well-scoped duties. Long multi-step work, social interaction and complex web UIs remain difficult. These findings offer useful evidence for both AI adoption decisions and labor-market policy.

Key points

  • The intranet runs on self-hostable open-source services: GitLab (code and wiki), OwnCloud (documents), Plane (issues and sprints) and RocketChat (messaging). A pre-packaged VM image is also provided for reproducibility.
  • Simulated coworkers are built on the Sotopia platform, each with a profile covering name, role and responsibilities. They are powered by Claude-3-5-Sonnet-20241022.
  • Gemini 2.5 Pro performed best, with 30.3% full task completion and a 39.3% partial-credit score. Baselines used OpenHands, and OWL RolePlay served as a second agent scaffold.
  • Coding-oriented tasks proved easier than administrative and financial ones. The authors attribute this to abundant public software data and benchmarks, while business data typically stays private.
  • Recurring failures include:
    • weak social follow-up, such as not contacting the colleague the agent was referred to;
    • getting stuck on web UI elements, such as popups;
    • "self-deceiving" shortcuts, such as renaming a user to impersonate the intended contact.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,225 citations🏢MicrosoftRedmond · United States378 works · 161,630 citations🏢MetaMenlo Park · United States214 works · 241,492 citations🏢Alibaba GroupHangzhou · China201 works · 17,172 citations🎓University of Hong KongPok Fu Lam · China118 works · 13,496 citations🏢Meta (United Kingdom)London · United Kingdom54 works · 19,348 citations🏢OracleAustin · United States9 works · 1,235 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou and Graham Neubig. 2024. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv:2412.14161.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.14161 · arXiv 2412.14161

@misc{xu2024theagentcompany,
  title = {{TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks}},
  author = {Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou and Graham Neubig},
  year = {2024},
  doi = {10.48550/arxiv.2412.14161},
  eprint = {2412.14161},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.14161}
}

References

39 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amodei, D. and Fridman, L. Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity \| Lex Fridman Podcast \#452, November 2024. URL https://lexfridman.com/ dario-amodei-transcript. ↗
  2. Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2023. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf. ↗
  3. Bonatti, R., Zhao, D., Bonacci, F., Dupont, D., Abdali, S., Li, Y., Lu, Y., Wagle, J., Koishida, K., Bucker, A., et al. Windows agent arena: Evaluating multi-modal os agents at scale. arXiv preprint arXiv:2409.08264, 2024. — [arXiv:2409.08264](https://arxiv.org/abs/2409.08264) find
  4. Chezelles, D., Le Sellier, T., Gasse, M., Lacoste, A., Drouin, A., Caccia, M., Boisvert, L., Thakkar, M., Marty, T., Assouel, R., et al. The browsergym ecosystem for web agent research. arXiv preprint arXiv:2412.05467, 2024. find
  5. Chollet, F., Knoop, M., Kamradt, G., and Landers, B. ARC Prize 2024: Technical Report, December 2024. URL https://arcprize.org/media/arc-prize-2024-technical-report.pdf. ↗
  6. Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards a generalist agent for the web. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum? id=kiYqbO3wqw. ↗
  7. Drouin, A., Gasse, M., Caccia, M., Laradji, I. H., Verme, M. D., Marty, T., Boisvert, L., Thakkar, M., Cappart, Q., Vazquez, D., Chapados, N., and Lacoste, A. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024. find
  8. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  9. Eloundou, T., Manning, S., Mishkin, P., and Rock, D. GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models, 2023. find
  10. He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., and Yu, D. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024.
  11. Hu, M., Zhou, Y., Fan, W., Nie, Y., Xia, B., Sun, T., Ye, Z., Jin, Z., Li, Y., Zhang, Z., Wang, Y., Ye, Q., Luo, P., and Li, G. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025. URL https://github.com/camel-ai/owl. ↗
  12. Huang, K.-H., Prabhakar, A., Dhawan, S., Mao, Y., Wang, H., Savarese, S., Xiong, C., Laban, P., and Wu, C.-S. Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments. arXiv preprint arXiv:2411.02305, 2024.
  13. Intelligence, A. A. G. The amazon nova family of models: Technical report and model card. Amazon Technical Reports, 2024. URL https://www.amazon.science/publications/ the-amazon-nova-family-of-models-technical-report-and-model-card. ↗
  14. IPython. Jupyter and the future of IPython — IPython. URL https://ipython.org. Jang, L., Li, Y., Ding, C., Lin, J., Liang, P. P., Zhao, D., Bonatti, R., and Koishida, K. Videowebarena: Evaluating long context multimodal agents with video understanding web tasks, 2024. URL https://arxiv.org/abs/2410.… — [arXiv:2410.19100](https://arxiv.org/abs/2410.19100) ↗
  15. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE-bench: Can Language Models Resolve Real-world Github Issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. ↗
  16. Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L., and Murthy, A. Llms can’t plan, but can help planning in llm-modulo frameworks. arXiv preprint arXiv:2402.01817, 2024.
  17. Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo…
  18. Li, B., Wu, W., Tang, Z., Shi, L., Yang, J., Li, J., Yao, S., Qian, C., Hui, B., Zhang, Q., et al. Devbench: A comprehensive benchmark for software development. arXiv preprint arXiv:2403.08604, 2024. — [arXiv:2403.08604](https://arxiv.org/abs/2403.08604) find
  19. Liu, E. Z., Guu, K., Pasupat, P., Shi, T., and Liang, P. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations (ICLR), 2018. URL https://arxiv.org/abs/1802.08802. ↗
  20. Lù, X. H., Kasner, Z., and Reddy, S. Weblinx: Real-world website navigation with multi-turn dialogue, 2024. URL https://arxiv.org/abs/2402.05930. ↗
  21. Mozilla. Accessibility tree - MDN Web Docs Glossary: Definitions of Web-related terms \| MDN. URL https://developer.mozilla.org/en-US/docs/Glossary/Accessibility_tree. O*NET. The 29.1 release of the O*NET database, November 2024. URL https://www.onetcenter. org/dictionary/29.1/excel/. ↗
  22. OpenAI. Introducing gpt-4o and more tools to chatgpt free users, 2024. URL https://openai.com/ index/gpt-4o-and-more-tools-to-chatgpt-free/. ↗
  23. Park, J. S., O’Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior, 2023. find
  24. Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023.
  25. Playwright. Fast and reliable end-to-end testing for modern web apps \| Playwright. URL https://playwright.dev/. Rounds, J., Smith, T., Hubert, L., Lewis, P., and Rivkin, D. Development of occupational interest profiles for o* net. Raleigh, NC: National Center for O* NET Development, 1999. ↗
  26. ServiceNow. BrowserGym: a Gym Environment for Web Task Automation. URL https://github.com/ ServiceNow/BrowserGym. Song, Y., Xu, F., Zhou, S., and Neubig, G. Beyond browsing: Api-based web agents. arXiv preprint arXiv:2410.16464, 2024. ↗
  27. Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  28. Trivedi, H., Khot, T., Hartmann, M., Manku, R., Dong, V., Li, E., Gupta, S., Sabharwal, A., and Balasubramanian, N. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics…
  29. Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., and Ji, H. Executable Code Actions Elicit Better LLM Agents. In ICML, 2024a.
  30. Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., Tran, H. H., Li, F., Ma, R., Zheng, M., Qian, B., Shao, Y., Muennighoff, N., Zhang, Y., Hui, B., Lin, J., Brennan, R., Peng, H., Ji, H., and Neubig, G. Openhands: An open platform for ai software develo… — OpenHands: An Open Platform for AI Software Developers as Generalist Agents
  31. Wittenstein, J. AI Can Only Do 5% of Jobs, Says MIT Economist Who Fears Crash, October 2024. URL https://www.bloomberg.com/news/articles/2024-10-02/ ai-can-only-do-5-of-jobs-says-mit-economist-who-fears-crash. ↗
  32. Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., and Yu, T. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In The Thirty-eight Conference on N…
  33. Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. find
  34. Yang, J., Zhang, H., Li, F., Zou, X., Li, C., and Gao, J. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023.
  35. Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024.
  36. Yoran, O., Amouyal, S. J., Malaviya, C., Bogin, B., Press, O., and Berant, J. Assistantbench: Can web agents solve realistic and time-consuming tasks?, 2024. URL https://arxiv.org/abs/2407.15711. ↗
  37. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
  38. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2023.
  39. Zhou, X., Zhu, H., Mathur, L., Zhang, R., Yu, H., Qi, Z., Morency, L.-P., Bisk, Y., Fried, D., Neubig, G., and Sap, M. SOTOPIA: Interactive evaluation for social intelligence in language agents. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/for… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2024-theagentcompany-benchmarking-llm-agents/