Elamur Research⌕
☰
home / papers / 2024 / AI Agents That Matter

AI Agents That Matter

Paper · 2024 · Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel et al.

Summary

Benchmarks steer the development of LLM-based agents. The Princeton authors argue that current evaluation habits reward topping leaderboards rather than building systems that work in practice. They identify four problems:

  • Evaluations look only at accuracy and ignore inference cost.
  • The needs of model developers and downstream builders are mixed up.
  • Many benchmarks lack suitable held-out data, which invites overfitting and shortcuts.
  • Procedures are not standardized, which undermines reproducibility.

To support these claims, they re-run published coding agents on HumanEval next to inexpensive baselines and plot everything on an accuracy–cost Pareto frontier. They also adapt DSPy to optimize cost and accuracy jointly on HotPotQA, and present case studies on NovelQA and WebArena. The cheap baselines match or beat elaborate agent designs for far less money. This leaves open whether reflection, planning or debugging actually cause the reported gains. The work matters because it gives practical guidance for telling real progress from hype: cost-aware comparison, holdouts matched to the intended generality, and shared evaluation tooling.

Key points

  • Three baselines were tested on all 164 HumanEval tasks: retry (up to five calls at temperature 0), warming (temperature raised from 0 to 0.5) and escalation (Llama-3 8B → GPT-3.5 → Llama-3 70B → GPT-4). Warming shows no significant accuracy gap to the best complex agent among LDB, LATS and Reflexion.
  • At similar accuracy, costs differ by nearly 100×. LATS costs over 50 times as much as warming, and Reflexion and LDB cost more than 1.5 times as much. Escalation is more accurate than LDB (GPT-3.5) at under half its cost.
  • The authors define four levels of agent generality and argue that each requires a different type of hold-out set. WebArena illustrates how shortcuts arise when holdouts are missing.
  • Reproducibility problems inflate reported accuracy:
    • LATS and STeP counted some failed tasks as successes and dropped 1 and 8 tasks, respectively.
    • Rate limits on WebArena's Reddit clone make task order matter.
  • Evaluation is expensive: one full SWE-Agent run on SWE-bench (more than 2,000 tasks, capped at USD 4 each) could exceed USD 8,000. As a result, error bars are rarely reported.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Princeton UniversityPrinceton · United States113 works · 48,612 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir and Arvind Narayanan. 2024. AI Agents That Matter. arXiv:2407.01502.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.01502 · arXiv 2407.01502

@misc{kapoor2024ai,
  title = {{AI Agents That Matter}},
  author = {Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir and Arvind Narayanan},
  year = {2024},
  doi = {10.48550/arxiv.2407.01502},
  eprint = {2407.01502},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.01502}
}

References

66 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohamed Aghzal, Erion Plaku, and Ziyu Yao. Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoning, February 2024. URL http://arxiv.org/abs/2310.03249. arXiv:2310.03249 [cs]. ↗
  2. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, pages 2623–2631, New York, NY, USA, July 2019…
  3. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan… find
  4. Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, Michelle Lin, Alex Mayhew, Katherine Collins, Maryam Molamohammadi, John Burden, Wanru Zhao, Shalaleh Rismani, Konstantinos Voudouris, Umang Bh… — [doi:10.1145/3593013.3594033](https://doi.org/10.1145/3593013.3594033) find
  5. Harrison Chase. What is an agent? https://blog.langchain.dev/what-is-an-agent/, 2024. LangChain blog, Jun 28, 2024. ↗
  6. Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems, March 2024. URL http://arxiv.org/abs/2403.02419. arXiv:2403.02419 [cs, eess]. ↗
  7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  8. Ching-An Cheng, Andrey Kolobov, Dipendra Misra, Allen Nie, and Adith Swaminathan. LLF-Bench: Benchmark for Interactive Learning from Language Feedback, December 2023. URL http://arxiv.org/abs/2312.06853. arXiv:2312.06853 [cs]. — [arXiv:2312.06853](https://arxiv.org/abs/2312.06853) ↗
  9. Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning, December 2019. URL http://arxiv.org/abs/1810.08272. arXiv:1810.08272 [cs]. ↗
  10. François Chollet. On the Measure of Intelligence, November 2019. URL http://arxiv.org/abs/1911.01547. arXiv:1911.01547 [cs]. ↗
  11. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. MIND2WEB: Towards a Generalist Agent for the Web.
  12. David Donoho. 50 Years of Data Science. Journal of Computational and Graphical Statistics, 26(4):745–766, October 2017. ISSN 1061-8600. doi: 10.1080/10618600.2017.1384734. URL https://doi.org/10.1080/10618600.2017.1384734. Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/10618600.2017.13… — [doi:10.1080/10618600.2017.1384734](https://doi.org/10.1080/10618600.2017.1384734) ↗
  13. Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?, April 2024. URL http://ar… — [arXiv:2403.07718](https://arxiv.org/abs/2403.07718) find
  14. Iason Gabriel, Arianna Manzini, Geoff Keeling, Lisa Anne Hendricks, Verena Rieser, Hasan Iqbal, Nenad Tomašev, Ira Ktena, Zachary Kenton, Mikel Rodriguez, Seliem El-Sayed, Sasha Brown, Canfer Akbulut, Andrew Trask, Edward Hughes, A. Stevie Bergman, Renee Shelby, Nahema Marchal, Conor Griffin, Juan M… — [arXiv:2404.16244](https://arxiv.org/abs/2404.16244) find
  15. Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. Interactive Fiction Games: A Colossal Adventure. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7903–7910, April 2020. ISSN 2374-3468. doi: 10.1609/aaai.v34i05.6297. URL https://ojs.aaai.org/inde… — [doi:10.1609/aaai.v34i05.6297](https://doi.org/10.1609/aaai.v34i05.6297) ↗
  16. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep Reinforcement Learning That Matters. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018. ISSN 2374-3468. doi: 10.1609/aaai.v32i1.11694. URL https://ojs.aaai.org/index.php/… ↗
  17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding, January 2021. URL http://arxiv.org/abs/2009.03300. arXiv:2009.03300 [cs]. ↗
  18. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large Language Models Cannot Self-Correct Reasoning Yet. October 2023. URL https://openreview.net/forum?id=IkmD3fKBPQ. ↗
  19. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation, April 2024. URL http://arxiv.org/abs/2310.03302. arXiv:2310.03302 [cs]. ↗
  20. Lujain Ibrahim, Saffron Huang, Lama Ahmad, and Markus Anderljung. Beyond static AI evaluations: advancing human interaction evaluations for LLM harms and risks, May 2024. URL http://arxiv.org/abs/2405.10632. arXiv:2405.10632 [cs]. — [arXiv:2405.10632](https://arxiv.org/abs/2405.10632) ↗
  21. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, October 2023. URL https://arxiv.org/abs/2310.06770v1. ↗
  22. Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks, February 2024. URL http://arxiv.org/abs/2402.01817. arXiv:2402.01817 [cs]. ↗
  23. Omar Khattab, Christopher Potts, and Matei Zaharia. Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval. In Advances in Neural Information Processing Systems, volume 34, pages 27670–27682. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/hash/e… ↗
  24. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines, October…
  25. Nathan Lambert. The end of the “best open LLM”, April 2024. URL https://www.interconnects.ai/p/compute-efficient-open-llms. ↗
  26. Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More Agents Is All You Need, February 2024. URL http://arxiv.org/abs/2402.05120. arXiv:2402.05120 [cs]. ↗
  27. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James M… — Competition-Level Code Generation with AlphaCode
  28. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… find
  29. Zachary C. Lipton and Jacob Steinhardt. Troubling Trends in Machine Learning Scholarship, July 2018. URL http://arxiv.org/abs/1807.03341. arXiv:1807.03341 [cs, stat]. ↗
  30. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as Age… — AgentBench: Evaluating LLMs as Agents
  31. Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents, January 2024. URL http://arxiv.org/abs/2401.13178. arXiv:2401.13178 [cs]. ↗
  32. Timothy R. McIntosh, Teo Susnjak, Tong Liu, Paul Watters, and Malka N. Halgamuge. Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence, February 2024. URL http://arxiv.org/abs/2402.09880. arXiv:2402.09880 [cs]. — [arXiv:2402.09880](https://arxiv.org/abs/2402.09880) ↗
  33. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for General AI Assistants, November 2023. URL http://arxiv.org/abs/2311.12983. arXiv:2311.12983 [cs]. ↗
  34. Mistral AI team. Cheaper, Better, Faster, Stronger, April 2024. URL https://mistral.ai/news/mixtral-8x22b/. Section: news. ↗
  35. Meredith Ringel Morris, Jascha Sohl-dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg. Levels of AGI for Operationalizing Progress on the Path to AGI, June 2024. URL http://arxiv.org/abs/2311.02462. arXiv:2311.02462 [cs]. — [arXiv:2311.02462](https://arxiv.org/abs/2311.02462) ↗
  36. Silen Naihin. agbenchmark: Benchmarking the performance of agents far and wide, regardless of how they are set up and how they work. find
  37. National Academies of Sciences, Engineering, and Medicine. Reproducibility and Replicability in Science. 2019. doi: 10.17226/25303. URL https://nap.nationalacademies.org/read/25303/chapter/7. — [doi:10.17226/25303](https://doi.org/10.17226/25303) ↗
  38. Andrew Ng. Welcoming Diverse Approaches Keeps Machine Learning Strong, June 2024. URL https://www.deeplearning.ai/the-batch/welcoming-diverse-approaches-keeps-machine-learning-strong/. ↗
  39. Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. Identifying the Risks of LM Agents with an LM-Emulated Sandbox, May 2024. URL http://arxiv.org/abs/2309.15817. arXiv:2309.15817 [cs]. ↗
  40. Stuart Jonathan Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall, 1 edition, 1995. ISBN 978-0-13-103805-9. Google-Books-ID: CUVeMwAACAAJ. find
  41. Yonadav Shavit, Cullen O’Keefe, Tyna Eloundou, Paul McMillan, Sandhini Agarwal, Miles Brundage, Steven Adler, Rosie Campbell, Teddy Lee, Pamela Mishkin, Alan Hickey, Katarina Slama, Lama Ahmad, Alex Beutel, Alexandre Passos, and David G Robinson. Practices for Governing Agentic AI Systems. December… find
  42. Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. TaskBench: Benchmarking Large Language Models for Task Automation, December 2023. URL http://arxiv.org/abs/2311.18760. arXiv:2311.18760 [cs]. — [arXiv:2311.18760](https://arxiv.org/abs/2311.18760) ↗
  43. Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can Language Models Solve Olympiad Programming?, April 2024. URL http://arxiv.org/abs/2404.10952. arXiv:2404.10952 [cs] version: 1. ↗
  44. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning, October 2023. URL http://arxiv.org/abs/2303.11366. arXiv:2303.11366 [cs]. ↗
  45. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning, March 2021. URL http://arxiv.org/abs/2010.03768. arXiv:2010.03768 [cs]. ↗
  46. Simons Foundation. Mark Liberman - Reproducible Research and the Common Task Method (April 1, 2015), July 2019. URL https://www.simonsfoundation.org/event/reproducible-research-and-the-common-task-method/. ↗
  47. Paloma Sodhi, S. R. K. Branavan, Yoav Artzi, and Ryan McDonald. SteP: Stacked LLM Policies for Web Actions, April 2024. URL http://arxiv.org/abs/2310.03720. arXiv:2310.03720 [cs]. — [arXiv:2310.03720](https://arxiv.org/abs/2310.03720) ↗
  48. Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision, March 2024. URL http://arxiv.org/abs/2403.09472. arXiv:2403.09472 [cs]. ↗
  49. OpenDevin Team. OpenDevin: An Open Platform for AI Software Developers as Generalist Agents, June 2024. URL https://github.com/OpenDevin/OpenDevin. original-date: 2024-03-13T03:33:31Z. ↗
  50. Mudit Verma, Siddhant Bhambri, and Subbarao Kambhampati. On the Brittle Foundations of ReAct Prompting for Agentic Large Language Models, May 2024. URL http://arxiv.org/abs/2405.13966. arXiv:2405.13966 [cs]. Trading Off Compute in Train- — [arXiv:2405.13966](https://arxiv.org/abs/2405.13966) ↗
  51. Pablo Villalobos and David Atkinson. ing and Inference, July 2023. URL https://epochai.org/blog/trading-off-compute-in-training-and-inference. ↗
  52. Kiri Wagstaff. Machine Learning that Matters, June 2012. URL http://arxiv.org/abs/1206.4656. arXiv:1206.4656 [cs, stat]. — [arXiv:1206.4656](https://arxiv.org/abs/1206.4656) ↗
  53. Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Qian Wang, and Yue Zhang. NovelQA: A Benchmark for Long-Range Novel Question Answering, March 2024. URL http://arxiv.org/abs/2403.12766. arXiv:2403.12766 [cs]. ↗
  54. Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. ScienceWorld: Is your Agent Smarter than a 5th Grader?, November 2022. URL http://arxiv.org/abs/2203.07540. arXiv:2203.07540 [cs]. ↗
  55. Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback, March 2024. URL http://arxiv.org/abs/2309.10691. arXiv:2309.10691 [cs]. ↗
  56. Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models, June 2024. URL http://arxiv.org/abs/2406.16838. arXiv:2406.16838 [cs]. ↗
  57. Lilian Weng. LLM Powered Autonomous Agents, June 2023. URL https://lilianweng.github.io/posts/2023-06-23-agent/. Section: posts. ↗
  58. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. TravelPlanner: A Benchmark for Real-World Planning with Language Agents, February 2024. URL http://arxiv.org/abs/2402.01622. arXiv:2402.01622 [cs]. ↗
  59. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-AGENT: Agent-Computer Interfaces Enable Automated Software Engineering. 2024. URL https://swe-agent.com/paper.pdf. ↗
  60. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering, September 2018. URL http://arxiv.org/abs/1809.09600. arXiv:1809.09600 [cs]. ↗
  61. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents, February 2023. URL http://arxiv.org/abs/2207.01206. arXiv:2207.01206 [cs]. ↗
  62. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. $\\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, June 2024. URL http://arxiv.org/abs/2406.12045. arXiv:2406.12045 [cs]. ↗
  63. Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The Shift from Models to Compound AI Systems, February 2024. URL http://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/. ↗
  64. Lily Zhong, Zilong Wang, and Jingbo Shang. LDB: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step, April 2024. URL http://arxiv.org/abs/2402.16906. arXiv:2402.16906 [cs]. ↗
  65. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models, December 2023. URL http://arxiv.org/abs/2310.04406. arXiv:2310.04406 [cs]. ↗
  66. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents, April 2024. URL http://arxiv.org/abs/2307.13854. arXiv:2307.13854 [cs]. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kapoor-2024-ai-agents-that-matter/