Elamur Research
☰
home / papers / 2023 / AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors

AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv non-exclusive distribution license" does not allow redistributing the text itself. Original — arxiv.org/pdf/2308.10848

Summary

Single LLM-driven agents such as AutoGPT can handle many tasks alone. Earlier multi-agent studies, however, were tied to narrow tasks and fixed agent roles. AgentVerse is a general framework, modelled on how human teams solve problems, in which a group of LLM agents works through repeated rounds, formalised as a Markov decision process. Each round has four stages:

  • a "recruiter" agent writes expert role descriptions suited to the goal;
  • the experts deliberate in a horizontal (democratic, decisions merged) or vertical (one solver, several reviewers) structure;
  • the agreed actions are executed in the environment;
  • an evaluator, human or agent, gives verbal feedback that reshapes the team in the next round.

The authors tested it zero-shot with GPT-3.5-Turbo and GPT-4 on conversation, creative writing, mathematical and logical reasoning, coding, tool use and Minecraft. The multi-agent setup usually beats CoT and single-agent baselines, though not in every case. The authors also describe emergent social behaviours among agents, some helpful and some harmful. These observations matter for both the design and the safety of multi-agent LLM systems.

Key points

  • Four-stage loop: Expert Recruitment → Collaborative Decision-Making → Action Execution → Evaluation. Feedback from the evaluation stage drives dynamic re-recruitment of the group.
  • Three settings were compared using GPT-3.5-Turbo-0613 and GPT-4-0613: CoT, Solo (AgentVerse with one decision-making agent) and Group.
  • Selected GPT-4 results (CoT / Solo / Group):
    • FED: 95.4 / 95.8 / 96.8
    • Commongen-Challenge: 95.9 / 99.0 / 99.1
    • Logic Grid Puzzles: 59.5 / 64.0 / 66.5
    • On MGSM, Group (95.2) did not beat Solo (96.0).
  • With GPT-3.5-Turbo, Group improved FED (85.1 vs. 81.6 CoT) but trailed Solo on Commongen-Challenge (92.3 vs. 93.6).
  • Three kinds of emergent behaviour appeared, especially in tool use and Minecraft (e.g., three agents crafting a bookshelf): volunteering to help peers, conformity after criticism, and occasional destructive actions. Code is available at github.com/OpenBMB/AgentVerse.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

184 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 177 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.10848 · arXiv 2308.10848

@misc{chen2023agentverse,
  title = {{AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors}},
  author = {Chen, Weize and Su, Yusheng and Zuo, Jingwei and Yang, Cheng and Yuan, Chenfei and Chan, Chi-Min and Yu, Heyang and Lu, Yaxi and Hung, Yi-Hsin and Qian, Chen and Qin, Yujia and Cong, Xin and Xie, Ruobing and Liu, Zhiyuan and Sun, Maosong and Zhou, Jie},
  year = {2023},
  doi = {10.48550/arxiv.2308.10848},
  eprint = {2308.10848},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.10848}
}

References

61 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J. Josh… find our card
  2. Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. Power to the people: The role of humans in interactive machine learning. AI Magazine, 35(4):105–120, Dec. 2014. doi: 10.1609/aimag.v35i4.2513. URL https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2513. — [doi:10.1609/aimag.v35i4.2513](https://doi.org/10.1609/aimag.v35i4.2513) ↗
  3. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… find our card
  4. J.D. Bransford and B.S. Stein. The Ideal Problem Solver: A Guide for Improving Thinking, Learning, and Creativity. W.H. Freeman, 1993. ISBN 978-0-7167-2205-2. URL https://books. google.com.tw/books?id=nnRxQgAACAAJ. find
  5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/23… find our card
  6. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate, 2023. URL https://doi.org/10.48550/arXiv.2308.07201. ↗ our card
  7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mi… find our card
  8. Robert B Cialdini and Noah J Goldstein. Social influence: Compliance and conformity. Annu. Rev. Psychol., 55:591–621, 2004. URL https://www.annualreviews.org/doi/abs/10.1146/annurev.psych.55.090902.142015. — [doi:10.1146/annurev.psych.55.090902.142015](https://doi.org/10.1146/annurev.psych.55.090902.142015) ↗
  9. Robert B Cialdini and Melanie R Trost. Social influence: Social norms, conformity and compliance. 1998. URL https://psycnet.apa.org/RECORD/1998-07091-021. ↗
  10. Jeff Clune. Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence. CoRR, abs/1905.10985, 2019. URL http://arxiv.org/abs/1905.10985. — [arXiv:1905.10985](https://arxiv.org/abs/1905.10985) ↗
  11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.1416… ↗ our card
  12. Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zen… find
  13. PMLR, 2023. URL https://proceedings. mlr.press/v202/driess23a.html. find
  14. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. CoRR, abs/2305.14325, 2023. doi: 10.48550/arXiv.2305.14325. URL https://doi.org/10.48550/arXiv.2305.14325. ↗ our card
  15. Ernst Fehr and Simon Gächter. Cooperation and punishment in public goods experiments. American Economic Review, 90(4):980–994, 2000. URL https://pubs.aeaweb.org/doi/pdf/10.1257/aer.90.4.980. — [doi:10.1257/aer.90.4.980](https://doi.org/10.1257/aer.90.4.980) ↗
  16. Ben Goertzel and Cassio Pennachin. Artificial general intelligence, volume 2. Springer, 2007. URL https://link.springer.com/book/10.1007/978-3-540-68677-4. — [doi:10.1007/978-3-540-68677-4](https://doi.org/10.1007/978-3-540-68677-4) ↗
  17. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: communicative agents for ”mind” exploration of large scale language model society. CoRR, abs/2303.17760, 2023. doi: 10.48550/arXiv.2303.17760. URL https://doi.org/10.48550/arXiv.2303.17760. ↗ our card
  18. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as age… find our card
  19. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. CoRR, abs… find
  20. Shikib Mehri and Maxine Eskénazi. Unsupervised evaluation of interactive dialog with dialogpt. In Olivier Pietquin, Smaranda Muresan, Vivian Chen, Casey Kennington, David Vandyke, Nina Dethlefs, Koji Inoue, Erik Ekstedt, and Stefan Ultes (eds.), Proceedings of the 21th Annual Meeting of the Special… find our card
  21. Marvin Minsky. The Society of Mind. Simon & Schuster, 1988. ISBN 0671657135. URL https: //jmvidal.cse.sc.edu/lib/minsky88a.html. find
  22. John C Mowen and Harish Sujan. Volunteer behavior: A hierarchical model approach for investigating its trait and functional motive antecedents. Journal of consumer psychology, 15(2):170– 182, 2005. URL https://myscp.onlinelibrary.wiley.com/doi/abs/10.1207/s15327663jcp1502\_9. — [doi:10.1207/s15327663jcp1502\_9](https://doi.org/10.1207/s15327663jcp1502_9) ↗
  23. Yohei Nakajima. Babyagi. 2023. URL https://github.com/yoheinakajima/babyagi. [Software]. ↗
  24. Allen M Omoto and Mark Snyder. Sustained helping without obligation: motivation, longevity of service, and perceived attitude change among aids volunteers. Journal of personality and social psychology, 68(4):671, 1995. URL https://psycnet.apa.org/record/1995-26640-001. ↗
  25. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023a. doi: 10.48550/arXiv.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774. ↗ our card
  26. OpenAI. Chatgpt can now see, hear, and speak, 2023b. URL https://openai.com/blog/chatgpt-can-now-see-hear-and-speak. ↗
  27. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. CoRR, abs/2304.03442, 2023. doi: 10.48550/arXiv.2304.03442. URL https://doi.org/10.48550/arXiv.2304.03442. ↗
  28. Katherine Phillips and Charles O’Reilly. Demography and diversity in organizations: A review of 40 years of research. Research in Organizational Behavior, 20:77–140, 01 1998. URL https://www.researchgate.net/publication/234022034\_Demography\_and\_Diversity\_in\_Organizations\_A\_Review\_of\_40\_Years\_of\_Rese… ↗
  29. Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. CoRR, abs/2307.07924, 2023a. doi: 10.48550/arXiv.2307.07924. URL https://doi.org/10.48550/arXiv.2307.07924. ↗ our card
  30. Cheng Qian, Chi Han, Yi R. Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. CREATOR: disentangling abstract and concrete reasonings of large language models through tool creation. CoRR, abs/2305.14318, 2023b. doi: 10.48550/arXiv.2305.14318. URL https://doi.org/10.48550/arXiv.2305.14318. — [doi:10.48550/arxiv.2305.14318](https://doi.org/10.48550/arxiv.2305.14318) ↗
  31. Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Z… find our card
  32. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023b. URL https://arxiv.org/abs/2307.16789. ↗ our card
  33. Reworkd. Agentgpt, 2023. URL https://github.com/reworkd/AgentGPT. [Software]. ↗
  34. Toran Bruce Richards and et al. Auto-gpt: An autonomous gpt-4 experiment, 2023. URL https: //github.com/Significant-Gravitas/Auto-GPT. [Software]. find
  35. Leonard Salewski, Stephan Alaniz, Isabel Rio-Torto, Eric Schulz, and Zeynep Akata. In-context impersonation reveals large language models’ strengths and biases. CoRR, abs/2305.14930, 2023. doi: 10.48550/arXiv.2305.14930. URL https://doi.org/10.48550/arXiv.2305.14930. — [doi:10.48550/arxiv.2305.14930](https://doi.org/10.48550/arxiv.2305.14930) ↗
  36. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. CoRR, abs/2302.04761, 2023a. doi: 10.48550/arXiv.2302.04761. URL https: //doi.org/10.48550/arXiv.2302.04… find our card
  37. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. CoRR, abs/2302.04761, 2023b. doi: 10.48550/arXiv.2302.04761. URL https: //doi.org/10.48550/arXiv.2302.04… find our card
  38. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations,… find our card
  39. OpenReview.net, 2023. URL https://openreview.net/pdf?id=fR3wGCk-IXp. ↗
  40. Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https: //doi.org/10.48550/arXiv.2303.11366. find our card
  41. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, A… find our card
  42. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neu… find
  43. Peter Stone and Manuela Veloso. Multiagent systems: A survey from a machine learning perspective. Auton. Robots, 8(3):345–383, jun 2000. ISSN 0929-5593. doi: 10.1023/A:1008942012299. URL https://doi.org/10.1023/A:1008942012299. — [doi:10.1023/a:1008942012299](https://doi.org/10.1023/a:1008942012299) ↗
  44. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2… find
  45. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  46. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. CoRR, abs/2305.16291, 2023a. doi: 10.48550/arXiv.2305.16291. URL https://doi.org/10.48550/arXiv.2305.16291. ↗
  47. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. CoRR, abs/2307.05300, 2023b. doi: 10.48550/arXiv.2307.05300. URL https: //doi.org/10.48550/arXiv.2307.0530… find our card
  48. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. find
  49. OpenReview.net, 2022a. URL https://openreview.net/forum?id=gEZrGCozdqR. ↗
  50. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022b. URL http://papers.nips.cc/paper\_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstr… ↗ our card
  51. Jimmy Wei, Kurt Shuster, Arthur Szlam, Jason Weston, Jack Urbanek, and Mojtaba Komeili. Multi-party chat: Conversational agents in group settings with humans and models. CoRR, abs/2304.13835, 2023. doi: 10.48550/arXiv.2304.13835. URL https://doi.org/10.48550/arXiv.2304.13835. — [doi:10.48550/arxiv.2304.13835](https://doi.org/10.48550/arxiv.2304.13835) ↗
  52. Michael J. Wooldridge and Nicholas R. Jennings. Intelligent agents: theory and practice. Knowl. Eng. Rev., 10(2):115–152, 1995. doi: 10.1017/S0269888900008122. URL https://doi.org/10.1017/S0269888900008122. — [doi:10.1017/s0269888900008122](https://doi.org/10.1017/s0269888900008122) ↗
  53. Anita Williams Woolley, Christopher F. Chabris, Alex Pentland, Nada Hashmi, and Thomas W. Malone. Evidence for a collective intelligence factor in the performance of human groups. Science, 330(6004):686–688, 2010. doi: 10.1126/science.1193147. URL https://www.science. org/doi/abs/10.1126/science.119… — [doi:10.1126/science.1193147](https://doi.org/10.1126/science.1193147) ↗
  54. Anita Williams Woolley, Ishani Aggarwal, and Thomas W. Malone. Collective intelligence and group performance. Current Directions in Psychological Science, 24(6):420–424, 2015. doi: 10.1177/0963721415599543. URL https://doi.org/10.1177/0963721415599543. — [doi:10.1177/0963721415599543](https://doi.org/10.1177/0963721415599543) ↗
  55. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation framework, 2023. URL https://doi.org/10.48550/arXiv.2308.08155. ↗ our card
  56. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. CoRR, abs/2305.10601, 2023a. doi: 10.48550/arXiv.2305.10601. URL https://doi.org/10.48550/arXiv.2305.10601. ↗
  57. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. find our card
  58. OpenReview.net, 2023b. URL https://openreview.net/pdf?id=WE\_vluYUL-X. ↗
  59. Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. CoRR, abs/2307.02485, 2023a. doi: 10.48550/arXiv.2307.02485. URL https: //doi.org/10.48550/arXiv.2307.02485. find
  60. Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper llm networks are fairer llm evaluators. arXiv preprint arXiv:2308.01862, 2023b. URL https://doi.org/10.48550/arXiv.2308.01862. ↗
  61. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. CoRR, abs/2307.13854, 2023. doi: 10.48550/arXiv.2307.13854. URL https://doi.org/10.4… find our card