Elamur Research⌕
☰
home / papers / 2025 / MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents

MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents

Paper · 2025 · Kunlun Zhu, Hongyi Du, Zhaochen Hong et al.

Summary

Most existing agent benchmarks test a single LLM agent or a narrow domain. They therefore miss what happens when several agents must cooperate or compete. MultiAgentBench fills this gap with six interactive scenarios: jointly writing research papers, constructing structures in Minecraft, diagnosing faults in databases, working together on software, playing Werewolf and haggling in bargaining. It runs on MARBLE (Multi-agent cooRdination Backbone with LLM Engine). In MARBLE, agents are nodes in a relation graph, and they communicate only along defined links such as collaboration, supervision or negotiation. A Coordination Engine separates planners from actors, and a Cognitive Module tracks personas, relationships and reasoning strategies.

Evaluation goes beyond whether a task is finished. It adds milestone-based KPIs, scores for planning and communication quality, and a separate competition score. Across models, gpt-4o-mini achieved the best average task score. Among coordination structures, graph-mesh worked best in the research scenario. A self-evolving cognitive planner raised milestone achievement by about 3%. Agents also showed emergent social strategies. The benchmark offers a reusable testbed for designing and comparing multi-agent LLM systems.

Key points

  • Coordination protocols: four topologies were compared. Two are centralized (star, tree) and two are decentralized (graph-mesh, chain).
  • Planning strategies: four planner approaches were tested: vanilla prompting, chain-of-thought, group discussion, and a Reflexion-like cognitive self-evolving planner. The last one improved milestone achievement by roughly 3%.
  • Team size ablation: teams of 1, 3, 5 and 7 agents were tested on 20 papers with at least seven main authors.
    • The overall KPI declined as teams grew.
    • The coordination score rose sharply when going from 1 to 3 agents.
  • Emergent behaviors: three patterns appeared, mainly in Werewolf and bargaining.
    • Strategic information sharing.
    • Trust-polarized collaboration.
    • Role-driven strategy shifts.
  • Availability and limitations: code and data are released at github.com/MultiagentBench/MARBLE. The authors acknowledge limited model coverage (e.g., no DeepSeek models) and mostly well-defined task goals.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji and Jiaxuan You. 2025. MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents. arXiv:2503.01935.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.01935 · arXiv 2503.01935

@misc{zhu2025multiagentbench,
  title = {{MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents}},
  author = {Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji and Jiaxuan You},
  year = {2025},
  doi = {10.48550/arxiv.2503.01935},
  eprint = {2503.01935},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.01935}
}

References

60 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Shubham Agarwal, Issam H. Laradji, Laurent Charlin, and Christopher Pal. 2024. Litllm: A toolkit for scientific literature review.
  3. Saaket Agashe, Yue Fan, Anthony Reyna, and Xin Eric Wang. 2024. Llm-coordination: Evaluating and analyzing multi-agent coordination abilities in large language models. find
  4. Asaniczka. 2023. Amazon products dataset 2023 (1.4m products). find
  5. Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. Repairagent: An autonomous, llm-based agent for program repair. arXiv preprint arXiv:2403.17134. — [arXiv:2403.17134](https://arxiv.org/abs/2403.17134) find
  6. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901.
  7. Dake Chen, Hanbin Wang, Yunhao Huo, Yuzhao Li, and Haoyang Zhang. 2023a. Gamegpt: Multi-agent collaborative framework for game development. arXiv preprint arXiv:2310.08067.
  8. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  9. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2023b. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors.
  10. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2024. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelft…
  11. Yubo Dong, Xukun Zhu, Zhengzhe Pan, Linchao Zhu, and Yi Yang. 2024. Villageragent: A graph-based multi-agent framework for coordinating complex task dependencies in minecraft. find
  12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  13. Ernst Fehr and Simon Gächter. 2000. Cooperation and punishment in public goods experiments. American Economic Review, 90(4):980–994. find
  14. Tao Feng, Chuanyang Jin, Jingyu Liu, Kunlun Zhu, Haoqin Tu, Zirui Cheng, Guanyu Lin, and Jiaxuan You. 2024. How far are we from AGI: Are LLMs all we need? Transactions on Machine Learning Research. Survey Certification. find
  15. Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, and Jianfeng Gao. 2023. Mindagent: Emergent gaming interaction. arXiv preprint arXiv:2309.09971. find
  16. Henning Gösling, Jacob Dudek, Thorsten Krause, and Oliver Thomas. 2024. Multi-agent-based peer tutoring in virtual learning environments. find
  17. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  18. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta programming for a multi-agent collaborative framework. In The T…
  19. Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. 2024. War and peace (waragent): Large language model-based multi-agent simulation of world wars.
  20. Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. 2023. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010.
  21. Yoichi Ishibashi and Yoshimasa Nishimura. 2024. Self-organized agents: A llm multi-agent framework toward ultra large-scale code generation and optimization. arXiv preprint arXiv:2404.02183. — [arXiv:2404.02183](https://arxiv.org/abs/2404.02183) find
  22. Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. Mapcoder: Multi-agent code generation for competitive problem solving. arXiv preprint arXiv:2405.11403.
  23. Yu He Ke, Rui Yang, Sui An Lie, Taylor Xin Yi Lim, Hairil Rizal Abdullah, Daniel Shu Wei Ting, and Nan Liu. 2024. Enhancing diagnostic accuracy through multi-agent conversations: Using large language models to mitigate cognitive bias. arXiv preprint arXiv:2401.14589. — [arXiv:2401.14589](https://arxiv.org/abs/2401.14589) find
  24. Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Cynthia Breazeal, and Hae Won Park. 2024. Adaptive collaboration strategy for llms in medical decision making. arXiv preprint arXiv:2404.15155. — [arXiv:2404.15155](https://arxiv.org/abs/2404.15155) find
  25. Gen Li, Shizhe Chen, Yinan Ge, Di Jin, and Zhiyuan Liu. 2023a. Chatdev: Generating software system with chatgpt. arXiv preprint arXiv:2307.04549. — [arXiv:2307.04549](https://arxiv.org/abs/2307.04549) find
  26. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023b. Camel: Communicative agents for "mind" exploration of large language model society.
  27. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688.
  28. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311.12983.
  29. OpenAI. 2023. Openai function calling documentation. https://platform.openai.com/docs/ guides/gpt/function-calling. ↗
  30. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023a. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442. find
  31. S. Park, J. Kim, and D. Lee. 2023b. Strategic silence in multi-agent social interaction: A social deduction perspective. In Proceedings of the 37th AAAI Conference on Artificial Intelligence, pages 123–131. find
  32. David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, 1(4):515–526. find
  33. Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2024. Scaling large-language-model-based multi-agent collaboration.
  34. Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2025. Scaling large language model-based multi-agent collaboration. In The Thirteenth International Conference on Learning Representations.
  35. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating large language models to master 16000+ real…
  36. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems.
  37. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. 2017. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815.
  38. Xu Sun, Xiaoya Zhang, Yicheng Feng, Shiyang Wang, Shuming Ma, Jiuding He, Zhixu Zhang, Yuxian Gu, Yi Xu, Hao Zhou, and Zhiyuan Liu. 2023. A systematic capability evaluation of large vision-language models. arXiv preprint arXiv:2305.16372. — [arXiv:2305.16372](https://arxiv.org/abs/2305.16372) find
  39. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  40. Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. 2024. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837. Equal contribution. Work done while at Google Research.
  41. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023a. Voyager: An open-ended embodied agent with large language models. find
  42. Sheng Wang, Emily Dinan, Jack Urbanek, Arthur Zhang, Douwe Kiela, and Jason Weston. 2023b. Role-playing as a platform for dialogue modeling, empathy, and data collection. arXiv preprint arXiv:2301.09663. — [arXiv:2301.09663](https://arxiv.org/abs/2301.09663) find
  43. Xiao Wang, Shixiang Shane Gu, Yizhu Liu, Harrison Jesse, and Pieter Abbeel. 2023c. Communicative agents for software development. arXiv preprint arXiv:2307.09250. find
  44. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, a… find
  45. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2024b. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. In Proc. 2024 Annual Conference of the North American Chapter of the Association for Computational… find
  46. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models.
  47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  48. Anita W. Woolley, Christopher F. Chabris, Alex Pentland, Nada Hashmi, and Thomas W. Malone. 2010. Evidence for a collective intelligence factor in the performance of human groups. Science, 330(6004):686– 688. find
  49. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023a. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155.
  50. S. Wu, A. Holtzman, J. Buys, R. Koncel-Kedziorski, and Y. Choi. 2023b. Autogen: A framework for multi-agent collaborative decision-making with large language models. arXiv preprint arXiv:2301.XXXX. find
  51. Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See-Kiong Ng, and Jiashi Feng. 2023. Magic: Investigation of large language model powered multi-agent in cognition, adaptability, rationality and collaboration. In ICLR 2024 Workshop on Large Language Model (LLM) Agents. find
  52. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models.
  53. Haofei Yu, Zhaochen Hong, Zirui Cheng, Kunlun Zhu, Keyang Xuan, Jinwei Yao, Tao Feng, and Jiaxuan You. 2024a. Researchtown: Simulator of human research community.
  54. Xianhao Yu, Jiaqi Fu, Renjia Deng, and Wenjuan Han. 2024b. Mineland: Simulating large-scale multi-agent interactions with limited multimodal senses and physical needs. find
  55. Lingyu Zhang, Zhengran Ji, and Boyuan Chen. 2025. Crew: Facilitating human-ai teaming research. find
  56. Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Xiangru Tang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. 2023a. Agents: An open-source framework for autonomous lan… find
  57. Xuanhe Zhou, Guoliang Li, Zhaoyan Sun, Zhiyuan Liu, Weize Chen, Jianming Wu, Jiesi Liu, Ruohang Feng, and Guoyang Zeng. 2023b. D-bot: Database diagnosis system using large language models. find
  58. Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. 2024a. Hypothesis generation with large language models. arXiv preprint arXiv:2404.04326.
  59. Zhilun Zhou, Yuming Lin, Depeng Jin, and Yong Li. 2024b. Large language model for participatory urban planning. find
  60. Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2025-multiagentbench-evaluating-collaboration-competition/