Elamur Research⌕
☰
home / papers / 2024 / GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents

GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents

Paper · 2024 · Anthony Costarelli, Mat Allen, Roman Hauksson et al.

Summary

LLM agents are increasingly used for strategic tasks, yet no broad framework existed to test the many kinds of reasoning that games demand. The authors built GameBench, a multi-agent benchmark of 9 little-known board, card and social-deception games. They chose games with no published strategy guides or dedicated forums, so strategy knowledge is unlikely to come from pretraining data. The games cover six reasoning dimensions, including abstract strategy, chance, hidden information, language, social deduction and cooperation. Each game is a Python environment with a shared interface.

They tested GPT-3 (gpt-3.5-turbo-1106) and GPT-4 (gpt-4-1106-preview) in base form and with Chain-of-Thought (CoT) or Reasoning via Planning (RAP) scaffolding. These agents played against each other, a random-move agent and human players. Ratings came from a bootstrapped exponential Bradley–Terry model.

Humans beat every LLM configuration. CoT gave the largest improvement, but even scaffolded models stayed well below human level. The benchmark is meant to be hard to saturate as models improve.

Key points

  • Games: Air, Land, Sea; Arctic Scavengers; Are You the Traitor?; Codenames; Hive; Pit; Santorini; Two Rooms and a Boom; and Sea Battle. They map onto six reasoning categories, for example 6 games involve abstract strategy and 2 involve social deduction.
  • Rating method: the exponential Bradley–Terry model was fit by maximum likelihood over 10,000 bootstrap resamples. Matches were weighted inversely to how many matches each game had, and results are reported with 90% confidence intervals.
  • Ranking: humans were best, followed by CoT agents, then GPT-4 with RAP. Base GPT-3 roughly matched random play, and base GPT-4 scored below random, largely because of a very poor Sea Battle rating.
  • Best LLM: GPT-4 with CoT was the strongest LLM agent overall and led on Sea Battle and Pit. RAP helped GPT-4 less than CoT did.
  • Limitations: the aggregate scores are sensitive to which games are included (removing Sea Battle changes the picture). The human baseline is small, and the out-of-distribution status of the games is not verified. Code is available at github.com/Joshuaclymer/GameBench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States254 works · 56,747 citations🎓University of OxfordOxford · United Kingdom122 works · 160,277 citations🎓University College LondonLondon · United Kingdom107 works · 22,923 citations🎓The University of Texas at DallasRichardson · United States63 works · 5,710 citations🎓University of ManchesterManchester · United Kingdom93 works · 45,138 citations🎓ShanghaiTech UniversityShanghai · China14 works · 1,974 citations

Also on the byline: Franklin W. Olin College of Engineering.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer and Arjun Yadav. 2024. GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents. arXiv:2406.06613.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.06613 · arXiv 2406.06613

@misc{costarelli2024gamebench,
  title = {{GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents}},
  author = {Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer and Arjun Yadav},
  year = {2024},
  doi = {10.48550/arxiv.2406.06613},
  eprint = {2406.06613},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.06613}
}

References

44 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schonherr, and Mario Fritz. Llm-deliberation: Evaluating llms with interactive multi-agent negotiation games. ArXiv, abs/2309.17234, 2023. URL https://api.semanticscholar.org/CorpusID:263310628. — [arXiv:2309.17234](https://arxiv.org/abs/2309.17234) ↗
  2. Marwa Abdulhai, Isadora White, Charles Burton Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models. ArXiv, abs/2311.18232, 2023. URL https://api.semanticscholar.org/ CorpusID:265506611. — [arXiv:2311.18232](https://arxiv.org/abs/2311.18232) ↗
  3. Saaket Agashe, Yue Fan, Anthony Reyna, and Xin Eric Wang. Llm-coordination: Evaluating and analyzing multi-agent coordination abilities in large language models, 2024. find
  4. Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. ArXiv, abs/2305.16867, 2023. URL https://api.semanticscholar.org/CorpusID:258947115. — [arXiv:2305.16867](https://arxiv.org/abs/2305.16867) ↗
  5. Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sandra Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe,… find
  6. Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. ISSN 00063444. URL http://www.jstor.org/stable/2334029. ↗
  7. Kranti Chalamalasetti, Jana Gotze, Sherzod Hakimov, Brielen Madureira, P. Sadler, and David Schlangen. clembench: Using game play to evaluate chat-optimized language models as conversational agents. ArXiv, abs/2305.13455, 2023. URL https://api.semanticscholar.org/ CorpusID:258841392. ↗
  8. Jiangjie Chen, Siyu Yuan, Rong Ye, Bodhisattwa Prasad Majumder, and Kyle Richardson. Put your money where your mouth is: Evaluating strategic planning and execution of llm agents in an auction arena. ArXiv, abs/2310.05746, 2023. URL https://api.semanticscholar.org/CorpusID: 263831697. — [arXiv:2310.05746](https://arxiv.org/abs/2310.05746) ↗
  9. Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang, Weijuan Tu, Zhaofeng He, and Lijie Wen. Llmarena: Assessing capabilities of large language models in dynamic multi-agent environments. ArXiv, abs/2402.16499, 2024. URL https://api.semanticscholar.org/CorpusID: 268032489. ↗
  10. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2023. URL https://d…
  11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — PaLM: Scaling Language Modeling with Pathways
  12. Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations. ArXiv, abs/2402.12348, 2024. URL https://api.semanticscholar.… ↗
  13. Arpad E Elo. The proposed uscf rating system, its development, theory, and applications. Chess life, 22(8):242–247, 1967. find
  14. Kanishk Gandhi, Dorsa Sadigh, and Noah D. Goodman. Strategic reasoning with language models. ArXiv, abs/2305.19165, 2023a. URL https://api.semanticscholar.org/CorpusID: 258968043. ↗
  15. Kanishk Gandhi, Dorsa Sadigh, and Noah D. Goodman. Strategic reasoning with language models, 2023b. find
  16. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023. URL https://doi.org/10.48550/arXiv.2305.14992. ↗
  17. David Ilić. Unveiling the general intelligence factor in language models: A psychometric approach, 2023. find
  18. Majeed Kazemitabaar, Xinying Hou, Austin Henley, Barbara J. Ericson, David Weintrop, and Tovi Grossman. How novices use llm-based code generators to solve cs1 coding tasks in a self-paced learning environment, 2023. find
  19. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms, 2023.
  20. Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. Avalonbench: Evaluating llms playing the game of avalon, 2023. URL https://api.semanticscholar.org/CorpusID:265302489. ↗
  21. Jiaju Lin, Haoran Zhao, Aochi Zhang, Yiting Wu, Huqiuyue Ping, and Qin Chen. Agentsims: An open-source sandbox for large language model evaluation, 2023. find
  22. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as age… — AgentBench: Evaluating LLMs as Agents
  23. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Yuxian Gu, Hangliang Ding, Kai Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Shengqi Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating l… — AgentBench: Evaluating LLMs as Agents
  24. Shaoguang Mao, Yuzhe Cai, Yan Xia, Wenshan Wu, Xun Wang, Fengyi Wang, Tao Ge, and Furu Wei. Alympics: Llm agents meet game theory – exploring strategic decision-making with ai agents, 2023. URL https://api.semanticscholar.org/CorpusID:265034042. ↗
  25. Lucas Maystre. choix: Inference algorithms for models based on luce’s choice axiom. https://github.com/lucasmaystre/choix, 2015. GitHub repository, accessed: 2024-06-04. ↗
  26. METR. Evaluating language-model agents on realistic autonomous tasks. https://metr.org/ blog/2023-08-01-new-report/, 2023. ↗
  27. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants, 2023.
  28. Dan Qiao, Chenfei Wu, Yaobo Liang, Juntao Li, and Nan Duan. Gameeval: Evaluating llms on conversational games. ArXiv, abs/2308.10032, 2023. URL https://api.semanticscholar.org/CorpusID:261048971. — [arXiv:2308.10032](https://arxiv.org/abs/2308.10032) ↗
  29. Toran Bruce Richards. Autogpt: An autonomous gpt-4 experiment. https://github.com/ Significant-Gravitas/AutoGPT/tree/master, 2023. ↗
  30. Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta, and Aran Komatsuzaki. Arb: Advanced reasoning benchmark for large language models, 2023.
  31. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023.
  32. David Silver, Aja Huang, Christopher Maddison, Arthur Guez, Laurent Sifre, George Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu… find
  33. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm…
  34. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024…
  35. Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. ArXiv, abs/2302.01560, 2023. URL https://api.semanticscholar.org/CorpusID:256598146. ↗
  36. Adam Watkins, Srijan Subedi, and Asim Shrestha. Agentgpt. https://github.com/reworkd/ AgentGPT, 2023. ↗
  37. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022a.
  38. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022b. URL https://doi.org/10.48550/arXiv.2201.11903. ↗
  39. Yue Wu, Xuan Tang, Tom M. Mitchell, and Yuanzhi Li. Smartplay : A benchmark for llms as intelligent agents. ArXiv, abs/2310.01557, 2023a. URL https://api.semanticscholar.org/ CorpusID:263608611. ↗
  40. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. ArXiv, abs/2307.02477, 2023b. URL https://api.semanticscholar.org… ↗
  41. Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See-Kiong Ng, and Jiashi Feng. Magic: Investigation of large language model powered multi-agent in cognition, adaptability, rationality and collaboration. ArXiv, abs/2311.08562, 2023a. URL https://api. semanticscholar.org/CorpusID… find
  42. Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. ArXiv, abs/2309.04658, 2023b. URL https://api.semanticscholar.org/CorpusID:261681932. ↗
  43. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022.
  44. Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Y. Qiao, Zhaoxiang Zhang, and Jifeng Dai. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. A…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/costarelli-2024-gamebench-evaluating-strategic-reasoning-abilities/