Elamur Research⌕
☰
home / papers / 2023 / Theory of Mind for Multi-Agent Collaboration via Large Language Models

Theory of Mind for Multi-Agent Collaboration via Large Language Models

Paper · 2023 · Huao Li, Yu Chong, Simon Stepputtis et al.

Summary

The paper asks whether agents driven by large language models can cooperate as a team and reason about teammates' hidden knowledge and intentions (Theory of Mind, ToM). The authors test this in an interactive setting rather than with static Sally-Anne-style puzzles. They built a text-based cooperative search-and-rescue game. Three agents explore a graph of rooms, find color-coded bombs and defuse them by cutting wires in the correct order. The wire cutters are split among the agents, so they must coordinate through chat. A rule-based interface turns game state into natural-language observations and maps the models' replies back to actions. Agents running gpt-3.5-turbo-0301 and gpt-4-0314 were compared with MARL and planning-based baselines. ToM was probed with inference questions checked against human-annotated answers. The LLM teams performed on par with a state-of-the-art RL method and showed emergent cooperation and signs of higher-order ToM. However, they lost track of long-horizon context and hallucinated the task state. Adding an explicit textual belief state to the prompt improved both team performance and ToM accuracy.

Key points

  • Environment: agents Alpha, Bravo and Charlie work in 5 rooms with 5 bombs (two single-phase, two double-phase, one triple-phase).
    • Bombs use 3 wire colors.
    • Each defused phase scores 10 points, for a maximum of 90.
    • A trial ends when all bombs are defused, after 30 rounds, or when agents get stuck repeating themselves.
  • Implementation: the task is built on the gym API and wrapped in a template-based text interface.
    • The interface gives error feedback for invalid actions.
    • Agents cannot see each other's actions unless these are communicated.
  • LLM context: agents receive the game rules plus the last two rounds of history, within 4096 tokens.
  • Main failures: the authors identify two systematic problems, neglecting long-horizon context and hallucinating the task state. False beliefs then spread to teammates through messages.
  • Fix and limitations: the proposed fix is prompt engineering with an explicit belief state. The authors acknowledge several limits:
    • only OpenAI models were tested;
    • the environment is small;
    • all agents use homogeneous policies;
    • ToM ground truth comes from human annotators.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓University of PittsburghPittsburgh · United States38 works · 21,591 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksPsychology · 798 worksDevelopmental and Educational Psychology · 260 works

Cite

Cite as Huao Li, Yu Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Charles Lewis and Katia Sycara. 2023. Theory of Mind for Multi-Agent Collaboration via Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.13

@inproceedings{li2023theory,
  title = {{Theory of Mind for Multi-Agent Collaboration via Large Language Models}},
  author = {Huao Li, Yu Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Charles Lewis and Katia Sycara},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.13},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.13}
}

References

31 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Terence X Lim, Sidney Tio, and Desmond C Ong. 2020. Improving multi-agent cooperation using theory of mind. arXiv preprint arXiv:2007.15703. — [arXiv:2007.15703](https://arxiv.org/abs/2007.15703) find
  2. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. 2022. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691.
  3. Chris L Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B Tenenbaum. 2017. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1(4):0064. find
  4. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. Openai gym. find
  5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
  7. Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. 2023. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477.
  8. Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought in large language models: a cognitive perspective. arXiv preprint arXiv:2301.06627. find
  9. Scott A Miller. 2009. Children’s understanding of second-order mental states. Psychological bulletin, 135(5):749. find
  10. Shima Rahimi Moghaddam and Christopher J Honey. 2023. Boosting theory-of-mind performance in large language models via prompting. arXiv preprint arXiv:2304.11490. — [arXiv:2304.11490](https://arxiv.org/abs/2304.11490) find
  11. Ini Oguntola, Joseph Campbell, Simon Stepputtis, and Katia Sycara. 2023. Theory of mind as intrinsic motivation for multi-agent reinforcement learning. arXiv preprint arXiv:2307.01158. — [arXiv:2307.01158](https://arxiv.org/abs/2307.01158) find
  12. OpenAI. 2023. Gpt-4 technical report.
  13. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442. find
  14. Iyad Rahwan, Manuel Cebrian, Nick Obradovich, Josh Bongard, Jean-François Bonnefon, Cynthia Breazeal, Jacob W Crandall, Nicholas A Christakis, Iain D Couzin, Matthew O Jackson, et al. 2019. Machine behaviour. Nature, 568(7753):477–486. enables open-world multi-task agents. arXiv preprint arXiv:2302.…
  15. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903.
  16. Jessica Williams, Stephen M Fiore, and Florian Jentsch. 2022. Supporting artificial social intelligence with theory of mind. Frontiers in artificial intelligence, 5. find
  17. Christoph Riedl, Young Ji Kim, Pranav Gupta, Thomas W Malone, and Anita Williams Woolley. 2021. Quantifying collective intelligence in human groups. Proceedings of the National Academy of Sciences, 118(21):e2005737118. find
  18. Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. 2019. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043. — [arXiv:1902.04043](https://arxiv.org/abs/1902.04043) find
  19. Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2023. Neural theory-of-mind? on the limits of social intelligence in large lms. find
  20. Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. 2023. Minding language models’(lack of) theory of mind: A plug-and-play multi-character belief tracker. arXiv preprint arXiv:2306.00924.
  21. Guni Sharon, Roni Stern, Ariel Felner, and Nathan R Sturtevant. 2015. Conflict-based search for optimal multi-agent pathfinding. Artificial Intelligence, 219:40–66. find
  22. Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. 2023. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128. — [arXiv:2302.05128](https://arxiv.org/abs/2302.05128) find
  23. Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611–24624. find
  24. Luyao Yuan, Zipeng Fu, Linqi Zhou, Kexin Yang, and Song-Chun Zhu. 2021. Emergence of theory of mind collaboration in multiagent systems. arXiv preprint arXiv:2110.00121. — [arXiv:2110.00121](https://arxiv.org/abs/2110.00121) find
  25. Jun Zhang, Trey Hedden, and Adrian Chia. 2012. Perspective-taking and depth of theory-of-mind reasoning in sequential-move games. Cognitive science, 36(3):560–573. find
  26. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. find
  27. Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. 2017. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296. — [arXiv:1706.05296](https://arxiv.org/abs/1706.05296) find
  28. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239.
  29. Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399. find
  30. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023a. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. find
  31. Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023b. Describe, explain, plan and select: Interactive planning with large language models find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2023-theory-mind-multi-agent-collaboration/