Elamur Research⌕
☰
home / papers / 2024 / Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities

Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities

Paper · 2024 · Alex Wilf, Sihyun Lee, Paul Pu Liang et al.

Summary

Large language models still handle Theory-of-Mind (ToM) questions poorly, and prompting tricks such as Chain-of-Thought (CoT) that boost reasoning elsewhere help little here. Drawing on the cognitive-science account called "Simulation Theory," the authors argue that a model asked to infer a character's beliefs in one pass must do two jobs at once: work out the character's point of view and then answer the question. They propose SimToM, a zero-shot prompting scheme that splits these into two calls. First, the LLM rewrites the story so that it keeps only the events the target character is aware of. Second, it answers the ToM question using just that reduced story. The method needs no fine-tuning and little prompt adjustment. Tested on two Sally-Anne-style false-belief benchmarks, ToMI and BigTOM, SimToM substantially outperforms 0-shot multiple-choice and CoT prompting. The analysis also suggests that current models reason about minds better than single-pass probing reveals, which makes perspective-taking a promising target for further ToM work.

Key points

  • Two-stage pipeline. Stage 1 is perspective-taking, which filters the context to what the character knows. Stage 2 is question-answering over that filtered context. The authors report that running the stages as separate prompts works better than merging them into one.
  • Benchmarks.
    • ToMI is CC-licensed, and the authors use the corrected release described by Arodi and Cheung (2021) and by Sap et al. (2022).
    • BigTOM was generated with GPT-4 and is MIT-licensed. Both are English-only.
  • Weak baselines. In earlier work, GPT-3.5-Turbo scored only 12.5% on BigTOM's "action" false-belief subset. GPT-4 still trails humans by a wide margin on ToMI.
  • Oracle perspective-taking. When given perfect perspective-taking, today's LLMs come close to solving these benchmarks, so better perspective-taking should translate into stronger ToM performance.
  • Limitations.
    • Perspective-taking is implemented only as hiding story content, not as inferring information the character never saw.
    • No models smaller than 7B parameters were tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,225 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksPsychology · 1,649 worksDevelopmental and Educational Psychology · 523 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Alex Wilf, Sihyun Lee, Paul Pu Liang and Louis-Philippe Morency. 2024. Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.451

@inproceedings{wilf2024think,
  title = {{Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities}},
  author = {Alex Wilf, Sihyun Lee, Paul Pu Liang and Louis-Philippe Morency},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.451},
  url = {https://doi.org/10.18653/v1/2024.acl-long.451}
}

References

34 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akshatha Arodi and Jackie Chi Kit Cheung. 2021. Textual time travel: A temporally informed approach to theory of mind. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4162–4172. find
  2. Luca Barlassina and Robert M. Gordon. 2017. Folk Psychology as Mental Simulation. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, Summer 2017 edition. Metaphysics Research Lab, Stanford University. find
  3. Simon Baron-Cohen, Alan M Leslie, and Uta Frith. 1985. Does the autistic child have a “theory of mind”? Cognition, 21(1):37–46. find
  4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  5. Peter Carruthers. 2009. How we know our own minds: The relationship between mindreading and metacognition. Behavioral and Brain Sciences, 32(2):121–138. find
  6. Alvin I. Goldman. 2006. Simulating Minds: The Philosophy, Psychology, and Neuroscience of Mindreading. New York, US: Oxford University Press USA. find
  7. Alvin I. Goldman. 2008. Hurley on simulation. Philosophy and Phenomenological Research, 77(3):775– 788. find
  8. Alison Gopnik and Henry M Wellman. 1994. The theory theory. In An earlier version of this chapter was presented at the Society for Research in Child Development Meeting, 1991. Cambridge University Press. find
  9. Robert M. Gordon. 1986. Folk psychology as simulation. Mind and Language, 1(2):158–71. find
  10. Robert M. Gordon. 1995. Simulation without introspection or inference from me to you. In Martin Davies and Tony Stone, editors, Mental Simulation. Blackwell. find
  11. Robert M. Gordon. 2007. Ascent routines for propositional attitudes. Synthese, 159(2):151–165. find
  12. Jane Heal. 1994. Simulation vs. theory-theory: What is at issue? In Christopher Peacocke, editor, Objectivity, Simulation and the Unity of Consciousness: Current Issues in the Philosophy of Mind. Oxford University Press. find
  13. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. arXiv preprint arXiv:1909.00277.
  14. Susan Hurley. 2008. Understanding simulation1. Philosophy and Phenomenological Research, 77(3):755– 774. find
  15. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  16. Michal Kosinski. 2023. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083.
  17. Computational Linguistics (Volume 1: Long Papers), pages 13960–13980, Toronto, Canada. Association for Computational Linguistics. find
  18. Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. 2023. Clever hans or neural theory of mind? stress testing social reasoning in large language models. arXiv preprint arXiv:2305.14763. find
  19. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR.
  20. Christelle Langley, Bogdan Ionut Cirstea, Fabio Cuzzolin, and Barbara J Sahakian. 2022. Theory of mind and preference learning at the interface of cognitive science, neuroscience, and ai: A review. Frontiers in Artificial Intelligence, 5:62. find
  21. Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. 2020. Proofwriter: Generating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012.13048.
  22. Matthew Le, Y-Lan Boureau, and Maximilian Nickel. 2019. Revisiting the evaluation of theory of mind through question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-I… find
  23. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  24. Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. 2021. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. Advances in Neural Information Processing Systems, 34:25192– 25204. find
  25. OpenAI. 2023. Gpt-4 technical report.
  26. David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515–526. find
  27. Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2022. Neural theory-of-mind? on the limits of social intelligence in large lms. arXiv preprint arXiv:2210.13312. find
  28. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi… find
  29. Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. 2023. Minding language models’ (lack of) theory of mind: A plug-and-play multi-character belief tracker. In Proceedings of the 61st Annual Meeting of the Association for Tomer Ullman. 2023. Large language models fai…
  30. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
  31. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  32. Henry M. Wellman, David Cross, and Julanne Watson. 2001. Meta-analysis of theory-of-mind development: The truth about false belief. Child Development, 72(3):655–684. find
  33. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601. find
  34. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wilf-2024-think-twice-perspective-taking-improves/