Elamur Research⌕
☰
home / papers / 2024 / MMToM-QA: Multimodal Theory of Mind Question Answering

MMToM-QA: Multimodal Theory of Mind Question Answering

Paper · 2024 · Chuanyang Jin, Yutong Wu, Jing Cao et al.

Summary

Existing Theory of Mind (ToM) benchmarks test models on a single modality, either video or text. Human mental-state reasoning, however, combines whatever evidence is available. The authors introduce MMToM-QA, a benchmark that pairs household activity videos with textual descriptions of the scene and the person's actions. Its two-choice questions probe the person's goals and beliefs, and some require combining both input sources.

They also propose BIP-ALM, a method that uses language models to speed up Bayesian inverse planning. The method converts video and text into a shared symbolic representation, then runs Bayesian inverse planning in which a finetuned language model scores how likely actions are under hypothesized beliefs and goals.

The authors compare humans, BIP-ALM and strong baselines, including GPT-4 and GPT-4V. Large language and multimodal models handle questions about the world state well. They fail, however, at tracking mental states and are especially weak at inferring goals. BIP-ALM beats all baselines by a wide margin and generalizes to real human behavior. This suggests that model-based inference combined with LMs is a promising route to machine social intelligence.

Key points

  • Benchmark size: The main set has 134 videos, each averaging about 1,462 frames and 36 actions. These videos support 600 questions in seven types: 300 on belief inference (100 per type) and 300 on goal inference (75 per type).
  • Training data: A training set of 1,000 procedurally generated videos comes with ground-truth annotations of scene, objects, goals and beliefs.
  • Generalization test: An extra human test set has 40 videos and 120 questions. It uses 2 unseen apartments and avatars controlled by 3 real participants, and BIP-ALM generalizes to this setting.
  • Baseline failures: Neither few-shot nor chain-of-thought prompting, nor larger model sizes, consistently improved baselines. A finetuned Video-LLaMA 2 (13B) still performed at roughly chance overall.
  • BIP-ALM limitations: It cannot infer state information missing from the video. It relies on purely symbolic planning, which hurts on Type 2.4, where continuous spatial reasoning is needed. It also inherits occasional LM planning errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States248 works · 106,397 citations🎓New York UniversityNew York · United States219 works · 103,487 citations🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations🎓Harvard UniversityCambridge · United States126 works · 45,130 citations🎓University of California San DiegoSan Diego · United States89 works · 54,032 citations🎓University of VirginiaCharlottesville · United States40 works · 10,099 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksPsychology · 1,649 worksDevelopmental and Educational Psychology · 523 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 811 worksQuestion answering · 487 works

Cite

Cite as Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua Tenenbaum and Tianmin Shu. 2024. MMToM-QA: Multimodal Theory of Mind Question Answering. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.851

@inproceedings{jin2024mmtom,
  title = {{MMToM-QA: Multimodal Theory of Mind Question Answering}},
  author = {Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua Tenenbaum and Tianmin Shu},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.851},
  url = {https://doi.org/10.18653/v1/2024.acl-long.851}
}

References

55 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433. find
  2. Chris L Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B Tenenbaum. 2017. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1(4):1–10. find
  3. John Hewitt and Michael Cohen. 2021. Exploring roberta’s theory of mind through textual entailment. find
  4. Chris L Baker, Rebecca Saxe, and Joshua B Tenenbaum. 2009. Action understanding as inverse planning. Cognition, 113(3):329–349. find
  5. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  6. Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. 2022. A persistent spatial semantic representation for high-level natural language instruction execution. In Conference on Robot Learning, pages 706–717. PMLR. find
  7. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  8. Rohan Chandra, Aniket Bera, and Dinesh Manocha. 2020. Stylepredict: Machine theory of mind for human driver behavior from trajectories. arXiv preprint arXiv:2011.04816. — [arXiv:2011.04816](https://arxiv.org/abs/2011.04816) find
  9. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning.
  10. Kerstin Dautenhahn. 2007. Socially intelligent robots: dimensions of human–robot interaction. Philosophical transactions of the royal society B: Biological sciences, 362(1480):679–704. find
  11. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394.
  12. Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D Goodman. 2023. Understanding social reasoning in language models with language models. arXiv preprint arXiv:2306.15448. find
  13. Kanishk Gandhi, Gala Stojnic, Brenden M Lake, and Moira R Dillon. 2021. Baby intuitions benchmark (bib): Discerning the goals, preferences, and actions of others. Advances in Neural Information Processing Systems, 34:9963–9976. find
  14. Andrew S. Gordon. 2016. Commonsense interpretation of triangle behavior. In AAAI Conference on Artificial Intelligence. find
  15. Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. 2016. Cooperative inverse reinforcement learning. In Advances in neural information processing systems. find
  16. Yinghui He, Yufan Wu, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. 2023. Hi-tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models. arXiv preprint arXiv:2310.16755. find
  17. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118–9147. PMLR.
  18. Julian Jara-Ettinger. 2019. Theory of mind as inverse reinforcement learning. Current Opinion in Behavioral Sciences, 29:105–110. find
  19. Julian Jara-Ettinger, Hyowon Gweon, Laura E Schulz, and Joshua B Tenenbaum. 2016. The naïve utility calculus: Computational principles underlying commonsense psychology. Trends in cognitive sciences, 20(8):589–604. find
  20. Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. 1998. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134. find
  21. Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. 2023. Fantom: A benchmark for stress-testing machine theory of mind in interactions. arXiv preprint arXiv:2310.15421. find
  22. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  23. Michal Kosinski. 2023. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083.
  24. Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253. find
  25. Matt Le, Y-Lan Boureau, and Maximilian Nickel. 2019. Revisiting the evaluation of theory of mind through question answering. In Conference on Empirical Methods in Natural Language Processing. find
  26. Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. 2023. M 3 it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306.04387. — [arXiv:2306.04387](https://arxiv.org/abs/2306.04387) find
  27. Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. 2022. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31199–31212.
  28. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. arXiv preprint arXiv:2304.08485.
  29. Shima Rahimi Moghaddam and Christopher J Honey. 2023. Boosting theory-of-mind performance in large language models via prompting. arXiv preprint arXiv:2304.11490. — [arXiv:2304.11490](https://arxiv.org/abs/2304.11490) find
  30. Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Thomas L Griffiths. 2018. Evaluating theory of mind in question answering. arXiv preprint arXiv:1808.09352. find
  31. Aviv Netanyahu, Tianmin Shu, Boris Katz, Andrei Barbu, and Joshua B Tenenbaum. 2021. Phase: Physically-grounded abstract social events for machine social perception. In Proceedings of the aaai conference on artificial intelligence, volume 35, pages 845–853. find
  32. OpenAI. 2023. Gpt-4 technical report. abs/2303.08774.
  33. ArXiv, Maithili Patel and Sonia Chernova. 2022. Proactive robot assistance via spatio-temporal object modeling. arXiv preprint arXiv:2211.15501. — [arXiv:2211.15501](https://arxiv.org/abs/2211.15501) find
  34. Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba. 2020. Watch-and-help: A challenge for social perception and human-ai collaboration. arXiv preprint arXiv:2010.09890. — [arXiv:2010.09890](https://arxiv.org/abs/2010.09890) find
  35. Xavier Puig, Tianmin Shu, Joshua B Tenenbaum, and Antonio Torralba. 2023. Nopa: Neurally-guided online probabilistic assistance for building socially intelligent home assistants. arXiv preprint arXiv:2301.05223. — [arXiv:2301.05223](https://arxiv.org/abs/2301.05223) find
  36. Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. 2018. Machine theory of mind. In International conference on machine learning, pages 4218–4227. PMLR. find
  37. Kate Sanders, David Etter, Reno Kriz, and Benjamin Van Durme. 2023. Multivent: Multilingual videos of events with aligned natural text. arXiv preprint arXiv:2307.03153. — [arXiv:2307.03153](https://arxiv.org/abs/2307.03153) find
  38. Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2022. Neural theory-of-mind? on the limits of social intelligence in large lms. arXiv preprint arXiv:2210.13312. find
  39. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728.
  40. Rebecca Saxe. 2012. The happiness of the fish: Evidence for a common theory of one’s own and others’ actions. In Handbook of Imagination and Mental Simulation, pages 257–309. Psychology Press. find
  41. Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. 2023. Minding language models’ (lack of) theory of mind: A plug-and-play multi-character belief tracker.
  42. Melanie Sclar, Graham Neubig, and Yonatan Bisk. 2022. Symmetric machine theory of mind. In International Conference on Machine Learning, pages 19450–19466. PMLR. find
  43. Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. 2023. Clever hans or neural theory of mind? stress testing social reasoning in large language models. arXiv preprint arXiv:2305.14763. find
  44. Tianmin Shu, Abhishek Bhandwaldar, Chuang Gan, Kevin Smith, Shari Liu, Dan Gutfreund, Elizabeth Spelke, Joshua Tenenbaum, and Tomer Ullman. 2021. Agent: A benchmark for core psychological reasoning. In International Conference on Machine Learning, pages 9614–9625. PMLR. find
  45. Hrituraj Singh, Anshul Nasery, Denil Mehta, Aishwarya Agarwal, Jatin Lamba, and Balaji Vasan Srinivasan. 2021. Mimoqa: Multimodal input multimodal output question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… find
  46. Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021. Multimodalqa: Complex question answering over text, tables and images. arXiv preprint arXiv:2104.06039. — [arXiv:2104.06039](https://arxiv.org/abs/2104.06039) find
  47. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  48. Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399. find
  49. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax. ↗
  50. Qiaosi Wang, Koustuv Saha, Eric Gregori, David Joyner, and Ashok Goel. 2021. Towards mutual theory of mind in human-ai interaction: How language reflects what students perceive about a virtual teaching assistant. In Proceedings of the 2021 CHI conference on human factors in computing systems, pages… find
  51. Alex Wilf, Sihyun Shawn Lee, Paul Pu Liang, and Louis-Philippe Morency. 2023. Think twice: Perspective-taking improves large language models’ theory-of-mind capabilities. arXiv preprint arXiv:2311.10227, arXiv:2311.10227v1.
  52. Heinz Wimmer and Josef Perner. 1983. Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children’s understanding of deception. Cognition, 13(1):103–128. find
  53. Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. 2019. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8807–8817. find
  54. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720–6731. find
  55. Hang Zhang, Xin Li, and Lidong Bing. 2023. Videollama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jin-2024-mmtom-qa-multimodal-theory-mind/