Elamur Research⌕
☰
home / papers / 2022 / CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions

CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions

Paper · 2022 · Tayfun Ates, M. Ateşoğlu, Çağatay Yiğit et al.

Summary

The paper tests whether vision-language models can reason about causes and effects in physical interactions between objects. The authors created CRAFT, a dataset for video question answering. It consists of synthetic 2D videos simulated in the Box2D physics engine. Questions about the videos are generated automatically from templates, and heuristics remove obvious biases in the data. Besides the descriptive and counterfactual questions known from CLEVRER, the authors added a new category of causal questions based on Force Dynamics Theory from cognitive linguistics. In these questions, the model must determine whether one object caused (cause), enabled (enable) or prevented (prevent) an event involving another object. Evaluation has an easy and a hard setting. In the hard setting, the test scenes have layouts that the model did not see during training. The evaluated models include text-only models, single-frame models and video models, among them MAC, TVQA and G-SWM. In the hard setting, none exceeded roughly 47% accuracy, while humans reached roughly 77%. The benchmark clearly reveals a gap in the physical and causal reasoning of current models.

Key points

  • Data size. About 58K video–question pairs based on 10K videos from 20 different scene layouts. Each video lasts 10 s and has a resolution of 256×256. The data is split into train/val/test in a 60:20:20 ratio by video. The abstract states 58K, while the data section states about 57K.
  • Structure. 48 task types in three categories: Descriptive, Counterfactual and Causal (cause / enable / prevent). A scene has 7 static elements, including a ramp, a platform, a button, a basket and walls. For counterfactual answers, additional variants of each video are generated in which each object is removed in turn.
  • Model results. In the easy setting, the best model was LSTM-CNN-V with 53.01%; in the hard setting, MAC-V with 47.31%. The text-only LSTM without video scores about 44.5%, so the visual models gain little. Models struggle most with descriptive questions: 23–48%.
  • Humans and oracles. The study involved 101 adults, of whom 56 were included in the final analysis. Overall human accuracy is 76.60%, and the gap with the models in the hard setting exceeds 29 percentage points. The oracle model BERT-D reaches 84.90–86.20%, and on descriptive questions it even slightly outperforms humans.
  • Availability. The data and code are published on the project website: sites.google.com/view/craft-benchmark. As next steps, the authors propose neuro-symbolic models (e.g., NS-DR) and object-centric methods with pre-training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Koç UniversityIstanbul · Türkiye4 works · 431 citations🎓Hacettepe UniversityAnkara · Türkiye3 works · 25 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputer Vision and Pattern Recognition · 3,621 worksEngineering · 206 worksControl and Systems Engineering · 72 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,948 worksEvaluation & benchmarks · 1,394 works

Cite

Cite as Tayfun Ates, M. Ateşoğlu, Çağatay Yiğit, Ilker Kesen, Mert Kobas, Erkut Erdem, Aykut Erdem, Tilbe Goksun and Deniz Yuret. 2022. CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions. Findings of the Association for Computational Linguistics: ACL 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-acl.205

@inproceedings{ates2022craft,
  title = {{CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions}},
  author = {Tayfun Ates, M. Ateşoğlu, Çağatay Yiğit, Ilker Kesen, Mert Kobas, Erkut Erdem, Aykut Erdem, Tilbe Goksun and Deniz Yuret},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.205},
  url = {https://doi.org/10.18653/v1/2022.findings-acl.205}
}

References

51 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kelsey R. Allen, Kevin A. Smith, and Joshua B. Tenenbaum. 2020. Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning. arXiv preprint arXiv:1907.09620. — [arXiv:1907.09620](https://arxiv.org/abs/1907.09620) find
  2. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In ICCV, pages 2425–2433. find
  3. Stéphane Aroca-Ouellette, Cory Paik, Alessandro Roncone, and Katharina Kann. 2020. Prost: Physical reasoning of objects through space and time. In ACL-Findinfs. find
  4. Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Kenneth Wong, Joshua B. Tenenbaum, and Chuang Gan. 2021. Grounding physical concepts of objects and events through dynamic visual reasoning. In ICLR. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  6. David Ding, Felix Hill, Adam Santoro, and Matt Botvinick. 2020. Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures. arXiv preprint arXiv:2012.08508. — [arXiv:2012.08508](https://arxiv.org/abs/2012.08508) find
  7. Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Joshua B Tenenbaum, and Chuang Gan. 2021. Dynamic visual reasoning by learning differentiable physics models from video and language. In Advances In Neural Information Processing Systems. find
  8. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2020. Datasheets for Datasets. arXiv:1803.09010. find
  9. Nathan R. George, Tilbe Göksun, Kathy Hirsh-Pasek, and Roberta Michnick Golinkoff. 2019. Any way the wind blows: Children’s inferences about force and motion events. Journal of Experimental Child Psychology, 177:119–131. find
  10. Rohit Girdhar and Deva Ramanan. 2020. CATER: A diagnostic dataset for compositional actions and temporal reasoning. In ICLR. find
  11. Renée Baillargeon. 1994. Physical reasoning in young infants: Seeking explanations for impossible events. British Journal of Developmental Psychology, 12(1):9–33. find
  12. Tilbe Göksun, Nathan R. George, Kathy Hirsh-Pasek, and Roberta M. Golinkoff. 2013. Forces and motion: Causal understanding of preschoolers. Child Development, 84:1285–1295. find
  13. Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. 2019. Phyre: A new benchmark for physical reasoning. In NeurIPS, pages 5082–5093.
  14. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, pages 6904–6913.
  15. Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. 2020. CoPhy: Counterfactual learning of physical dynamics. In ICLR.
  16. Ari Beller, Erin Bennett, and Tobias Gerstenberg. 2020. The language of causation. In CogSci. find
  17. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In AAAI, volume 34, pages 7432–7439.
  18. Erin Catto. 2010. Box2d v2.0.1 user manual. find
  19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735– 1780. find
  20. Xin Hong, Yanyan Lan, Liang Pang, Jiafeng Guo, and Xueqi Cheng. 2021. Transformation driven visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6903–6912. find
  21. Drew A. Hudson and Christopher D. Manning. 2018. Compositional attention networks for machine reasoning. In ICLR.
  22. Michael Janner, Sergey Levine, William T. Freeman, Joshua B. Tenenbaum, Chelsea Finn, and Jiajun Wu. 2019. Reasoning about physical interactions with object-oriented prediction and planning. In ICLR. find
  23. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, pages 2901–2910.
  24. Sangeet S Khemlani, Aron K Barbey, and Philip N Johnson-Laird. 2014. Causal reasoning with mental models. Front. Hum. Neurosci., 8:849. find
  25. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  26. Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. 2018. TVQA: Localized, compositional video question answering. In EMNLP.
  27. Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. 2020. TVQA+: Spatio-temporal grounding for video question answering. In ACL. find
  28. Adam Lerer, Sam Gross, and Rob Fergus. 2016. Learning physical intuition of block towers by example. In ICML. find
  29. Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. 2019. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. ICLR. find
  30. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV, pages 740– 755. Springer.
  31. Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. 2020. Improving generative imagination in object-centric world models. In International Conference on Machine Learning, pages 6140–6149. PMLR. find
  32. Mateusz Malinowski and Mario Fritz. 2014. A multi-world approach to question answering about real-world scenes based on uncertain input. In NeurIPS, pages 1682–1690. find
  33. Mengye Ren, Ryan Kiros, and Richard Zemel. 2015. Exploring models and data for image question answering. In NeurIPS, pages 2953–2961.
  34. Shailaja Keyur Sampat, Akshay Kumar, Yezhou Yang, and Chitta Baral. 2021. CLEVR_HYP: A challenge dataset and baselines for visual question answering with hypothetical actions over images. In NAACL-HLT. find
  35. Rebecca Saxe, JB Tenenbaum, and Susan Carey. 2005. Secret agents: Inferences about hidden causes by 10-and 12-month-old infants. Psychological Science, 16(12):995–1001. find
  36. Steven Sloman, Aron K Barbey, and Jared M Hotal-ing. 2009. A causal model theory of the meaning of cause, enable, and prevent. Cognitive Science, 33(1):21–50. find
  37. Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. MovieQA: Understanding stories in movies through question-answering. In CVPR, pages 4631– 4640. find
  38. Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. 2018. A closer look at spatiotemporal convolutions for action recognition. In CVPR, pages 6450–6459. find
  39. Misha Wagner, Hector Basevi, Rakshith Shetty, Wenbin Li, Mateusz Malinowski, Mario Fritz, and Ales Leonardis. 2018. Answering visual what-if questions: From actions to predicted scene descriptions. In ECCV Workshops. find
  40. Phillip Wolff. 2007. Representing causation. Journal of experimental psychology: General, 136(1):82. find
  41. Phillip Wolff. 2013. Direct causation in the linguistic coding and individuation of causal events. Cognition, 88(1):1–48. find
  42. Phillip Wolff and Aron K Barbey. 2015. Causal reasoning with forces. Front. Hum. Neurosci., 9:1. find
  43. Phillip Wolff, Bianca Klettke, Tatyana Ventura, and Grace Song. 2005. Expressing causation in english and other languages. In Categorization inside and outside the laboratory: Essays in honor of Douglas L. Medin., pages 29–48. American Psychological Association. find
  44. A Michotte. 1963. The perception of causality. Basic Books. find
  45. Phillip Wolff and Grace Song. 2003. Models of causation and the semantics of causal verbs. Cognitive psychology, 47(3):276–332. find
  46. Roozbeh Mottaghi, Hessam Bagherinezhad, Mohammad Rastegari, and Ali Farhadi. 2016. Newtonian scene understanding: Unfolding the dynamics of objects in static images. In CVPR, pages 3521–3529. find
  47. Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. 2020. Clevrer: Collision events for video representation and reasoning. In ICLR. find
  48. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  49. Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, volume 33, pages 9127–9134.
  50. Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016. Yin and yang: Balancing and answering binary visual questions. In CVPR, pages 5014–5022. find
  51. Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question answering in images. In CVPR, pages 4995–5004.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ates-2022-craft-benchmark-causal-reasoning-about/