Elamur Research⌕
☰
home / papers / 2019 / CoPhy: Counterfactual Learning of Physical Dynamics

CoPhy: Counterfactual Learning of Physical Dynamics

Paper · 2019 · Fabien Baradel, Natalia Neverova, Julien Mille et al.

Summary

The authors introduce a new task: predicting how the outcome of a physical experiment would change if its initial state were altered. The model first observes how a scene evolves (a tower of blocks falls, balls bounce, objects collide). It is then asked to predict where the objects would end up if, for example, one of them were displaced at the start. The main difficulty lies in hidden factors (confounders): mass, friction, gravity. They cannot be seen from a single frame, but they can be inferred from the observed "original" scenario. For this task, the authors created the synthetic 3D benchmark CoPhy and the model CoPhyNet. The model recovers the 3D positions of objects from RGB frames, builds a latent representation of the confounders without direct supervision, and uses it to predict the alternative future. CoPhyNet clearly outperforms baseline methods for direct video prediction, transfers its knowledge to new initial configurations and, according to the authors, outperforms humans. The work is important because it carries counterfactual causal reasoning over from low-dimensional data to video.

Key points

  • CoPhy covers three physical scenarios and about 300 thousand synthetic experiments. Each comes with rendered frames, metadata (positions, angles, sizes) and confounder values (masses, friction, gravity). Each experiment, lasting 6 s, is given as a pair: an original and a modified sequence with shared confounders.
  • Interventions are described using Pearl's do-operator. The parameter distributions in the benchmark are balanced so that the confounders genuinely affect the outcome.
  • The CoPhyNet architecture consists of a CNN for de-rendering (input 224×224), graph-based MLP modules and a GRU, as well as a separate layer that estimates mass and friction. The comparison was made against IN and NPE.
  • On BlocktowerCF, the mean MSE of the 3D pose is: CoPhyNet 0.309 (3→3), 0.298 (new confounders) and 0.504 (transfer from 3 to 4 blocks). For the baselines Copy C and Copy B it is 0.470/0.365/0.754 and 0.601/0.592/0.846, respectively.
  • If the model is trained on ground-truth positions instead of estimated ones, the result is only slightly better (0.294/0.289/0.482). The data are published at projet.liris.cnrs.fr/cophy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States224 works · 241,960 citations🔬Centre National de la Recherche ScientifiqueParis · France173 works · 62,617 citations🎓Simon Fraser UniversityBurnaby · Canada21 works · 6,264 citations🎓Institut National des Sciences Appliquées de LyonLyon · France4 works · 44 citations🔬Laboratoire d'Informatique Fondamentale et Appliquée de ToursTours · France2 works · 45 citations🔬Laboratoire d'Informatique en Images et Systèmes d'InformationVilleurbanne · France2 works · 40 citations

Also on the byline: Institut National des Sciences Appliquées Centre Val de Loire · Center of Innovation in Telecommunications and Integration of Service.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksPhysics and Astronomy · 70 worksStatistical and Nonlinear Physics · 60 works

Cite

Cite as Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori and Christian Wolf. 2019. CoPhy: Counterfactual Learning of Physical Dynamics. arXiv:1909.12000.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1909.12000 · arXiv 1909.12000

@misc{baradel2019cophy,
  title = {{CoPhy: Counterfactual Learning of Physical Dynamics}},
  author = {Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori and Christian Wolf},
  year = {2019},
  doi = {10.48550/arxiv.1909.12000},
  eprint = {1909.12000},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.12000}
}

References

53 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. Phyre: A new benchmark for physical reasoning. In NeurIPS, 2019.
  2. Alexander Balke and Judea Pearl. Counterfactual probabilities: Computational methods, bounds and applications. In UAI, 1994. find
  3. Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori. Object level visual reasoning in videos. In ECCV, 2018. find
  4. Peter W. Battaglia, Jessica B. Hamrick, , and Joshua B. Tenenbaum. Simulation as an engine of physical scene understanding. In PNAS, 2013. find
  5. Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. In NIPS, 2016. find
  6. P.W. Battaglia, J.B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V.F. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, Ç. Gülçehre, F. Song, A.J. Ballard, J. Gilmer, G.E. Dahl, A. Vaswani, K. Allen, C. Nash, V/ Langston, C. Dyer, N. Heess, D. Wierstra, P. Kohli, M. Botvinick… — Relational inductive biases, deep learning, and graph networks
  7. Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach to learning physical dynamics. In ICLR, 2017. find
  8. S. Ehrhardt, A. Monszpart, N. Mitra, and A. Vedaldi. Unsupervised intuitive physics from past experiences. arXiv preprint, arXiv:1905.10793, 2019. — [arXiv:1905.10793](https://arxiv.org/abs/1905.10793) find
  9. Tobias Gerstenberg, Noah D. Goodman, David A. Lagnado, and Joshua A. Tenenbaum. How, whether, why: Causal judgments as counterfactual contrasts. In Annual Conference of the Cognitive Science Society, 2015. find
  10. Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and temporal reasoning. In ICLR, 2019. find
  11. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwiska, Sergio Gmez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, AdriPuigdomnech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil B… find
  12. Oliver Groth, Fabian B. Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision-based physical intuition for generalised object stacking. In ECCV, 2018. find
  13. Kaiming He, Georgia Gkioxari, Piotr Dollr, and Ross Girshick. Mask r-cnn. In ICCV, 2017. find
  14. R. Hu, J. Andreas, Rohrbach R., T. Darrell, and K. Saenko. Learning to reason: End-to-end module networks for visual question answering. In ICCV, 2017. find
  15. D. A. Hudson and C. D. Manning. Compositional attention networks for machine reasooning. In ICLR, 2018. find
  16. Herbert Jaeger. Artificial intelligence: Deep neural reasoning. Nature, 2016. find
  17. Michael Janner, Sergey Levine, William T. Freeman, Joshua B. Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning about physical interactions with object-centric models. In ICLR, 2019. find
  18. J. Johnson, B. Hariharan, L. van der Maaten, J. Hoffman, L. Fei-Fei, C. Zitnick, and Girshick R. Inferring and executing programs for visual reasoning. In ICCV, 2017.
  19. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICML, 2015.
  20. Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
  21. Murat Kocaoglu, Christopher Snyder, Alexandros G. Dimakis, and Sriram Vishwanath. CausalGAN: Learning causal implicit generative models with adversarial training. In ICLR, 2018. find
  22. James R. Kubricht, Keith J. Holyoak, and Hongjing Lu. Intuitive physics: Current research and controversies. Trends in Cognitive Sciences, 2017. find
  23. Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. In Behavioral and Brain Sciences, 2017. find
  24. Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. In ICML, 2016. find
  25. David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests for gan evaluation and causal discovery. In ICLR, 2017. find
  26. David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Leon Bottou. Discovering causal signals in images. In CVPR, 2017. find
  27. J. Mao, C. Gan, P. Kohli, J. Tenenbaum, and J. Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In ICLR, 2019.
  28. G. Martin-Ordas, J. Call, and F. Colmenares. Tubes, tables and traps: great apes solve twofunctionally equivalent trap tasks but show no evidence of transfer across tasks. Animal cognition, 2008. find
  29. Michal Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In ICLR, 2016. find
  30. M. McClooskey, A. Caramazza, and B. Green. Curvilinear motion in the absence of external forces: Naive beliefs about the motion of objects. Journal of Experimental Psychoology: Learning, Memory, and Cognition, 210(4474):1139–1141, 1980. find
  31. M. McClooskey, A. Washburn, and L. Felch. Intuitive physics: the straight-down belief and its origin. Journal of Experimental Psychology: Learning, Memory, and Cognition, 1983. find
  32. M. McCloskey and D. Kohl. Naive physics: The curvilinear impetus principle and its role in interactionos with moving objects. Journal of Experimental Psychoology: Learning, Memory, and Cognition, 9(1):146, 1983. find
  33. A. Michotte. The perception of causality. In Basic Books, 1963. find
  34. Joris M. Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Distinguishing cause from effect using observational data: Methods and benchmarks. JMLR, 2016. find
  35. Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, 2nd edition, 2009. find
  36. Judea Pearl. Theoretical impediments to machine learning with seven sparks from the causal revolution. arXiv preprint arXiv:1801.04016, 2018. find
  37. Judea Pearl and Dana McKenzie. The Book of Why: The New Science of Cause and Effect. Basic Books, 2018. find
  38. E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
  39. Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. In NIPS, 2015. find
  40. Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning. arXiv preprint, arXiv:1803.07616, 2018. find
  41. M. Rojas-Carulla, M. Baroni, and D. Lopez-Paz. Causal discovery using proxy variables. In ICLR Workshop, 2018. find
  42. Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Tim Lillicrap. A simple neural network module for relational reasoning. In NIPS, 2017.
  43. Adam Santoro, Felix Hill, David G. T. Barrett, Ari S. Morcos, and Timothy P. Lillicrap. Measuring abstract reasoning in neural networks. In ICML, 2018.
  44. B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. Mooij. On causal and anticausal learning. In ICML, 2012. find
  45. Peter Spirtes. Introduction to causal inference. JMLR, 2010. find
  46. Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jrgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In ICLR, 2018. find
  47. Petar Velikovi, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Li, and Yoshua Bengio. Graph attention networks. In ICLR, 2018. find
  48. Eric Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied question answering in photorealistic environments with point cloud perception. In CVPR, 2019. find
  49. JIajun Wu, Ilker Yildirim, Joseph J. Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. In NIPS, 2015. find
  50. Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physics via visual de-animation. In NIPS, 2017. find
  51. Tian Ye, Xiaolong Wang, James Davidson, and Abhinav Gupta. Interpretable intuitive physics model. In ECCV, 2018. find
  52. Kexin Yi, Chuang Gan, Yunshu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. In ICLR, 2019. find
  53. David Zheng, Vinson Luo, Jiajun Wu, and Joshua B. Tenenbaum. Unsupervised learning of latent physical properties using perception-prediction networks. UAI, 2018. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baradel-2019-cophy-counterfactual-learning-physical-dynamics/