Elamur Research⌕
☰
home / papers / 2019 / PHYRE: A New Benchmark for Physical Reasoning

PHYRE: A New Benchmark for Physical Reasoning

Paper · 2019 · Anton Bakhtin, Laurens van der Maaten, Justin Johnson et al.

Summary

The paper introduces PHYRE, a benchmark that tests whether machine learning algorithms can reason about physics and transfer what they have learned to new situations. The authors, from Facebook AI Research, built a two-dimensional environment with deterministic Newtonian mechanics: gravity, friction and collisions. Each task in it is a puzzle. In a single move, the agent adds new bodies to the scene so that, once the simulation runs, one specified object touches another. The goal is color-coded in a raster image, so natural language is not needed. Tasks are grouped into templates. This makes it possible to evaluate generalization within a template and across different templates separately. The AUCCESS metric awards more points to agents that solve a task in fewer attempts. The baseline agents tested (random, MEM, DQN and their online variants) fall far short of human performance. They do worst at transferring knowledge to entirely new two-ball templates. The value of the resource is that it offers an open, extensible platform for developing agents that learn quickly from few attempts.

Key points

  • The benchmark has two difficulty tiers. In PHYRE-B the agent places one ball: the action space is continuous and three-dimensional (position and radius). In PHYRE-2B it places two balls, i.e. six dimensions. Each tier contains 25 templates of 100 tasks each.
  • The agent does not see exact coordinates, masses or velocities. Instead, it receives a 256×256 raster in which each cell takes one of 7 values: goal objects, static and dynamic bodies, the agent's body, background. The only success condition is contact between two bodies lasting at least 3 seconds.
  • The setup resembles a contextual bandit but differs in two ways. First, there is an offline training phase with unlimited access to the simulator. Second, after each attempt the agent sees the intermediate states of the simulation.
  • The oracle agent OPTIMAL, ranking fewer than 100,000 actions, solves all PHYRE-B tasks and 95% of PHYRE-2B tasks. Among the regular agents, DQN ranks actions considerably better than MEM, and the online versions (marked "-O") clearly outperform the offline ones.
  • The hardest scenario is cross-template generalization in PHYRE-2B. As directions for future work, the authors name using intermediate observations, forward-prediction models, diversifying attempts, and ICP methods. The code and the game are available at player.phyre.ai.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States224 works · 241,960 citations🏢Meta (United Kingdom)London · United Kingdom60 works · 19,677 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,948 worksEvaluation & benchmarks · 1,394 works

Cite

Cite as Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson and Ross Girshick. 2019. PHYRE: A New Benchmark for Physical Reasoning. arXiv:1908.05656.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1908.05656 · arXiv 1908.05656

@misc{bakhtin2019phyre,
  title = {{PHYRE: A New Benchmark for Physical Reasoning}},
  author = {Anton Bakhtin, Laurens {van der Maaten}, Justin Johnson, Laura Gustafson and Ross Girshick},
  year = {2019},
  doi = {10.48550/arxiv.1908.05656},
  eprint = {1908.05656},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.05656}
}

References

61 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K. Allen, K. Smith, and J. Tenenbaum. The tools challenge: Rapid trial-and-error learning in physical problem solving. In arXiv 1907.09620, 2019. — [arXiv:1907.09620](https://arxiv.org/abs/1907.09620) find
  2. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh. VQA: Visual question answering. In ICCV, 2015. find
  3. P. W. Battaglia, J. B. Hamrick, and J. B. Tenenbaum. Simulation as an engine of physical scene understanding. PNAS, 2013. find
  4. M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 449–458. JMLR. org, 2017. find
  5. C. D. Bird and N. J. Emery. Rooks use stones to raise the water level to reach a floating worm. Current Biology, 19(16):1410–1414, 2009. find
  6. O. Chapelle and L. Li. An empirical evaluation of thompson sampling. In NeurIPS, 2011. find
  7. L. G. Cheke, E. Loissel, and N. S. Clayton. How do children solve aesop’s fable? PloS one, 7 (7):e40574, 2012. find
  8. W. Chu, L. Li, L. Reyzin, and R. Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011. find
  9. E. Davis. Physical reasoning. 2006. find
  10. A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. In Proceedings of the Annual Conference on Robot Learning, pages 1–16, 2017. find
  11. C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. In NeurIPS, 2016. find
  12. O. Groth, F. B. Fuchs, I. Posner, and A. Vedaldi. Shapestacks: Learning vision-based physical intuition for generalised object stacking. In ECCV, 2018. find
  13. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
  14. C. Heinze-Deml, J. Peters, and N. Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference, 6(2), 2018. find
  15. M. Henaff, J. Zhao, and Y. LeCun. Prediction under uncertainty with error-encoding networks. In arXiv 1711.04994, 2017. — [arXiv:1711.04994](https://arxiv.org/abs/1711.04994) find
  16. R. Hu, J. Andreas, M. Rohrbach, T. Darrell, and K. Saenko. Learning to reason: End-to-end module networks for visual question answering. In ICCV, 2017. find
  17. D. A. Hudson and C. D. Manning. Compositional attention networks for machine reasoning. In ICLR, 2018.
  18. M. Janner, S. Levine, W. T. Freeman, J. B. Tenenbaum, C. Finn, and J. Wu. Reasoning about physical interactions with object-oriented prediction and planning. In ICLR, 2019. find
  19. S. A. Jelbert, A. H. Taylor, L. G. Cheke, N. S. Clayton, and R. D. Gray. Using the aesop’s fable paradigm to investigate causal understanding of water displacement by new caledonian crows. PloS one, 9(3):e92895, 2014. find
  20. J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. Zitnick, and R. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
  21. J. Johnson, B. Hariharan, L. van der Maaten, J. Hoffman, L. Fei-Fei, C. Zitnick, and R. Girshick. Inferring and executing programs for visual reasoning. In ICCV, 2017.
  22. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  23. J. Langford and T. Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. In NeurIPS, 2008. find
  24. A. Lerer, S. Gross, and R. Fergus. Learning physical intuition of block towers by example. In ICML, 2016. find
  25. L. Li, W. Chu, J. Langford, and R. E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661–670. ACM, 2010. find
  26. W. Li, A. Leonardis, and M. Fritz. To fall or not to fall: A visual approach to physical stability prediction. In arXiv:1604.00066, 2016. — [arXiv:1604.00066](https://arxiv.org/abs/1604.00066) find
  27. W. Li, A. Leonardis, and M. Fritz. Visual stability prediction and its application to manipulation. In ICRA, 2017. find
  28. I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. find
  29. J. Mao, C. Gan, P. Kohli, J. Tenenbaum, and J. Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In ICLR, 2019.
  30. G. Martin-Ordas, J. Call, and F. Colmenares. Tubes, tables and traps: great apes solve two functionally equivalent trap tasks but show no evidence of transfer across tasks. Animal cognition, 11(3):423–430, 2008. find
  31. M. McCloskey and D. Kohl. Naive physics: The curvilinear impetus principle and its role in interactions with moving objects. Journal of Experimental Psychology: Learning, Memory, and Cognition, 9(1):146, 1983. find
  32. M. McCloskey, A. Caramazza, and B. Green. Curvilinear motion in the absence of external forces: Naive beliefs about the motion of objects. Science, 210(4474):1139–1141, 1980. find
  33. M. McCloskey, A. Washburn, and L. Felch. Intuitive physics: the straight-down belief and its origin. Journal of Experimental Psychology: Learning, Memory, and Cognition, 9(4):636, 1983. find
  34. M. Mirza, A. Courville, and Y. Bengio. Generalizable features from unsupervised learning. In ICLR, 2017. find
  35. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning.… find
  36. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML, 2016. find
  37. E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
  38. J. Peters, J. Mooij, D. Janzing, and B. Schölkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15:2009–2053, 2014. find
  39. L. Piloto, A. Weinstein, A. Ahuja, M. Mirza, G. Wayne, D. Amos, C.-c. Hung, and M. Botvinick. Probing physics knowledge using tools from developmental psychology. arXiv preprint arXiv:1804.01128, 2018. — [arXiv:1804.01128](https://arxiv.org/abs/1804.01128) find
  40. D. Povinelli. Folk physics for apes: The chimpanzee’s theory of how the world works. 2000. find
  41. R. Riochet, M. Y. Castro, M. Bernard, A. Lerer, R. Fergus, V. Izard, and E. Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning. In arXiv 1803.07616, 2018. find
  42. C. Riquelme, G. Tucker, and J. Snoek. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. arXiv preprint arXiv:1802.09127, 2018. — [arXiv:1802.09127](https://arxiv.org/abs/1802.09127) find
  43. A. Santoro, D. Raposo, D. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems, pages 4967–4976, 2017.
  44. M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A platform for embodied ai research. arXiv preprint arXiv:1904.01201, 2019.
  45. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. In arXiv 1707.06347, 2017. find
  46. M. Smith. Running the table: An AI for computer billiards. In American Association for Artificial Intelligence, 2006. find
  47. N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995, 2009. — [arXiv:0912.3995](https://arxiv.org/abs/0912.3995) find
  48. A. H. Taylor, G. R. Hunt, F. S. Medina, and R. D. Gray. Do new caledonian crows solve physical problems through causal reasoning? Proceedings of the Royal Society B: Biological Sciences, 276(1655):247–254, 2008. find
  49. I. Teschke and S. Tebbich. Physical cognition and tool-use: performance of darwin’s finches in the two-trap tube task. Animal Cognition, 14(4):555–563, 2011. find
  50. E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In Proceedings of the International Conference on Intelligent Robots and Systems, 2012. find
  51. V. Vapnik and A. Vashist. A new learning paradigm: Learning using privileged information. Neural Networks, 22(5-6):544–557, 2009. find
  52. E. Visalberghi and L. Limongelli. Lack of comprehension of cause effect relations in tool-using capuchin monkeys (cebus apella). Journal of Comparative Psychology, 108(1):15, 1994. find
  53. E. Visalberghi and L. Limongelli. Acting and understanding: Tool use revisited through the minds of capuchin monkeys. Reaching into thought: The minds of the great apes, pages 57–79, 1996. find
  54. E. Visalberghi and L. Trinca. Tool use in capuchin monkeys: Distinguishing between performing and understanding. Primates, 30(4):511–521, 1989. find
  55. F. Wilcoxon. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80–83, 1945. find
  56. T. Winograd. Procedures as a representation for data in a computer program for understanding natural language. Technical report, MIT AI, 1971. find
  57. J. Wu, I. Yildirim, J. J. Lim, B. Freeman, and J. Tenenbaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. In NeurIPS, 2015. find
  58. J. Wu, E. Lu, P. Kohli, B. Freeman, and J. Tenenbaum. Learning to see physics via visual de-animation. In NeurIPS, 2017. find
  59. Y. Wu, Y. Wu, G. Gkioxari, and Y. Tian. Building generalizable agents with a realistic and rich 3D environment. In arXiv:1801.02209, 2018. — [arXiv:1801.02209](https://arxiv.org/abs/1801.02209) find
  60. T. Xue, J. Wu, K. Bouman, and B. Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In NeurIPS, 2016. find
  61. R. Zhang, J. Wu, C. Zhang, W. Freeman, and J. Tenenbaum. A comparative evaluation of approximate probabilistic simulation and deep neural networks as accounts of human physical scene understanding. In CogSci, 2016. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bakhtin-2019-phyre-new-benchmark-physical-reasoning/