Elamur Research⌕
☰
home / papers / 2020 / Discovering Symbolic Models from Deep Learning with Inductive Biases

Discovering Symbolic Models from Deep Learning with Inductive Biases

Paper · 2020 · Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia et al.

Summary

Symbolic regression produces compact, interpretable formulas, but its search cost grows exponentially with the number of inputs. Deep networks handle high-dimensional data well but are opaque. The authors combine the two. They train Graph Networks (GNs) whose edge, node and global functions mirror how interacting particles behave. During training, L1 or KL penalties keep latent messages sparse. Each internal function is then fitted separately with eureqa, the resulting expressions are substituted back into the model, and the constants are refitted. This breaks one intractable search into several small ones. On simulated systems, the method recovers known force laws and Hamiltonians, and the learned messages turn out to be linear transforms of the true forces. Applied to a dark matter simulation, it yields a new formula that predicts halo overdensity from neighbouring halos. This formula outperforms the expert-designed one and extrapolates better than the GN it was extracted from. The work suggests a practical route from black-box models to explicit scientific laws.

Key points

  • Pipeline: build a GN with physically motivated inductive biases, train it end-to-end with sparsity regularization, fit symbolic expressions to φe, φv and φu with eureqa, then swap them in and refit the constants. The authors' package PySR could be used in place of eureqa. The models are implemented in PyTorch and PyTorch Geometric.
  • Example of the factorization gain: a joint search over 10^18 candidate equations reduces to about 2 × 10^9 when the two sub-functions are fitted separately.
  • Dark matter overdensity results (mean absolute error):
    • Constant baseline: 0.421
    • Hand-designed formula: 0.121
    • New formula without mass: 0.120 (relative speed acts as a proxy for mass)
    • New formula with mass: 0.0882
  • Generalization test: halos with δ > 1 (20% of the data) were held out. The GN scored 0.0634 on training data but 0.142 on held-out data. The extracted formula scored 0.0811 and 0.0892, respectively.
  • The paper was published at NeurIPS 2020, and code is available at github.com/MilesCranmer/symbolic_deep_learning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksPhysics and Astronomy · 70 worksStatistical and Nonlinear Physics · 60 worksMaterials Science · 58 worksMaterials Chemistry · 58 works

Cite

Cite as Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel and Shirley Ho. 2020. Discovering Symbolic Models from Deep Learning with Inductive Biases. arXiv:2006.11287.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2006.11287 · arXiv 2006.11287

@misc{cranmer2020discovering,
  title = {{Discovering Symbolic Models from Deep Learning with Inductive Biases}},
  author = {Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel and Shirley Ho},
  year = {2020},
  doi = {10.48550/arxiv.2006.11287},
  eprint = {2006.11287},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.11287}
}

References

65 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eugene P. Wigner. The unreasonable effectiveness of mathematics in the natural sciences. Communications on Pure and Applied Mathematics, 13(1):1–14, 1960. doi: 10.1002/cpa.3160130102. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/cpa.3160130102. — [doi:10.1002/cpa.3160130102](https://doi.org/10.1002/cpa.3160130102) ↗
  2. Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. science, 324(5923):81–85, 2009. find
  3. Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Advances in neural information processing systems, pages 2924–2932, 2014. find
  4. Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.
  5. Miles D Cranmer, Rui Xu, Peter Battaglia, and Shirley Ho. Learning Symbolic Physics with Graph Networks. arXiv preprint arXiv:1909.05862, 2019. — [arXiv:1909.05862](https://arxiv.org/abs/1909.05862) find
  6. Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems, pages 4502–4510, 2016. find
  7. Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61–80, 2009. find
  8. Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34 (4):18–42, 2017. find
  9. Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1263–1272. JMLR. org, 2017.
  10. Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach to learning physical dynamics. arXiv preprint arXiv:1612.00341, 2016. find
  11. Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. arXiv preprint arXiv:1806.01242, 2018. — [arXiv:1806.01242](https://arxiv.org/abs/1806.01242) find
  12. Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li F Fei-Fei, Josh Tenenbaum, and Daniel L Yamins. Flexible neural representation for physics prediction. In Advances in Neural Information Processing Systems, pages 8799–8810, 2018. find
  13. Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.01566, 2018. — [arXiv:1810.01566](https://arxiv.org/abs/1810.01566) find
  14. Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. arXiv preprint arXiv:1802.04687, 2018. find
  15. Victor Bapst, Thomas Keck, A Grabska-Barwińska, Craig Donner, Ekin Dogus Cubuk, SS Schoenholz, Annette Obika, AWR Nelson, Trevor Back, Demis Hassabis, et al. Unveiling the predictive power of static structure in glassy systems. Nature Physics, 16(4):448–454, 2020. find
  16. Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter W Battaglia. Learning to simulate complex physics with graph networks. arXiv preprint arXiv:2002.09405, 2020. find
  17. Norman H Packard, James P Crutchfield, J Doyne Farmer, and Robert S Shaw. Geometry from a time series. Physical review letters, 45(9):712, 1980. find
  18. Bryan C. Daniels and Ilya Nemenman. Automated adaptive inference of phenomenological dynamical models. Nature Communications, 6(1):1–8, August 2015. ISSN 2041-1723. find
  19. Miguel Jaques, Michael Burke, and Timothy Hospedales. Physics-as-Inverse-Graphics: Joint Unsupervised Learning of Objects and Physics from Video. arXiv:1905.11169 [cs], May 2019. — [arXiv:1905.11169](https://arxiv.org/abs/1905.11169) find
  20. Michael Janner, Sergey Levine, William T. Freeman, Joshua B. Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning About Physical Interactions with Object-Centric Models. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HJx9EhC9tQ. ↗
  21. Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. PHYRE: A New Benchmark for Physical Reasoning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 5082–50…
  22. Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S. Du, Ken ichi Kawarabayashi, and Stefanie Jegelka. What Can Neural Networks Reason About? In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rJxbJeHFPS. ↗
  23. Samuel S Schoenholz, Ekin D Cubuk, Daniel M Sussman, Efthimios Kaxiras, and Andrea J Liu. A structural approach to relaxation in glassy liquids. Nature Physics, 12(5):469–471, 2016. find
  24. Erik J Bekkers. B-Spline CNNs on Lie Groups. 2019. find
  25. Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data. 2020. find
  26. Rex Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. GNN Explainer: A tool for post-hoc explanation of graph neural networks. arXiv preprint arXiv:1903.03894, 2019. find
  27. Miles Cranmer. PySR: Fast & Parallelized Symbolic Regression in Python/Julia, September 2020. URL https://doi.org/10.5281/zenodo.4052869. — [doi:10.5281/zenodo.4052869](https://doi.org/10.5281/zenodo.4052869) ↗
  28. Dario Izzo and Francesco Biscani. dcgp: Differentiable Cartesian Genetic Programming made easy., May 2020. URL https://doi.org/10.5281/zenodo.3802555. — [doi:10.5281/zenodo.3802555](https://doi.org/10.5281/zenodo.3802555) ↗
  29. Subham Sahoo, Christoph Lampert, and Georg Martius. Learning Equations for Extrapolation and Control. volume 80 of Proceedings of Machine Learning Research, pages 4442–4450, Stockholmsmässan, Stockholm Sweden, 10–15 Jul 2018. PMLR. URL http://proceedings.mlr.press/v80/sahoo18a.html. ↗
  30. Georg Martius and Christoph H. Lampert. Extrapolation and learning equations, 2016. find
  31. Félix-Antoine Fortin, François-Michel De Rainville, Marc-André Gardner, Marc Parizeau, and Christian Gagné. DEAP: Evolutionary algorithms made easy. Journal of Machine Learning Research, 13(70):2171–2175, 2012. find
  32. Silviu-Marian Udrescu and Max Tegmark. AI Feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay2631, 2020.
  33. Tony Worm and Kenneth Chiu. Prioritized Grammar Enumeration: Symbolic Regression by Dynamic Programming. In Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, GECCO ’13, page 1021–1028, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450319638.… — [doi:10.1145/2463372.2463486](https://doi.org/10.1145/2463372.2463486) find
  34. Roger Guimerà, Ignasi Reichardt, Antoni Aguilar-Mogas, Francesco A. Massucci, Manuel Miranda, Jordi Pallarès, and Marta Sales-Pardo. A Bayesian machine scientist to aid in the solution of challenging scientific problems. Science Advances, 6(5), 2020. doi: 10.1126/sciadv. aav6971. URL https://advance… — [doi:10.1126/sciadv](https://doi.org/10.1126/sciadv) find
  35. Matt J. Kusner, Brooks Paige, and José Miguel Hernández-Lobato. Grammar Variational Autoencoder, 2017. find
  36. Li Li, Minjie Fan, Rishabh Singh, and Patrick Riley. Neural-guided symbolic regression with asymptotic constraints. arXiv preprint arXiv:1901.07714, 2019. — [arXiv:1901.07714](https://arxiv.org/abs/1901.07714) find
  37. Gert-Jan Both, Subham Choudhury, Pierre Sens, and Remy Kusters. DeepMoD: Deep learning for Model Discovery in noisy data, 2019. find
  38. Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932–3937, 2016. find
  39. Steven Atkinson, Waad Subber, Liping Wang, Genghis Khan, Philippe Hawi, and Roger Ghanem. Data-driven discovery of free-form governing differential equations. arXiv preprint arXiv:1910.05117, 2019. — [arXiv:1910.05117](https://arxiv.org/abs/1910.05117) find
  40. Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, and Ali Ramadhan. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.04385, 2020. find
  41. Zhao Chen, Yang Liu, and Hao Sun. Deep learning of physical laws from scarce data, 2020. find
  42. Harsha Vaddireddy, Adil Rasheed, Anne E. Staples, and Omer San. Feature engineering and symbolic regression methods for detecting hidden physics from sparse sensor observation data. Physics of Fluids, 32(1):015113, 2020. doi: 10.1063/1.5136351. URL https://doi.org/10.1063/1.5136351. — [doi:10.1063/1.5136351](https://doi.org/10.1063/1.5136351) ↗
  43. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and S… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  44. Matthias Fey and Jan E. Lenssen. Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
  45. Diederik P Kingma and J Adam Ba. A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  46. Alvaro Sanchez-Gonzalez, Victor Bapst, Kyle Cranmer, and Peter Battaglia. Hamiltonian graph networks with ode integrators. arXiv preprint arXiv:1909.12790, 2019. — [arXiv:1909.12790](https://arxiv.org/abs/1909.12790) find
  47. Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, pages 15353–15363, 2019. find
  48. Peter Toth, Danilo Jimenez Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. arXiv preprint arXiv:1909.13789, 2019. — [arXiv:1909.13789](https://arxiv.org/abs/1909.13789) find
  49. Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian Neural Networks. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020. find
  50. Jiang Wang, Simon Olsson, Christoph Wehmeyer, Adrià Pérez, Nicholas E Charron, Gianni De Fabritiis, Frank Noé, and Cecilia Clementi. Machine learning of coarse-grained molecular dynamics force fields. ACS central science, 5(5):755–767, 2019. find
  51. Francisco Villaescusa-Navarro, ChangHoon Hahn, Elena Massara, Arka Banerjee, Ana Maria Delgado, Doogesh Kodi Ramanah, Tom Charnock, Elena Giusarma, Yin Li, Erwan Allys, et al. The Quijote simulations. arXiv preprint arXiv:1909.05273, 2019. — [arXiv:1909.05273](https://arxiv.org/abs/1909.05273) find
  52. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018. find
  53. Scott Dodelson. Modern cosmology. 2003. find
  54. David N Spergel, Licia Verde, Hiranya V Peiris, E Komatsu, MR Nolta, CL Bennett, M Halpern, G Hinshaw, N Jarosik, A Kogut, et al. First-year Wilkinson Microwave Anisotropy Probe (WMAP)* observations: determination of cosmological parameters. The Astrophysical Journal Supplement Series, 148(1):175, 2… find
  55. Planck Collaboration. Planck 2018 results. VI. Cosmological parameters. arXiv e-prints, art. arXiv:1807.06209, July 2018. — [arXiv:1807.06209](https://arxiv.org/abs/1807.06209) find
  56. Carlos S. Frenk, Simon D. M. White, Marc Davis, and George Efstathiou. The Formation of Dark Halos in a Universe Dominated by Cold Dark Matter. Astrophysical Journal, 327:507, April 1988. doi: 10.1086/166213. — [doi:10.1086/166213](https://doi.org/10.1086/166213) find
  57. Gary Marcus. The next decade in AI: four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.06177, 2020.
  58. S. van der Walt, S. C. Colbert, and G. Varoquaux. The NumPy Array: A Structure for Efficient Numerical Computation. Computing in Science Engineering, 13(2):22–30, 2011. find
  59. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern,… — SciPy 1.0 — Fundamental Algorithms for Scientific Computing in Python
  60. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machin… find
  61. Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, and Carol Willing. Jupyter Notebooks – a publishing format for reproducible computational… find
  62. J. D. Hunter. Matplotlib: A 2D Graphics Environment. Computing in Science & Engineering, 9 (3):90–95, 2007. find
  63. Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman, editors, Proceedings of the 9th Python in Science Conference, pages 56 – 61, 2010. doi: 10.25080/Majora-92bf1922-00a.
  64. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16), pages…
  65. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milne. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cranmer-2020-discovering-symbolic-models-deep/