Elamur Research⌕
☰
home / papers / 2020 / Learning advanced mathematical computations from examples

Learning advanced mathematical computations from examples

Paper · 2020 · François Charton, Amaury Hayat and Guillaume Lample

Summary

The paper tests whether a neural network can perform computations that normally depend on advanced mathematical theory, learning only from solved examples. The authors chose three problems. The first is local stability of differential systems, including an estimate of the convergence rate from the Jacobian's eigenvalues (the Spectral Mapping Theorem). The second is controllability, with prediction of a control feedback matrix (the Kalman condition, also in the non-autonomous case). The third is whether linear constant-coefficient PDEs have solutions and whether those solutions decay over time, decided through Fourier analysis. For each problem, they randomly sampled systems, labelled them with the standard symbolic-numeric procedure, and trained sequence-to-sequence transformers with no mathematical knowledge built in. The models answer the qualitative questions almost perfectly and approximate the numerical quantities reasonably well. They stay robust when the test distribution shifts. For the feedback matrix, they sometimes produce valid solutions that differ from the generator's. The work suggests transformers could act as fast learned solvers for mixed symbolic and numerical mathematics, although their internal strategy remains unexplained.

Key points

  • About 50 million problems were generated with no duplicates. A bag-of-words baseline was used to rule out trivial shortcuts or generator artifacts.
  • Numerical prediction tasks reach roughly 65–85% accuracy. Per-task figures fall in a portion of the extracted text that was omitted.
  • Out-of-distribution results for end-to-end stability (Table 7):
    • 96.4% in-distribution
    • 95.7% without trigonometric operators
    • 96.1% with 10% integers
    • 89.5% on longer expressions (n+3 to 3n+3)
    • 78.7% on degree-6 systems, after training only on degrees 2–5
  • Several findings suggest the models do not follow the human procedure:
    • Supplying intermediate results (Jacobians, frequencies) did not improve accuracy.
    • The longer non-autonomous control computation was not harder for the models.
    • Attention weights were close to uniform, which made interpretation difficult.
  • Transformer inference was 10–100 times faster than the classical algorithms on these problems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States224 works · 241,960 citations🏢Meta (United Kingdom)London · United Kingdom60 works · 19,677 citations🎓Rutgers, The State University of New JerseyNew Brunswick · United States42 works · 12,558 citations🎓École nationale des ponts et chausséesChamps-sur-Marne · France5 works · 3,092 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksMathematics · 233 worksPhysics and Astronomy · 70 worksStatistical and Nonlinear Physics · 60 worksModeling and Simulation · 12 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as François Charton, Amaury Hayat and Guillaume Lample. 2020. Learning advanced mathematical computations from examples. arXiv:2006.06462.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2006.06462 · arXiv 2006.06462

@misc{charton2020learning,
  title = {{Learning advanced mathematical computations from examples}},
  author = {François Charton, Amaury Hayat and Guillaume Lample},
  year = {2020},
  doi = {10.48550/arxiv.2006.06462},
  eprint = {2006.06462},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.06462}
}

References

45 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Forough Arabshahi, Sameer Singh, and Animashree Anandkumar. Combining symbolic expressions and black-box function evaluations for training neural programs. In International Conference on Learning Representations, 2018a. find
  2. Forough Arabshahi, Sameer Singh, and Animashree Anandkumar. Towards solving differential equations through neural programming. 2018b. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
  4. Hajer Bahouri, Jean-Yves Chemin, and Raphaël Danchin. Fourier analysis and nonlinear partial differential equations, volume 343. Springer Science & Business Media, 2011. find
  5. Pierre Bernhard, Marc Deschamps, et al. Kalman on dynamics and contro, linear system theory, optimal control, and filter. Technical report, 2017. find
  6. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341, 2019. find
  7. Jean-Michel Coron. Control and nonlinearity, volume 136 of Mathematical Surveys and Monographs. American Mathematical Society, Providence, RI, 2007. ISBN 978-0-8218-3668-2; 0-8218-3668-4. find
  8. George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303–314, 1989. find
  9. Lawrence C Evans. Partial differential equations, volume 19. American Mathematical Soc., 2010. find
  10. Richard Evans, David Saxton, David Amos, Pushmeet Kohli, and Edward Grefenstette. Can neural networks understand logical entailment? arXiv preprint arXiv:1802.08535, 2018.
  11. Joseph Funke, Matthew Brown, Stephen M Erlien, and J Christian Gerdes. Collision avoidance and stabilization for autonomous vehicles in emergency scenarios. IEEE Transactions on Control Systems Technology, 25(4):1204–1216, 2016. find
  12. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9 (8):1735–1780, 1997. find
  13. Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2):251–257, 1991. find
  14. Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural networks, 3(5):551–560, 1990. find
  15. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759, 2016. find
  16. Lukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. CoRR, abs/1511.08228, 2015. find
  17. Rudolf E. Kalman, Yu-Chi Ho, and Kumpati S. Narendra. Controllability of linear dynamical systems. Contributions to Differential Equations, 1:189–213, 1963. ISSN 0589-5839. find
  18. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  19. David Kleinman. An easy way to stabilize a linear constant system. IEEE Transactions on Automatic Control, 15(6):692–692, 1970. find
  20. Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 9(5):987–1000, 1998. find
  21. Isaac E Lagaris, Aristidis C Likas, and Dimitris G Papageorgiou. Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions on Neural Networks, 11(5):1041–1049, 2000. find
  22. Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=S1eZYeHFDS. ↗
  23. Hyuk Lee and In Seok Kang. Neural algorithm for solving differential equations. Journal of Computational Physics, 91(1):110–131, 1990. find
  24. Dahlard L Lukes. Stabilizability and optimal control. Funkcial. Ekvac, 11:39–50, 1968. find
  25. MathWorks. Matlab optimization toolbox (r2019a), 2019. The MathWorks, Natick, MA, USA. find
  26. James Clerk Maxwell. I. on governors. Proceedings of the Royal Society of London, pages 270–283, 1868. find
  27. Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabia… — [doi:10.7717/peerj-cs.103](https://doi.org/10.7717/peerj-cs.103) find
  28. Nicolas Minorsky. Automatic steering tests. Journal of the American Society for Naval Engineers, 42(2):285–310, 1930. find
  29. Philipp Petersen and Felix Voigtlaender. Optimal approximation of piecewise smooth functions using deep relu neural networks. Neural Networks, 108:296–330, 2018. find
  30. Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8: 143–195, 1999. find
  31. Michael Polanyi and Amartya Sen. The Tacit Dimension. University of Chicago Press, 2009. ISBN 9780226672984. find
  32. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. find
  33. Keith Rudd. Solving partial differential equations using artificial neural networks. PhD thesis, Duke University Durham, NC, 2013. find
  34. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations, 2019.
  35. Daniel Selsam, Matthew Lamm, Benedikt Bünz, Percy Liang, Leonardo de Moura, and David L Dill. Learning a sat solver from single-bit supervision. arXiv preprint arXiv:1802.03685, 2018.
  36. Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339–1364, 2018. find
  37. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112, 2014.
  38. Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075, 2015. find
  39. Andrew Trask, Felix Hill, Scott E Reed, Jack Rae, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. In Advances in Neural Information Processing Systems, pages 8035–8044, 2018. find
  40. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010, 2017.
  41. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern,… — SciPy 1.0 — Fundamental Algorithms for Scientific Computing in Python
  42. Eugene P Wigner. The unreasonable effectiveness of mathematics in the natural sciences. communications on pure and applied mathematics, 12:1–14, 1960. find
  43. Wolfram-Research. Mathematica, version 12.0, 2019. find
  44. Gal Yehuda, Moshe Gabel, and Assaf Schuster. It’s not what machines can learn, it’s what we cannot teach. arXiv preprint arXiv:2002.09398, 2020. — [arXiv:2002.09398](https://arxiv.org/abs/2002.09398) find
  45. Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint arXiv:1410.4615, 2014. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/charton-2020-learning-advanced-mathematical-computations-examples/