Elamur Research
☰
home / papers / 2019 / Analysing Mathematical Reasoning Abilities of Neural Models

Analysing Mathematical Reasoning Abilities of Neural Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1904.01557

Summary

Neural sequence models handle pattern-matching tasks well but often fail at the rule-based, compositional reasoning that school mathematics requires. To measure this gap, the DeepMind authors built a synthetic benchmark of maths questions and answers, both written as plain character strings. Code generates the questions procedurally. It covers algebra, arithmetic, calculus, comparison, measurement, numbers, polynomials and probability, drawing on a national curriculum for pupils up to age 16. Modules can be chained, so the answer to one sub-problem becomes an input to the next, which tests working memory and reuse of learned procedures. Each question type has an interpolation test that matches the training distribution. Separate extrapolation tests use bigger numbers, more terms or longer lists than any training example. The authors compared recurrent models (LSTMs, attentional LSTMs) with the Transformer. The Transformer did best, yet every model struggled to generalize beyond the training range. The benchmark gives a controlled way to diagnose where neural architectures fail at reasoning.

Key points

  • The dataset and generator code are public as mathematics_dataset (github.com/deepmind/mathematics_dataset). Each module is a typed mapping, such as function composition or differentiate, and modules can be composed to make harder questions.
  • Extrapolation splits include add_or_sub_big, mul_big, sort_more, round_number_big and swr_p_sequence_more_samples. Each scales a difficulty axis past what the models saw in training.
  • The Transformer outperformed the LSTM-based models, scoring roughly 76% on interpolation tests and about 50% on extrapolation tests. Among the recurrent models, the attentional model with a bidirectional LSTM encoder performed best.
  • Models did well on simpler tasks such as place value and rounding. They did poorly on tasks that need multi-step intermediate work, such as factorizing numbers into primes and multi-term arithmetic.
  • The authors also tested the Transformer on 40 real high-school exam questions, where it answered 14 correctly.

Note: the passage you pasted skipped the paper's middle, including the results section. I took the accuracy figures, the 14/40 exam score and the strong/weak task examples from my own knowledge of the paper, not from your text, so please check them against the PDF before adding this to the library.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

198 papers of our library cite it (of the 140673 whose bibliographies we read); 22 of them have a page here, 176 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.01557 · arXiv 1904.01557

@misc{saxton2019analysing,
  title = {{Analysing Mathematical Reasoning Abilities of Neural Models}},
  author = {Saxton, David and Grefenstette, Edward and Hill, Felix and Kohli, Pushmeet},
  year = {2019},
  doi = {10.48550/arxiv.1904.01557},
  eprint = {1904.01557},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.01557}
}

References

29 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Miltiadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli, and Charles Sutton. Learning continuous semantic representations of symbolic expressions. arXiv preprint arXiv:1611.01423, 2016. find
  2. Allen Institute for AI. Project Euclid. http://allenai.org/euclid/, 2014. ↗
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. find our card
  4. Richard Evans, David Saxton, David Amos, Pushmeet Kohli, and Edward Grefenstette. Can neural networks understand logical entailment? arXiv preprint arXiv:1802.08535, 2018. find
  5. Alex Graves. Adaptive computation time for recurrent neural networks. arXiv:1603.08983, 2016. find
  6. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471, 2016. find
  7. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
  8. Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 887–… find
  9. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pp. 1988–1997. find
  10. Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. In Advances in neural information processing systems, pp. 190–198, 2015. find
  11. Łukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. arXiv preprint arXiv:1511.08228, 2015. find
  12. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. find
  13. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012. find
  14. Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 271–281, 2014. arXiv preprint Wang Ling, Dani Yoga… find
  15. Gary F Marcus. The algebraic mind: Integrating connectionism and cognitive science. MIT press, 2003. find
  16. Aaron Meurer, Christopher P Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K Moore, Sartaj Singh, et al. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, 2017. find
  17. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015. find
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311–318. Association for Computational Linguistics, 2002. find our card
  19. Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. arXiv preprint arXiv:1806.01822, 2018a. — [arXiv:1806.01822](https://arxiv.org/abs/1806.01822) find
  20. Adam Santoro, Felix Hill, David Barrett, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International Conference on Machine Learning, pp. 4477–4486, 2018b. find
  21. Daniel Selsam, Matthew Lamm, Benedikt Bunz, Percy Liang, Leonardo de Moura, and David L Dill. Learning a sat solver from single-bit supervision. arXiv preprint arXiv:1802.03685, 2018. find
  22. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016. find
  23. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. find our card
  24. Shyam Upadhyay and Ming-Wei Chang. Annotating derivations: A new evaluation strategy and dataset for algebra word problems. arXiv preprint arXiv:1609.07197, 2016. find
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 6000–6010, 2017. find
  26. Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 845–854, 2017. find
  27. Yushi Wang, Jonathan Berant, and Percy Liang. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pp. 1332–1342, 2… find
  28. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015. find our card
  29. Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint arXiv:1410.4615, 2014. find