Elamur Research
☰
home / papers / 2021 / Measuring Mathematical Problem Solving With the MATH Dataset

Measuring Mathematical Problem Solving With the MATH Dataset

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2103.03874v2

Summary

Machine learning systems handle routine calculation well but still struggle with real mathematical problem solving. Earlier benchmarks either relied on formal theorem provers or used simple, algorithmically generated exercises. The authors build MATH, a set of 12,500 high-school competition problems across seven subjects, graded on five difficulty levels. Every problem comes with a worked solution written in LaTeX and natural language. Final answers are normalized, so scoring uses exact match rather than fuzzy metrics like BLEU. The authors also release AMPS, a 23GB pretraining corpus that combines Khan Academy exercises with millions of problems generated by Mathematica. Fine-tuned GPT-2 models and few-shot GPT-3 reach only about 3–7% accuracy. For comparison, a computer science PhD student scored roughly 40% and a three-time IMO gold medalist scored 90%. If the log-linear scaling trend holds, a model would need around 10^35 parameters to reach 40%. The authors therefore argue that progress will need new algorithms rather than just bigger models. MATH has since become a widely used benchmark for mathematical reasoning in language models.

Key points

  • MATH dataset: 12,500 problems in subjects such as algebra, geometry and number theory. Each has a difficulty level from 1 to 5 and a full step-by-step solution, and geometry diagrams are written in the Asymptote language.
  • AMPS corpus: over 100,000 Khan Academy problems with step-by-step solutions, plus about 5 million problems from 100 hand-designed Mathematica modules. Pretraining on AMPS let a 0.1B-parameter model match a fine-tuned model about 130× larger.
  • Model accuracy: GPT-2 (1.5B) gets about 15% on the easiest level and about 4% on the hardest. Its confidence scores separate right from wrong answers poorly (AUROC 68.8%).
  • Step-by-step solutions: training on worked solutions helps, but making the model write its own solution before answering lowered accuracy to 5.3%. Even when shown 99% of the correct solution, GPT-2 (0.7B) reaches only about 40%.
  • Scaling: accuracy grows only slowly with model size, so the authors conclude that scaling alone will not solve MATH.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

2042 papers of our library cite it (of the 140673 whose bibliographies we read); 135 of them have a page here, the first 100 below, 1907 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2103.03874 · arXiv 2103.03874

@misc{hendrycks2021measuring,
  title = {{Measuring Mathematical Problem Solving With the MATH Dataset}},
  author = {Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob},
  year = {2021},
  doi = {10.48550/arxiv.2103.03874},
  eprint = {2103.03874},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2103.03874}
}

References

42 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. ArXiv, abs/1905.13319, 2019. find our card
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. ArXiv, abs/2108.07732, 2021. find our card
  3. K. Bansal, S. Loos, Markus N. Rabe, Christian Szegedy, and S. Wilcox. Holist: An environment for machine learning of higher-order theorem proving (extended version). ArXiv, abs/1904.03241, 2019. find
  4. T. Brown, B. Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, G. Krüger, T. Henighan, R. Child, Aditya Ramesh, D. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, E… find
  5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, J. Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea.… find our card
  6. Maxwell Crouse, I. Abdelaziz, Cristina Cornelio, Veronika Thost, L. Wu, Kenneth D. Forbus, and Achille Fokoue. Improving graph neural network representations of logical formulae with subgraph pooling. ArXiv, abs/1911.06904, 2019. — [arXiv:1911.06904](https://arxiv.org/abs/1911.06904) find
  7. J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019. find
  8. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks. ArXiv, abs/2004.10964, 2020. find
  9. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. ArXiv, abs/2006.03654, 2020. find our card
  10. Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. ICLR, 2017. find
  11. Dan Hendrycks, C. Burns, Steven Basart, Andy Zou, Mantas Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. ICLR, 2021. find our card
  12. T. Henighan, J. Kaplan, Mor Katz, Mark Chen, Christopher Hesse, J. Jackson, Heewoo Jun, T. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, B. Mann, A. Radford, Aditya Ramesh, Nick Ryder, D. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative mode… find
  13. Danny Hernandez, J. Kaplan, T. Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv, 2021. find
  14. J. Hernández-Orallo. Beyond the turing test. Journal of Logic, Language and Information, 9:447–466, 2000. find
  15. Jose Hernández-Orallo. A formal definition of intelligence based on an intensional variant of algorithmic complexity. EIS, 1998. find
  16. D. Huang, Shuming Shi, Chin-Yew Lin, J. Yin, and W. Ma. How well do computers solve math word problems? large-scale dataset construction and evaluation. In ACL, 2016. find
  17. Daniel Huang, Prafulla Dhariwal, D. Song, and Ilya Sutskever. Gamepad: A learning environment for theorem proving. ArXiv, abs/1806.00608, 2019. find
  18. J. Kaplan, Sam McCandlish, T. Henighan, T. Brown, Benjamin Chess, R. Child, Scott Gray, A. Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, abs/2001.08361, 2020. find
  19. Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system, 2020. find our card
  20. Guillaume Lample and Franccois Charton. Deep learning for symbolic mathematics. ArXiv, abs/1912.01412, 2020. find
  21. S. Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and Machines, 17:391–444, 2007. find
  22. M. Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. ArXiv, abs/1910.13461, 2020. find our card
  23. Wenda Li, L. Yu, Yuhuai Wu, and Lawrence C. Paulson. Modelling high-level mathematical reasoning in mechanised declarative proofs. ArXiv, abs/2006.09265, 2020. find
  24. Wang Ling, Dani Yogatama, Chris Dyer, and P. Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In ACL, 2017. find
  25. J. Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. LogiQA: A challenge dataset for machine reading comprehension with logical reasoning. In IJCAI, 2020. find our card
  26. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019. find
  27. David A. McAllester. MathZero, the classification problem, and set-theoretic type theory. ArXiv, abs/2005.05512, 2020. — [arXiv:2005.05512](https://arxiv.org/abs/2005.05512) find
  28. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In ACL, 2020. find
  29. Shikhar Murty, Pang Wei Koh, and Percy Liang. Expbert: Representation engineering with natural language explanations. In ACL, 2020. find
  30. Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. ArXiv, abs/2009.03393, 2020. find our card
  31. George Pólya. How to Solve It. Princeton University Press, 1945. find
  32. Markus N. Rabe, Dennis Lee, K. Bansal, and Christian Szegedy. Mathematical reasoning via self-supervised skip-tree training. arXiv: Learning, 2020. find
  33. A. Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR, abs/1511.06434, 2016. find our card
  34. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, M. Matena, Yanqi Zhou, W. Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020. find
  35. D. Saxton, Edward Grefenstette, Felix Hill, and P. Kohli. Analysing mathematical reasoning abilities of neural models. ArXiv, abs/1904.01557, 2019. find our card
  36. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS, 2014. find
  37. Christian Szegedy. A promising path towards autoformalization and general artificial intelligence. In CICM, 2020. find
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/1706.03762, 2017. find
  39. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS, 2019. find our card
  40. Eugene Wigner. The unreasonable effectiveness of mathematics in the natural sciences. Communications in Pure and Applied Mathematics, 1960. find
  41. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… find our card
  42. Yuhuai Wu, M. Rabe, Wenda Li, Jimmy Ba, Roger B. Grosse, and Christian Szegedy. Lime: Learning inductive bias for primitives of mathematical reasoning. ArXiv, abs/2101.06223, 2021. find