Elamur Research
☰
home / papers / 2022 / Solving Quantitative Reasoning Problems with Language Models

Solving Quantitative Reasoning Problems with Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2206.14858v2

Summary

General-purpose large language models handle many language tasks well but often fail on college-level maths, science and engineering problems. These problems need correct reading of the question, recall of domain facts, and accurate symbolic and numerical work. The authors present Minerva. They took the 8B, 62B and 540B PaLM models and trained them further on a curated technical corpus built from arXiv preprints and web pages with mathematical content. The pages were processed so that LaTeX and equations survive extraction. The model writes step-by-step solutions in natural language mixed with LaTeX. It uses no calculator or code interpreter. At inference, the final answer is chosen by majority vote over many sampled solutions, which beats greedy decoding by a wide margin. Minerva set new state-of-the-art results on MATH, GSM8k and the STEM part of MMLU using only few-shot prompting. It also solved roughly a third of a new set of undergraduate science problems. The work shows that better data and larger models can greatly improve quantitative reasoning without external tools.

Key points

  • Minerva starts from pretrained PaLM models (8B, 62B, 540B) and continues training on a technical dataset made from arXiv papers and filtered maths web pages.
  • Evaluation is few-shot, without fine-tuning on the benchmarks. The 540B model with majority voting reached about 50% on MATH, about 78.5% on GSM8k and about 75% on MMLU-STEM. These figures come from the full paper because the excerpt you supplied left out the results section, so please check them against the PDF.
  • The authors built a new benchmark of over 200 undergraduate problems from MIT OpenCourseWare (OCW), covering physics, chemistry, biology, economics and other subjects. Minerva answered nearly one third correctly.
  • Three checks found little evidence of memorization:
    • a BLEU-based search of the training data for MATH problems found no matching questions;
    • accuracy was similar on problems with altered numbers or wording;
    • majority-vote accuracy held up after removing samples that closely match the reference solutions.
  • Limitations: there is no automatic way to verify answers, no tool access for heavy arithmetic, and little control over which skills the model learns. It still scores well below humans.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

645 papers of our library cite it (of the 140673 whose bibliographies we read); 69 of them have a page here, 576 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2206.14858 · arXiv 2206.14858

@misc{lewkowycz2022solving,
  title = {{Solving Quantitative Reasoning Problems with Language Models}},
  author = {Lewkowycz, Aitor and Andreassen, Anders and Dohan, David and Dyer, Ethan and Michalewski, Henryk and Ramasesh, Vinay and Slone, Ambrose and Anil, Cem and Schlag, Imanol and Gutman-Solo, Theo and Wu, Yuhuai and Neyshabur, Behnam and Gur-Ari, Guy and Misra, Vedant},
  year = {2022},
  doi = {10.48550/arxiv.2206.14858},
  eprint = {2206.14858},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2206.14858}
}

References

45 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alemi, A. A., Chollet, F., Een, N., Irving, G., Szegedy, C., and Urban, J. (2016). Deepmath - Deep Sequence Models for Premise Selection. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pages 2243–2251, USA. Curran Associates Inc. find
  2. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021). Program synthesis with large language models. arXiv preprint arXiv:2108.07732. find our card
  3. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  4. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C. (2022). Quantifying memorization across neural language models. find
  5. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Till… find our card
  6. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Aust… find
  7. CKE (2021). Skale centylowe wyników - matura 2021. find
  8. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. de Moura, L. M., Kong, S., Avigad, J., van Doorn, F., and von Raumer, J. (2015). The lean theore… find our card
  9. Drori, I., Zhang, S., Shuttleworth, R., Tang, L., Lu, A., Ke, E., Liu, K., Chen, L., Tran, S., Cheng, N., Wang, R., Singh, N., Patti, T. L., Lynch, J., Shporer, A., Verma, N., Wu, E., and Strang, G. (2021). A neural network solves, explains, and generates university math problems by program synthesi… find
  10. Goertzel, Z. A., Chvalovský, K., Jakubuv, J., Olsák, M., and Urban, J. (2021). Fast and slow enigmas and parental guidance. CoRR, abs/2107.06750. find
  11. Grabowski, A., Kornilowicz, A., and Naumowicz, A. (2010). Mizar in a nutshell. J. Formalized Reasoning, 3(2):153–245. find
  12. Han, J. M., Rute, J., Wu, Y., Ayers, E., and Polu, S. (2022). Proof artifact co-training for theorem proving with language models. In International Conference on Learning Representations. find
  13. Harrison, J. (1996). Hol light: A tutorial introduction. In Srivas, M. and Camilleri, A., editors, Formal Methods in Computer-Aided Design, pages 265–269, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  14. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2020). Measuring massive multitask language understanding. CoRR, abs/2009.03300. find our card
  15. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. find our card
  16. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G. v. d., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. (2… find
  17. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2019). The curious case of neural text degeneration. find
  18. Hosseini, M. J., Hajishirzi, H., Etzioni, O., and Kushman, N. (2014). Learning to solve arithmetic word problems with verb categorization. EMNLP, 523533. find our card
  19. Jiang, A. Q., Li, W., Tworkowski, S., Czechowski, K., Odrzygózdz, T., Milos, P., Wu, Y., and Jamnik, M. (2022). Thor: Wielding hammers to integrate language models and automated theorem provers. CoRR, abs/2205.10893. find
  20. Kaliszyk, C., Urban, J., Michalewski, H., and Olšák, M. (2018). Reinforcement learning of theorem proving. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc. find
  21. Koncel-Kedziorski, R., Hajishirzi, H., Sabharwal, A., Etzioni, O., and Ang, S. D. (2015). Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3. find
  22. Kovács, L. and Voronkov, A. (2013). First-order theorem proving and Vampire. In CAV. find
  23. Li, W., Yu, L., Wu, Y., and Paulson, L. C. (2021). Isarstep: a benchmark for high-level mathematical reasoning. In ICLR. find
  24. Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., Hubert, T., Choy, P., d’Autume, C. d. M., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Robson, E. S., Kohli, P.,… find our card
  25. Megill, N. D. and Wheeler, D. A. (2019). Metamath: A Computer Language for Pure Mathematics. Lulu Press, Morrisville, North Carolina. http://us.metamath.org/downloads/metamath.pdf. ↗
  26. Meurer, A., Smith, C. P., Paprocki, M., Čertík, O., Kirpichev, S. B., Rocklin, M., Kumar, A., Ivanov, S., Moore, J. K., Singh, S., Rathnayake, T., Vig, S., Granger, B. E., Muller, R. P., Bonazzi, F., Gupta, H., Vats, S., Johansson, F., Pedregosa, F., Curry, M. J., Terrel, A. R., Roučka, v., Saboo, A… find
  27. Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., Sutton, C., and Odena, A. (2021). Show Your Work: Scratchpads for Intermediate Computation with Language Models. arXiv e-prints, page arXiv:2112.00114. find our card
  28. Otten, J. (2008). leancop 2.0 and ileancop 1.2: High performance lean theorem proving in classical and intuitionistic logic (system descriptions). In Armando, A., Baumgartner, P., and Dowek, G., editors, Automated Reasoning, pages 283–291, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  29. Polu, S., Han, J. M., Zheng, K., Baksys, M., Babuschkin, I., and Sutskever, I. (2022). Formal mathematics statement curriculum learning. CoRR, abs/2202.01344. find our card
  30. Polu, S. and Sutskever, I. (2020). Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.03393. find our card
  31. Rabe, M. N., Lee, D., Bansal, K., and Szegedy, C. (2021). Mathematical reasoning via self-supervised skip-tree training. find
  32. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  33. Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021). Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. find
  34. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  35. Roberts, A., Chung, H. W., Levskaya, A., Mishra, G., Bradbury, J., Andor, D., Narang, S., Lester, B., Gaffney, C., Mohiuddin, A., Hawthorne, C., Lewkowycz, A., Salcianu, A., van Zee, M., Austin, J., Goodman, S., Soares, L. B., Hu, H., Tsvyashchenko, S., Chowdhery, A., Bastings, J., Bulian, J., Garci… find
  36. Schulz, S. (2013). System Description: E 1.8. In McMillan, K., Middeldorp, A., and Voronkov, A., editors, Proc. of the 19th LPAR, volume 8312 of LNCS. Springer. find
  37. Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. (2022). Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990. find our card
  38. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H. S., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhao, V., Zhou, Y.,… find our card
  39. Trinh, T. H. and Le, Q. V. (2018). arXiv:1806.02847. find
  40. A simple method for commonsense reasoning. arXiv preprint Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. find our card
  41. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. (2022). Chain of thought prompting elicits reasoning in large language models. find our card
  42. Welleck, S., Liu, J., Bras, R. L., Hajishirzi, H., Choi, Y., and Cho, K. (2021). Naturalproofs: Mathematical theorem proving in natural language. CoRR, abs/2104.01112. find
  43. Wenzel, M., Paulson, L. C., and Nipkow, T. (2008). The isabelle framework. In Mohamed, O. A., Muñoz, C., and Tahar, S., editors, Theorem Proving in Higher Order Logics, pages 33–38, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  44. Wu, Y., Jiang, A. Q., Li, W., Rabe, M. N., Staats, C., Jamnik, M., and Szegedy, C. (2022). Autoformalization with large language models. CoRR, abs/2205.12615. find our card
  45. Wu, Y., Rabe, M. N., Li, W., Ba, J., Grosse, R. B., and Szegedy, C. (2021). LIME: learning inductive bias for primitives of mathematical reasoning. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, vo… find