Elamur Research⌕
☰
home / papers / 2023 / Solving Math Word Problems by Combining Language Models With Symbolic Solvers

Solving Math Word Problems by Combining Language Models With Symbolic Solvers

Paper · 2023 · Joy He-Yueya, Gabriel Poesia, Rose E. Wang et al.

Summary

The paper addresses a weakness in tool-augmented LLM approaches to math word problems. Methods like PAL turn a problem into a Python program, which works well for step-by-step procedures. It is awkward, however, when a solution is better stated as relations among unknown quantities. The authors instead have an LLM (Codex, code-davinci-002) translate the problem gradually into variables and equations. A few-shot DECLARATIVE prompt guides this translation, and the resulting equation system goes to SymPy for solving. They also release ALGEBRA, a set of 222 harder word problems taken from two openly licensed algebra textbooks. On GSM8K the method roughly ties the original PAL. On ALGEBRA it beats PAL by about 20 absolute points. Ablations show two things: letting the LLM do the algebra itself sharply hurts accuracy, and building the formalization step by step beats producing it all at once. The work suggests that declarative, incremental representations are a better fit than procedural code for harder problems. The authors see this as useful for educational tools that produce worked solutions.

Key points

  • New dataset ALGEBRA: 222 problems drawn from Basic Algebra with Applications (CC BY-SA) and Elementary Algebra 2e (CC BY). It covers topics up to systems of equations, excluding geometry, graphing and inequalities. The data and prompts are available at github.com/joyheyueya/declarative-math-word-problem.
  • Prompt design: The 3-shot DECLARATIVE prompt mixes natural-language sentences with variable and equation declarations in double square brackets. It follows five principles, including one variable per quantity, equations that use only previously defined variables, and an explicitly stated goal variable.
  • GSM8K (1319 test problems): DECLARATIVE 3-shot + principles + SymPy reaches 69.4%. Original PAL 8-shot reaches 70.2%, PAL 3-shot 73.3%, and original CoT 62.5%.
  • ALGEBRA: The DECLARATIVE method scores about 76.3%. PAL 3-shot scores 56.2%, PAL 8-shot 51.7%, and CoT 45.3–47.9%.
  • Ablations on GSM8K: Solving the equations with the LLM instead of SymPy drops accuracy from 69.4% to 22.4%. One-step (non-incremental) formalization drops it to 57.5%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputational Theory and Mathematics · 321 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 works

Cite

Cite as Joy He-Yueya, Gabriel Poesia, Rose E. Wang and Noah D. Goodman. 2023. Solving Math Word Problems by Combining Language Models With Symbolic Solvers. arXiv:2304.09102.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.09102 · arXiv 2304.09102

@misc{heyueya2023solving,
  title = {{Solving Math Word Problems by Combining Language Models With Symbolic Solvers}},
  author = {Joy He-Yueya, Gabriel Poesia, Rose E. Wang and Noah D. Goodman},
  year = {2023},
  doi = {10.48550/arxiv.2304.09102},
  eprint = {2304.09102},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.09102}
}

References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  2. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  3. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  4. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  5. D. D. Cummins. Children’s interpretations of arithmetic word problems. Cognition and instruction, 8(3):261–289, 1991. find
  6. J. del Olmo-Muñoz, J. A. González-Calero, P. D. Diago, D. Arnau, and M. Arevalillo-Herráez. Intelligent tutoring systems for word problem solving in covid-19 days: could they have been (part of) the solution? ZDM–Mathematics Education, pages 1–14, 2022. find
  7. L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program-aided language models. arXiv preprint arXiv:2211.10435, 2022. find
  8. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
  9. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022.
  10. L. Marecek, M. Anthony-Smith, and A. H. Mathis. Elementary Algebra 2E. OpenStax, 2020. find
  11. A. Meurer, C. P. Smith, M. Paprocki, O. Čertík, S. B. Kirpichev, M. Rocklin, A. Kumar, S. Ivanov, J. K. Moore, S. Singh, et al. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, 2017. find
  12. O. Polozov, E. O’Rourke, A. M. Smith, L. Zettlemoyer, S. Gulwani, and Z. Popović. Personalized mathematical word problem generation. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015. find
  13. N. Pongsakdi, A. Kajamies, K. Veermans, K. Lertola, M. Vauras, and E. Lehtinen. What makes mathematical word problem solving challenging? exploring the roles of word problem characteristics, text comprehension, and arithmetic skills. ZDM, 52:33–44, 2020. find
  14. S. Ritter, J. R. Anderson, K. R. Koedinger, and A. Corbett. Cognitive tutor: Applied research in mathematics education. Psychonomic bulletin & review, 14:249–255, 2007. find
  15. J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
  16. I. G. Zaigralin. Basic Algebra with Applications. Ivan G. Zaigralin, 6 edition, 2018. find
  17. D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, O. Bousquet, Q. Le, and E. Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yueya-2023-solving-math-word-problems-combining/