MathPrompter: Mathematical Reasoning using Large Language Models
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.acl-industry.4.pdf
Summary
Large language models often get multi-step arithmetic word problems wrong, and they give users no sign of how far an answer can be trusted. Earlier Zero-shot-CoT prompting also never checks whether the intermediate reasoning is correct. MathPrompter addresses both problems by copying the way students double-check their work. First, the numbers in a question are replaced with variables to produce an algebraic template. Next, the LLM is prompted to solve that template in several independent ways, here as an algebraic expression and as a Python function. These candidate solutions are run with Python's eval() on random variable values to see whether they agree. When they match, the original numbers are plugged back in. The whole cycle runs about five times, and the most frequent answer is kept. With GPT-3 DaVinci (175B) on the MultiArith dataset, accuracy rises from 78.7% to 92.5% compared with Zero-shot-CoT. That is comparable to 540B-parameter models and the best Few-shot-CoT setups, even though MathPrompter needs no worked examples. How well the solutions agree also gives a rough measure of confidence.
Key points
- The pipeline has four stages: (I) build an algebraic template by mapping numbers to variables, e.g. {A:5, B:15, C:8}; (II) Math-prompts that ask for an algebraic expression and a Python function; (III) compute verification with eval() on random inputs; (IV) a majority vote over N ≈ 5 repetitions.
- MultiArith accuracy, mostly 175B models: Zero-shot 17.7%, Zero-shot-CoT (Kojima et al., 2022) 78.7%, MathPrompter 92.5%.
- Larger and few-shot reference points: Zero-shot-CoT + self-consistency with PaLM 540B reaches 89.0%, and Few-Shot-CoT with 8 examples reaches 93.0%.
- Compared with CoT text, the Python and algebraic outputs need fewer tokens. Running the arithmetic in code avoids cases where the reasoning is right but the final calculation is wrong.
- Limitations: both solution types can agree on the same wrong answer, as in a commonsense error on a candy-box problem. The method was tested on MultiArith only.
Cited by
- Large Language Models for Mathematical Reasoning: Progresses and Challenges
- Cutting Through the Noise: Boosting LLM Performance on Math Word Problems
- Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
- Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
- A Comprehensive Overview of Large Language Models
- Reasoning with Language Model Prompting: A Survey
- ReFT: Reasoning with Reinforced Fine-Tuning
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges
- ToolQA: A Dataset for LLM Question Answering with External Tools
Cite
@inproceedings{imani2023mathprompter,
title = {{MathPrompter: Mathematical Reasoning using Large Language Models}},
author = {Imani, Shima and Du, Liang and Shrivastava, Harsh},
booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)},
year = {2023},
doi = {10.18653/v1/2023.acl-industry.4},
eprint = {2303.05398},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/2023.acl-industry.4}
}References
- MathPrompter. Mapping = {A:14, B:5, C:6} Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33… find