Training Verifiers to Solve Math Word Problems
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2110.14168v2
Summary
Large language models still make serious mistakes when a math problem takes several reasoning steps. An autoregressive generator can't go back and fix an error once its solution goes wrong. The OpenAI authors tackle this in two ways. First, they release GSM8K, a set of 8.5K grade-school math word problems written by human contributors. Each problem comes with a step-by-step solution in natural language. The wording varies a lot from problem to problem, but solving them needs only basic arithmetic. Second, they propose verification. A finetuned GPT-3 generator samples many candidate solutions at high temperature. A separately trained verifier then scores each one, and the highest-scoring solution is returned. The verifier learns only from whether a sample reached the correct final answer. Compared with plain finetuning, verification gives about the same gain as making the model roughly 30 times larger, and its advantage grows as more training data is added. This matters because it shows that ranking samples with a learned judge can scale better than simply making models bigger. GSM8K later became a widely used benchmark for reasoning in LLMs.
Key points
- GSM8K: 7.5K training and 1K test problems, each needing 2–8 steps. After quality control, fewer than 2% are estimated to have serious errors. It is released at github.com/openai/grade-school-math.
- A 6B generator with a verifier slightly beats a finetuned 175B model. Extrapolating the finetuning results, a model would need about 10^16 parameters to reach an 80% solve rate.
- With the 6B model, accuracy rises up to about 400 sampled completions per problem and then drops, because the verifier starts getting fooled by bad solutions that look convincing. A majority vote among the top-ranked samples adds a further gain.
- Token-level verifiers overfit less than solution-level ones. Residual dropout of 20% improves both finetuning and verification.
- The models are trained to call a calculator through annotations added to the training data, which cuts down arithmetic mistakes.
Cited by
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- Many-Shot In-Context Learning
- Large Language Models for Mathematical Reasoning: Progresses and Challenges
- Yi: Open Foundation Models by 01.AI
- Least-to-most prompting enables complex reasoning in large language models
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- The Falcon Series of Open Language Models
- L-Eval: Instituting Standardized Evaluation for Long Context Language Models
- PaLM 2 Technical Report
- Refusal in Language Models Is Mediated by a Single Direction
- Aya 23: Open Weight Releases to Further Multilingual Progress
- A General Language Assistant as a Laboratory for Alignment
- Llemma: An Open Language Model For Mathematics
- Qwen Technical Report
- Qwen2.5-VL Technical Report
- LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- Sparks of Artificial General Intelligence: Early experiments with GPT-4
- InternLM2 Technical Report
- Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
- AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors
- TheoremQA: A Theorem-driven Question Answering Dataset
- AlphaMath Almost Zero: Process Supervision without Process
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
- Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
- Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier
- DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
- Investigating Data Contamination in Modern Benchmarks for Large Language Models
- RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
- Improving Factuality and Reasoning in Language Models through Multiagent Debate
- LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
- Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
- Mathematical Capabilities of ChatGPT
- Complexity-Based Prompting for Multi-Step Reasoning
- Specializing Smaller Language Models towards Multi-Step Reasoning
- Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
- Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities
- Retrieval-Augmented Generation for Large Language Models: A Survey
- Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
- Gemini: A Family of Highly Capable Multimodal Models
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models
- ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
- ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving
- rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
- LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
- Evaluating Large Language Models: A Comprehensive Survey
- Large Language Model based Multi-Agents: A Survey of Progress and Challenges
- Token-Budget-Aware LLM Reasoning
- ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
- OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
- V-STaR: Training Verifiers for Self-Taught Reasoners
- LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
- Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
- Large Language Models Can Self-Improve
- Large Language Models Cannot Self-Correct Reasoning Yet
- Towards Reasoning in Large Language Models: A Survey
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
- Qwen2.5-Coder Technical Report
- OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
- LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
- A Survey on Large Language Models for Code Generation
- Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations
- Challenges and Applications of Large Language Models
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
- Language Models can Solve Computer Tasks
- Kimi K2: Open Agentic Intelligence
- Understanding the Effects of RLHF on LLM Generalisation and Diversity
- Large language models are zero-shot reasoners
- ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model
- Dicta-LM 3.0: Advancing The Frontier of Hebrew Sovereign LLMs
- CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
- Solving Quantitative Reasoning Problems with Language Models
- Competition-Level Code Generation with AlphaCode
- CMMLU: Measuring massive multitask language understanding in Chinese
- Making Language Models Better Reasoners with Step-Aware Verifier
- Textbooks Are All You Need II: phi-1.5 technical report
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
- CultureLLM: Incorporating Cultural Differences into Large Language Models
- DataComp-LM: In search of the next generation of training sets for language models
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers
- Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
- Let's Verify Step by Step
- RA-DIT: Retrieval-Augmented Dual Instruction Tuning
Cite
@misc{cobbe2021training,
title = {{Training Verifiers to Solve Math Word Problems}},
author = {Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John},
year = {2021},
doi = {10.48550/arxiv.2110.14168},
eprint = {2110.14168},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2110.14168}
}References
- A. Amini, S. Gabriel, P. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019. find our card
- T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. find
- K. Chen, Q. Huang, H. Palangi, P. Smolensky, K. D. Forbus, and J. Gao. Mapping natural-language problems to formal-language solutions using structured neural representations. In ICML, 2020. find
- X. Chen, C. Liang, A. W. Yu, D. Zhou, D. Song, and Q. V. Le. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In International Conference on Learning Representations, 2019. find
- T.-R. Chiang and Y.-N. Chen. Semantically-aligned equation generation for solving and reasoning math word problems. arXiv preprint arXiv:1811.00720, 2018. find
- D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. find our card
- D. Huang, S. Shi, C.-Y. Lin, J. Yin, and W.-Y. Ma. How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 887–896, 2016. find
- D. Huang, J. Liu, C.-Y. Lin, and J. Yin. Neural math word problem solver with reinforcement learning. In Proceedings of the 27th International Conference on Computational Linguistics, pages 213–223, 2018. find
- J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. find
- B. Kim, K. S. Ki, D. Lee, and G. Gweon. Point to the expression: Solving algebraic word problems using the expression-pointer transformer model. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3768–3779, 2020. find
- N. Kushman, Y. Artzi, L. Zettlemoyer, and R. Barzilay. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 271–281, 2014. find
- G. Lample and F. Charton. Deep learning for symbolic mathematics. arXiv preprint arXiv:1912.01412, 2019. find
- S. Li, L. Wu, S. Feng, F. Xu, F. Xu, and S. Zhong. Graph-to-tree neural networks for learning structured input-output translation with applications to semantic parsing and math word problem. EMNLP, 2020. find
- Z. Liang, J. Zhang, J. Shao, and X. Zhang. Mwp-bert: A strong baseline for math word problems, 07 2021. find
- W. Ling, D. Yogatama, C. Dyer, and P. Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146, 2017. find
- J. Liu, L. Cui, H. Liu, D. Huang, Y. Wang, and Y. Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In IJCAI, 2020. find our card
- S.-Y. Miao, C.-C. Liang, and K.-Y. Su. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprint arXiv:2106.15772, 2021. find
- E. Nichols, L. Gao, and R. Gomez. Collaborative storytelling with large-scale neural language models. arXiv preprint arXiv:2011.10208, 2020. — [arXiv:2011.10208](https://arxiv.org/abs/2011.10208) find
- S. Peng, K. Yuan, L. Gao, and Z. Tang. Mathbert: A pre-trained model for mathematical formula understanding. ArXiv, abs/2105.00377, 2021. — [arXiv:2105.00377](https://arxiv.org/abs/2105.00377) find
- S. Roy and D. Roth. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743–1752, Lisbon, Portugal, Sept. 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1202. URL https://aclanthology.org/D… ↗ our card
- J. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034, 2021a. find
- J. T. Shen, M. Yamashita, E. Prihar, N. Heffernan, X. Wu, B. Graff, and D. Lee. Mathbert: A pre-trained language model for general nlp tasks in mathematics education, 08 2021b. find
- I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112, 2014. find
- A. Talmor, J. Herzig, N. Lourie, and J. Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937, 2018. find
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017. find
- A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019. find our card
- Y. Wang, X. Liu, and S. Shi. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 845–854, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1088. URL https://aclanth… find
- Z. Xie and S. Sun. A goal-driven tree-structured neural model for math word problems. In IJCAI, 2019. find
- W. Zhao, M. Shang, Y. Liu, L. Wang, and J. Liu. Ape210k: A large-scale and template-rich dataset of math word problems. arXiv preprint arXiv:2009.11506, 2020. find