Elamur Research⌕
☰
home / papers / 2024 / OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data

OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data

Paper · 2024 · Shubham Toshniwal, Wei Du, Ivan Moshkov et al.

Summary

The best math-reasoning LLMs increasingly rely on synthetic training data that is either kept private or released under restrictive terms, often because GPT models were used to generate it. This hurts reproducibility and trust in benchmark results, and it makes it hard to study how data design choices matter.

The NVIDIA authors run systematic ablations on synthetic SFT data built with the Llama3.1 family. They train Llama3.1-8B-Base and evaluate it on a MATH validation split. Their findings:

  • Compact chain-of-thought solutions work better than verbose ones.
  • A strong teacher produces better data than a weak student at the same data size.
  • Training tolerates a fair share of flawed solutions, so filtering can be imprecise.
  • A larger number of unique questions drives scaling gains.

Based on these findings, they use Llama-3.1-405B-Instruct to solve MATH and GSM8K training problems and to generate similar new problems. They then decontaminate the data against test sets using an LLM-based pipeline and manual review.

Fine-tuning on the resulting OpenMathInstruct-2 corpus raises the 8B base model to 67.8% on MATH. The data, code and models are released under a license that allows commercial use.

Key points

  • Dataset scale: OpenMathInstruct-2 contains 14M question-solution pairs covering about 600K unique questions, 592K of them synthesized. It is roughly 8× larger than OpenMathInstruct-1 (1.8M).
  • Model results:
    • OpenMath2-Llama3.1-8B reaches 67.8% on MATH, compared with 51.9% for Llama3.1-8B-Instruct (+15.9 points). It also beats NuminaMath-7B-CoT on all benchmarks reported.
    • OpenMath2-Llama3.1-70B, trained on a 5M subset, reaches 71.9% (+3.9 points over Llama3.1-70B-Instruct), but improves on only some benchmarks.
  • Ablation numbers:
    • The proposed CoT format scores +3.9% over Llama's format while producing solutions about 40% shorter.
    • Data from the strong teacher scores +7.8% over student-generated data.
    • SFT remains robust with up to 20% low-quality data.
    • Increasing unique questions from 1K to 6.5K gives +10.5% on validation.
  • Scaling: Accuracy keeps rising up to the full 14M with no sign of saturation. Even a 1M fair-downsampled subset already surpasses Llama3.1-8B-Instruct.
  • Resources and evaluation:
    • Data and models are on the Hugging Face nvidia collection; code is in the NeMo-Skills GitHub repo.
    • Decontamination used the lm-sys pipeline.
    • GPT-4o served as the answer judge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputational Theory and Mathematics · 321 works

Cite

Cite as Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan and Igor Gitman. 2024. OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data. arXiv:2410.01560.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.01560 · arXiv 2410.01560

@misc{toshniwal2024openmathinstruct,
  title = {{OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data}},
  author = {Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan and Igor Gitman},
  year = {2024},
  doi = {10.48550/arxiv.2410.01560},
  eprint = {2410.01560},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.01560}
}

References

40 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AIME 2024. https://artofproblemsolving.com/wiki/index.php/2024_AIME_I, 2024. ↗
  2. Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-yeol Ahn. Can we trust the evaluation on ChatGPT? In 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), 2023.
  3. AMC 2023. https://github.com/QwenLM/Qwen2.5-Math/blob/main/evaluation/data/amc23/test.jsonl, 2023. ↗
  4. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An Open Language Model for Mathematics. In ICLR, 2024.
  5. Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q. Tran, and Mehran Kazemi. Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling, 2024.
  6. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, 2024.
  7. Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling Synthetic Data Creation with 1,000,000,000 Personas, 2024.
  8. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. TMLR, 2023. find
  9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168, 2021.
  10. DeepSeek-AI. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence, 2024.
  11. DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model, 2024. find
  12. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving. In ICLR, 2024.
  13. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. In NeurIPS Datasets and Benchmarks, 2021.
  14. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The Curious Case of Neural Text Degeneration. In ICLR, 2020.
  15. Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E. Gonzalez, Koushik Sen, and Ion Stoica. LLM-Assisted Code Cleaning For Training Accurate Code Generators. In ICLR, 2024. find
  16. Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7B Language Models Already Possess Strong Math Capabilities. arXiv preprint arXiv:2403.04706, 2024.
  17. Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7B Language Models Already Possess Strong Math Capabilities, 2024.
  18. Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning. In ACL, 2024. find
  19. Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. NuminaMath: The largest public dataset in AI4Maths with 860k pairs of competition math problems and solutions, 2024. find
  20. Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In ICLR, 2019. find
  21. Meta-AI. The Llama 3 Herd of Models, 2024.
  22. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? In EMNLP, 2022.
  23. Mistral AI. https://mistral.ai/news/mathstral/, 2024. ↗
  24. NVIDIA. Nemotron-4 340B Technical Report, 2024.
  25. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show Your Work: Scratchpads for Intermediate Computation with Language Models, 2021.
  26. Omni-Math. https://omni-math.github.io/, 2024. ↗
  27. OpenAI. Gpt-4 technical report, 2023. URL https://openai.com/research/gpt-4. ↗
  28. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Cr… find
  29. Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning, 2024.
  30. Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset. In NeurIPS Datasets and Benchmarks, 2024. [41] Liang Zeng, Liangjun Zhong, Liang Zhao, Tian-
  31. Pablo Villalobos and David Atkinson. Trad-wen Wei, Liu Yang, Jujie He, Cheng Cheng, Rui ing Off Compute in Training and Inference, Hu, Yang Liu, Shuicheng Yan, Han Fang, and 2023. URL https://epochai.org/blog/Yahui Zhou. Skywork-Math: Data Scaling Laws trading-off-compute-in-training-and-inference.… ↗
  32. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha [42] Bo-Wen Zhang, Yan Yan, Lin Li, and Guang Liu. InfinityMATH: A Scalable Instruction Tuning Chowdhery, and Denny Zhou. Self-consistency Dataset in Programmatic Mathematical Reason-improves chain of thought reasonin… — [arXiv:2408.07089](https://arxiv.org/abs/2408.07089) find
  33. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In ICLR, 2023.
  34. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. HelpSteer2: Open-source dataset for training top-performing reward models, 2024.
  35. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022.
  36. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement, 2024.
  37. Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples, 2023.
  38. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models. In ICLR, 2024.
  39. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MAmmoTH: Building math generalist models through hybrid instruction tuning. In ICLR, 2024.
  40. Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. MAmmoTH2: Scaling Instructions from the Web. arXiv preprint arXiv:2405.03548, 2024. [43] Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele L…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/toshniwal-2024-openmathinstruct-2-accelerating-ai-math/