Elamur Research⌕
☰
home / papers / 2025 / Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision

Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision

Paper · 2025 · Wei Du; Shubham Toshniwal; Branislav Kisacanin; Sadegh Mahdavi; Ivan Moshkov; George Armstrong; Stephen Ge; Edgar Minasyan; Feng Chen; and Igor Gitman

Summary

Existing math-reasoning corpora are usually distilled from a single-mode teacher and focus on competition problems. As a result, they offer little variety in reasoning depth or tool use and cover a narrow range of mathematical topics. NVIDIA researchers address this with Nemotron-Math, built with gpt-oss-120b, which supports three reasoning-effort levels (high, medium, low). Each problem is solved in all three modes, once using Python tool-integrated reasoning (TIR) and once without it. Problems come from AoPS items in OpenMathReasoning and from Math Stack Exchange and MathOverflow (StackExchange-Math). Proof questions and problems the teacher solves too easily are removed. An LLM judge keeps only traces that reach the verified answer.

The authors also introduce a sequential bucketed fine-tuning scheme that trains on progressively longer sequences, from 16K up to 128K tokens. Compared with OpenMathReasoning, the data gives stronger students, and the StackExchange portion improves generalization on HLE-Math. This makes the corpus a practical open resource for training long-context, tool-using math reasoners.

Key points

  • Dataset size and sources: about 7.5M solution traces, up to 128K tokens long, covering 347K problems. These are 85K AoPS problems (filtered from 175K) and 262K StackExchange-Math problems (filtered from 651K). The dataset is released on Hugging Face as nvidia/Nemotron-Math-v2.
  • Difficulty and correctness filtering: a problem is dropped if 16 low-mode attempts reach a pass rate of at least 0.8. Answer checking uses Qwen2.5-32B-Instruct as the judge. Noisy forum answers are replaced by the majority vote of high-mode solutions.
  • Gain over the baseline: in high mode without TIR, Qwen3-30B-A3B's AIME25 pass@1 rises by 13.1% compared with the OpenMathReasoning baseline.
  • Faster long-context training: bucketed training speeds up 128K-context fine-tuning by 2–3×, while staying within 1–3% of the accuracy of full-length joint training.
  • Peak results: Qwen3-8B and Qwen3-30B-A3B converge to similar accuracy. With high mode and Python TIR, both reach 100% maj@16 on AIME 2024 and AIME 2025.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksMathematics · 233 worksComputational Theory and Mathematics · 496 worksApplied Mathematics · 73 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,948 worksEfficient models & compression · 328 works

Cite

Cite as Wei Du; Shubham Toshniwal; Branislav Kisacanin; Sadegh Mahdavi; Ivan Moshkov; George Armstrong; Stephen Ge; Edgar Minasyan; Feng Chen; and Igor Gitman. 2025. Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision. arXiv:2512.15489.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2512.15489 · arXiv 2512.15489

@misc{chen2025nemotron,
  title = {{Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision}},
  author = {Wei Du; Shubham Toshniwal; Branislav Kisacanin; Sadegh Mahdavi; Ivan Moshkov; George Armstrong; Stephen Ge; Edgar Minasyan; Feng Chen; and Igor Gitman},
  year = {2025},
  doi = {10.48550/arxiv.2512.15489},
  eprint = {2512.15489},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2512.15489}
}

References

28 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [1] Art of problem solving. https://artofproblemsolving.com/, 2025. Accessed: 2025-09-25. ↗
  2. [2] Nemo rl: A scalable and efficient post-training library. https://github.com/NVIDIA-NeMo/RL, 2025. GitHub repository. ↗
  3. [3] NeMo-Skills. https://github.com/NVIDIA-NeMo/Skills/, 2025. GitHub repository. ↗
  4. [4] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025.
  5. [5] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022. find
  6. [6] Debrup Das, Debopriyo Banerjee, Somak Aditya, and Ashish Kulkarni. Mathsensei: a tool-augmented large language model for mathematical reasoning. arXiv preprint arXiv:2402.17231, 2024.
  7. [10] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024.
  8. [11] Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. NuminaMath: The largest public dataset in AI4Maths with 860k pairs of competition math problems and solutions, 2024. find
  9. [12] Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383, 2025.
  10. [13] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. find
  11. OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. ↗
  12. [14] Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathcoder2: Better math reasoning from continued pretraining on model-translated mathematical code. arXiv preprint arXiv:2410.08196, 2024.
  13. [15] Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504.16891, 2025.
  14. [7] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452, 2023.
  15. [16] OpenR1 Team. OpenR1 Math 220k, February 2025. URL https://huggingface.co/ datasets/open-r1/OpenR1-Math-220k. Dataset available on Hugging Face. ↗
  16. [8] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  17. [17] Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dan Hendrycks, et al. Humanity’s last exam (hle). https://arxiv.org/abs/2501.14249, 2025. preprin… ↗
  18. [9] Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456, 2025.
  19. [18] Qwen-team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. ↗
  20. [19] Stack Exchange Community. MathOverflow. https://mathoverflow.net/, 2025. Accessed: 2025-10-16. ↗
  21. [20] Stack Exchange Community. Mathematics Stack Exchange. https://math.stackexchange.com/, 2025. Accessed: 2025-10-16. ↗
  22. [21] Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560, 2024.
  23. [22] Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-1: A 1.8 million math instruction tuning dataset. Advances in Neural Information Processing Systems, 37:34737–34774, 2024.
  24. [23] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
  25. [24] Liang Zeng, Liangjun Zhong, Liang Zhao, Tianwen Wei, Liu Yang, Jujie He, Cheng Cheng, Rui Hu, Yang Liu, Shuicheng Yan, Han Fang, and Yahui Zhou. Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models – The Story Goes On, 2024. — Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models — The Story Goes On
  26. [25] Shaowei Zhang and Deyi Xiong. BackMATH: Towards Backward Reasoning for Solving Math Problems Step by Step. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, pages 466–482, 2025. find
  27. [26] Han Zhao, Haotian Wang, Yiping Peng, Sitong Zhao, Xiaoyu Tian, Shuaiting Chen, Yunjie Ji, and Xiangang Li. 1.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Training. arXiv preprint arXiv:2503.19633, 2025. — [arXiv:2503.19633](https://arxiv.org/abs/2503.19633) find
  28. Model Context length TP CP PP ETP EMP Qwen3-30B-A3B 16k 32k 64k 128k 4 4 4 4 2 4 8 8 1 1 1 1 1 1 1 1 4 8 8 8 *Table 6: Training configuration by bucket length.* Model/Context Length 16K 32K 64K 128K Full context 128K Speedup Qwen3-30B-A3B 2.2 *Table 7: Training time breakdown by bucket length and st… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/du-2025-nemotron-math-efficient-long-context/