Elamur Research⌕
☰
home / papers / 2024 / MathScale: Scaling Instruction Tuning for Mathematical Reasoning

MathScale: Scaling Instruction Tuning for Mathematical Reasoning

Paper · 2024 · Zhengyang Tang, Xingxing Zhang, Benyou Wang et al.

Summary

LLMs still have trouble with multi-step math problems. One reason is that the main training sets are small: GSM8K and MATH each have only about 7.5K examples. Earlier augmentation methods such as WizardMath and MetaMath rewrite existing questions, so their output stays close to the seed questions and is hard to scale. MathScale takes a different route. GPT-3.5 first reduces about 20K seed questions to abstract topics and knowledge points. These concepts are linked in a weighted co-occurrence graph, and new problems are generated from combinations of concepts sampled from the graph. The authors present this as similar to how people learn math: first compressing concepts, then making connections between them. The pipeline produced MathScaleQA, which has two million question–answer pairs. LLaMA-2 and Mistral were fine-tuned on it. For a fairer comparison, the authors also built MWPBench, which combines ten datasets from primary school to college level and evaluates them all the same way: zero-shot, greedy decoding and accuracy. MathScale-7B was the strongest model of its size on every MWPBench dataset. This suggests that generating data from concepts scales better than rewriting existing questions.

Key points

  • About 2K topics and 8K knowledge points were extracted from the MWPBench training split; concepts that appeared only once were removed. The graph has three edge types: topic–topic, topic–knowledge point and knowledge point–knowledge point.
  • On MWPBench, MathScale-7B reached 35.0% micro and 37.5% macro average accuracy. That is a relative gain of 42.9% and 43.7% over the best models of the same size.
  • MWPBench adds a new set called CollegeMath, built from nine textbooks with permissive licences (1,281 training and 2,818 test items). Chinese datasets (Math23k, Ape210k, Gaokao) were translated into English, and multiple-choice items were turned into open-answer word problems.
  • GPT-4 judged 26% of a 5K sample of GPT-3.5 solutions to be wrong. Replacing those solutions did not improve the fine-tuned model, so the validation step was removed from the pipeline.
  • On Fresh-GaokaoMath-2023 (30 Gaokao questions from 2023, unlikely to be in any training data), MathScale-7B scored 30.0. WizardMath-7B scored 13.3, MetaMath-7B 16.6, GPT-3.5-Turbo 40.0 and GPT-4 43.3.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksMathematics · 189 worksComputational Theory and Mathematics · 321 worksApplied Mathematics · 52 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 works

Cite

Cite as Zhengyang Tang, Xingxing Zhang, Benyou Wang and Furu Wei. 2024. MathScale: Scaling Instruction Tuning for Mathematical Reasoning. arXiv:2403.02884.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.02884 · arXiv 2403.02884

@misc{tang2024mathscale,
  title = {{MathScale: Scaling Instruction Tuning for Mathematical Reasoning}},
  author = {Zhengyang Tang, Xingxing Zhang, Benyou Wang and Furu Wei},
  year = {2024},
  doi = {10.48550/arxiv.2403.02884},
  eprint = {2403.02884},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.02884}
}

References

32 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022. find
  2. Chern, E., Zou, H., Li, X., Hu, J., Feng, K., Li, J., and Liu, P. Generative ai for math: Abel. https://github.com/GAIR-NLP/abel, 2023. ↗
  3. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  4. Corral, M. CORRAL’S VECTOR CALCULUS. 2008. find
  5. Evans, M. J. and Rosenthal, J. S. Probability and statistics: The science of uncertainty. Macmillan, 2004. find
  6. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. PAL: Program-aided language models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of P… find
  7. Gou, Z., Shao, Z., Gong, Y., Yang, Y., Huang, M., Duan, N., Chen, W., et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452, 2023.
  8. Grinstead, C. M. and Snell, J. L. Grinstead and Snell’s introduction to probability. Chance Project, 2006. find
  9. Guichard, D. Calculus. 2009. find
  10. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
  11. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. find
  12. Kim, Y. and Rush, A. M. Sequence-level knowledge distillation. In Su, J., Duh, K., and Carreras, X. (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317–1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1…
  13. Kuttler, K. and Farah, I. A First Course in Linear Algebra, 2017A version (Lyryx). Lyryx, 2017. find
  14. Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., and Zhang, D. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023.
  15. Selinger, P. Matrix theory and linear algebra, 2018. URL https://www.mathstat.dal.ca/˜selinger/linear-algebra/. An introduction to linear algebra for first or second year university students. Licensed under Creative Commons CC BY 4.0 License. Last updated on October 26, 2018. ↗
  16. Stitz, C. and Zeager, J. Precalculus. Stitz Zeager Open Source Mathematics, 2013. find
  17. TAL. Tal-scq5k, 2023. URL https://github.com/math-eval/TAL-SCQ5K. GitHub repository. ↗
  18. Tall, D. How humans learn to think mathematically: Exploring the three worlds of mathematics. Cambridge University Press, 2013. find
  19. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. ↗
  20. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  21. Trench, W. F. Elementary Differential Equations. Brooks/-Cole Thomson Learning, San Antonio, Texas, USA, 2001. URL http://ramanujan.math.trinity.edu/wtrench/texts/TRENCH_DIFF_EQNS_I.PDF. Free Edition 1.01 (December 2013). ↗
  22. Wallace, T. Beginning and intermediate algebra. 2010. find
  23. Wang, Y., Liu, X., and Shi, S. Deep neural solver for math word problems. In Proceedings of the 2017 conference on empirical methods in natural language processing, pp. 845–854, 2017. find
  24. Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K. R., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.04751, 2023.
  25. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
  26. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824–24837, 2022.
  27. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023.
  28. Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023.
  29. Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023.
  30. Zhang, X., Li, C., Zong, Y., Ying, Z., He, L., and Qiu, X. Evaluating the performance of large language models on gaokao benchmark. 2023.
  31. Zhao, W., Shang, M., Liu, Y., Wang, L., and Liu, J. Ape210k: A large-scale and template-rich dataset of math word problems. arXiv preprint arXiv:2009.11506, 2020. find
  32. Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2024-mathscale-scaling-instruction-tuning-mathematical/