Elamur Research⌕
☰
home / papers / 2025 / MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusion

MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusion

Paper · 2025 · Qizhi Pei, Lijun Wu, Zhuoshi Pan et al.

Summary

Most data augmentation for mathematical SFT changes one problem at a time, for example by rewording it or adjusting its difficulty. This ignores how separate problems relate to each other. MathFusion instead builds new training problems by combining pairs of existing ones. Each problem from GSM8K and MATH is paired with its most similar neighbour, measured by embedding similarity from text-embedding-3-large. GPT-4o-mini then merges each pair using one of three strategies:

  • Sequential fusion: the answer to the first problem becomes an input to the second.
  • Parallel fusion: two analogous problems are merged into one task that captures their shared concept.
  • Conditional fusion: a context-dependent problem asks the solver to choose between options.

GPT-4o-mini also writes the solutions. The resulting MathFusionQA dataset was used to fine-tune DeepSeekMath-7B, Mistral-7B and Llama3-8B. Across six in-domain and out-of-domain benchmarks, accuracy rose by 18.0 points on average over standard fine-tuning, using only 45K extra synthetic instructions. The work shows that combining problems across instances is a data-efficient way to improve math reasoning, and it complements existing augmentation methods.

Key points

  • Resource: MathFusionQA (about 60K examples) and the fine-tuned models are released on Hugging Face and GitHub (QizhiPei/MathFusion).
  • Combination with DART-Math: MathFusionQA plus 90K samples from DART-Math-Hard (150K in total) outperforms full DART-Math, which uses 590K samples. This is less than a third of the data.
  • Difficulty: Measured with Mathstral-7B, fused data shows higher instruction-following difficulty (IFD) but lower solution perplexity than the original data.
  • Scaling: On Llama3-8B, performance grows roughly logarithmically as fused data increases from 0 to 22.5K. The gains also carry over to out-of-domain benchmarks.
  • Limitations: Data quality is bounded by the GPT-4o-mini teacher. Fusing three or more problems and scaling the dataset to millions of examples remain unexplored.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations🎓Renmin University of ChinaBeijing · China114 works · 16,118 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China117 works · 11,281 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMathematics · 177 worksComputational Theory and Mathematics · 252 worksApplied Mathematics · 46 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Qizhi Pei, Lijun Wu, Zhuoshi Pan, Yu Li, Honglin Lin, Chenlin Ming, Xin Gao, Conghui He and Rui Yan. 2025. MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusion. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.367

@inproceedings{pei2025mathfusion,
  title = {{MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusion}},
  author = {Qizhi Pei, Lijun Wu, Zhuoshi Pan, Yu Li, Honglin Lin, Chenlin Ming, Xin Gao, Conghui He and Rui Yan},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.367},
  url = {https://doi.org/10.18653/v1/2025.acl-long.367}
}

References

55 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. In EACL (Student Research Workshop), pages 225–237. Association for Computational Linguistics.
  2. Yujie Huang, Yuxiang Zhang, et al. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.
  3. Mehdi Bagherzadeh, Andrei Gurca, and Sabine Brunswicker. 2019. Problem types and open innovation governance modes: A project-level empirical exploration. IEEE Transactions on Engineering Management, 69(2):287–301. find
  4. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1.
  5. Chengtai Cao, Fan Zhou, Yurou Dai, Jianping Wang, and Kunpeng Zhang. 2025. A survey of mix-based data augmentation: Taxonomy, methods, applications, and explainability. ACM Comput. Surv., 57(2):37:1–37:38. find
  6. Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In ACL (Findings), pages 1049–1065. Association for Computational Linguistics.
  7. Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. 2023. TheoremQA: A theorem-driven question answering dataset. In The 2023 Conference on Empirical Methods in Natural Language Processing.
  8. Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. 2024. Key-point-driven data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.02333. find
  9. Jennifer Chu-Carroll, Andrew Beck, Greg Burnham, David OS Melville, David Nachman, A Erdem Özcan, and David Ferrucci. 2024. Beyond llms: Advancing the landscape of complex reasoning. arXiv preprint arXiv:2402.08064. — [arXiv:2402.08064](https://arxiv.org/abs/2402.08064) find
  10. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  12. Yuyang Ding, Xinyu Shi, Xiaobo Liang, Juntao Li, Qiaoming Zhu, and Min Zhang. 2024. Unleashing reasoning capability of llms via scalable question synthesis from scratch. arXiv preprint arXiv:2410.18693. — [arXiv:2410.18693](https://arxiv.org/abs/2410.18693) find
  13. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  14. Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. 2025. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519.
  15. Demi Guo, Yoon Kim, and Alexander M. Rush. 2020. Sequence-level mixed sample data augmentation. In EMNLP (1), pages 5547–5552. Association for Computational Linguistics. find
  16. Hongyu Guo, Yongyi Mao, and Richong Zhang. 2019. Augmenting data with mixup for sentence classification: An empirical study. arXiv preprint arXiv:1905.08941. find
  17. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Xin Jin, Hongyu Zhu, Siyuan Li, Zedong Wang, Zicheng Liu, Chang Yu, Huafeng Qin, and Stan Z Li. 2024. A survey on mixup augmentations and beyond. arXiv preprint arXiv:2409.05202. — [arXiv:2409.05202](https://arxiv.org/abs/2409.05202) find
  18. Jikun Kang, Xin Zhe Li, Xi Chen, Amirreza Kazemi, Qianyi Sun, Boxing Chen, Dong Li, Xu He, Quan He, Feng Wen, et al. 2024. Mindstar: Enhancing math reasoning in pre-trained llms at inference time. arXiv preprint arXiv:2405.16265. — [arXiv:2405.16265](https://arxiv.org/abs/2405.16265) find
  19. Simran Kaur, Simon Park, Anirudh Goyal, and Sanjeev Arora. Instruct-skillmix: A powerful pipeline for llm instruction tuning. In NeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability. find
  20. Komarudin Komarudin, Suherman Suherman, and Anita Anggraini. 2021. Analysis of mathematical concept understanding capabilities: The impact of makerspae stem learning approach models and student learning activities. Journal of Innovation in Educational and Cultural Research, 2(1):35–43. find
  21. Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. 2024a. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706. find
  22. Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. 2024b. Mugglemath: Assessing the impact of query and response augmentation on math reasoning. In ACL (1), pages 10230– 10258. Association for Computational Linguistics. find
  23. Ming Li, Pei Chen, Chenguang Wang, Hongyu Zhao, Yijun Liang, Yupeng Hou, Fuxiao Liu, and Tianyi Zhou. 2024c. Mosaic it: Enhancing instruction tuning with data mosaics. arXiv preprint arXiv:2405.13326. — [arXiv:2405.13326](https://arxiv.org/abs/2405.13326) find
  24. Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2024d. From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning. In NAACL-HLT, pages 7602– 7635. Association for Computational Linguistic…
  25. Honglin Lin, Zhuoshi Pan, Yu Li, Qizhi Pei, Xin Gao, Mengzhang Cai, Conghui He, and Lijun Wu. 2025. Metaladder: Ascending mathematical solution quality via analogical-problem reasoning transfer. arXiv preprint arXiv:2503.14891. find
  26. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations.
  27. Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. 2024. Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold. arXiv preprint arXiv:2406.14532. find
  28. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  29. KV Aditya Srivatsa and Ekaterina Kochmar. 2024. What makes math word problems challenging for llms? In NAACL-HLT (Findings), pages 1138–1148. Association for Computational Linguistics. find
  30. Haoxiong Liu, Yifan Zhang, Yifan Luo, and Andrew Chi-Chih Yao. 2024. Augmenting math word problems via iterative question composing. Preprint, arXiv:2401.09003.
  31. Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. 2024. Mathscale: Scaling instruction tuning for mathematical reasoning. In ICML. OpenReview.net. find
  32. Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. 2024. Step-controlled dpo: Leveraging stepwise error for enhanced mathematical reasoning. arXiv preprint arXiv:2407.00782.
  33. Mistral AI team. 2024. Learning to reason with llms. find
  34. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583.
  35. Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. 2024. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830.
  36. Josh OpenAI, Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  37. Zhuoshi Pan, Yu Li, Honglin Lin, Qizhi Pei, Zinan Tang, Wei Wu, Chenlin Ming, H Vicky Zhao, Conghui He, and Lijun Wu. 2025. Lemma: Learning from errors for mathematical advancement in llms. arXiv preprint arXiv:2503.17439. find
  38. Harsa Wara Prabawa, Rizky Rosjanuardi, and Elah Nurlaelah. 2023. Problem decomposition skills, mathematical maturity, and their relation to mathematics problem-solving in a computer science learning class. Jurnal Kependidikan: Jurnal Hasil Peneli-tian dan Kajian Kepustakaan di Bidang Pendidikan, Pen… find
  39. Sunil Thulasidasan, Gopinath Chennupati, Jeff A. Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. 2019. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In NeurIPS, pages 13888–13899. find
  40. Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. 2024. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. In NeurIPS. find
  41. Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. In The Thirteenth International Conference on Learning Representations. find
  42. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11). find
  43. Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2024a. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. In ICLR. OpenReview.net.
  44. Ming Wang, Yuanzhong Liu, Xiaoyu Liang, Songlian Li, Yijie Huang, Xiaoming Zhang, Sijia Shen, Chaofeng Guan, Daling Wang, Shi Feng, et al. 2024b. Langgpt: Rethinking structured reusable prompt design framework for llms from the programming language. arXiv preprint arXiv:2402.16929. — [arXiv:2402.16929](https://arxiv.org/abs/2402.16929) find
  45. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS.
  46. Xueqing Wu, Yingce Xia, Jinhua Zhu, Lijun Wu, Shufang Xie, Yang Fan, and Tao Qin. 2021. mixseq: A simple data augmentation methodfor neural machine translation. In IWSLT, pages 192–197. Association for Computational Linguistics. find
  47. Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, et al. 2024. Enhancing llm reasoning via critique models with test-time and training-time supervision. arXiv preprint arXiv:2411.16579. — [arXiv:2411.16579](https://arxiv.org/abs/2411.16579) find
  48. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. — Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
  49. Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. 2024. Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06332. find
  50. Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations.
  51. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825.
  52. Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In ICLR (Poster). OpenReview.net.
  53. Rongzhi Zhang, Yue Yu, and Chao Zhang. 2020. Seqmix: Augmenting active sequence labeling via sequence mixup. In EMNLP (1), pages 8566–8579. Association for Computational Linguistics. find
  54. Zhihan Zhang, Tao Ge, Zhenwen Liang, Wenhao Yu, Dian Yu, Mengzhao Jia, Dong Yu, and Meng Jiang. 2024. Learn beyond the answer: Training language models with reflection for mathematical reasoning. In EMNLP, pages 14720–14738. Association for Computational Linguistics. find
  55. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations),…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pei-2025-mathfusion-enhancing-mathematical-problem-solving/