Elamur Research⌕
☰
home / papers / 2024 / Augmenting Math Word Problems via Iterative Question Composing

Augmenting Math Word Problems via Iterative Question Composing

Paper · 2024 · Haoxiong Liu, Yifan Zhang, Yifan Luo et al.

Summary

Open-source LLMs still do poorly on competition-level mathematics such as the MATH benchmark when they cannot call external tools. The authors combine two common strategies: continued training on web math text and supervised fine-tuning on synthetic question-answer pairs. Their result is MMIQC, a training mixture with two parts:

  • roughly 1.2M filtered Math StackExchange pairs taken from RedPajama;
  • several kinds of GPT-4-generated data, checked by rejection sampling with GPT-3.5-Turbo.

Their new augmentation technique, Iterative Question Composing (IQC), asks GPT-4 to build progressively harder problems from seed questions over several rounds. A new problem is kept only if GPT-3.5-Turbo's answers match the composed answer. Models fine-tuned on MMIQC beat the same base models fine-tuned on MetaMathQA at every size tested. Qwen-72B-MMIQC reaches 45.0% on MATH, ahead of the earlier open-source best and the original 2023 GPT-4. Results on the 2023 Hungarian national math finals indicate the gains carry over to unseen problems. The work shows that mixing pre-training-style corpora with diverse synthetic data is an effective recipe.

Key points

  • MMIQC components:
    • about 1203.6K Math StackExchange pairs;
    • answer augmentation and question bootstrapping data;
    • similar-problem generation from MATH training seeds;
    • four IQC iterations;
    • a 203.7K subset of MetaMathQA.
  • MATH pass@1 without tools:
    • Mistral-7B-MMIQC: 36.0%;
    • Llemma-34B-MMIQC: 38.6%;
    • DeepSeek-67B-MMIQC: 41.0%;
    • Qwen-72B-MMIQC: 45.0%, which is 8.2 points above the earlier open-source best (DeepSeek-67B-MetaMathQA, 36.8%).
  • Ablation on Mistral-7B: the IQC rounds add about 55.1K samples and raise accuracy from 31.5% to 34.4%. Later rounds still help. Halving MetaMathQA costs only 1.8 points, which suggests the original set is redundant.
  • Contamination check:
    • A 30-gram check found 44 overlaps with the MATH test solutions, compared with 168 between the MATH train and test sets.
    • Manual review showed 43 of the 44 overlaps were shared intermediate steps, not memorized test problems.
  • Dataset and code are public: HuggingFace Vivacem/MMIQC, GitHub iiis-ai/IterativeQuestionComposing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🔬Shanghai Qi Zhi InstituteShanghai · China6 works · 3,008 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMathematics · 177 worksComputational Theory and Mathematics · 252 worksApplied Mathematics · 46 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Haoxiong Liu, Yifan Zhang, Yifan Luo and Andrew Chi-Chih Yao. 2024. Augmenting Math Word Problems via Iterative Question Composing. arXiv:2401.09003.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.09003 · arXiv 2401.09003

@misc{liu2024augmenting,
  title = {{Augmenting Math Word Problems via Iterative Question Composing}},
  author = {Haoxiong Liu, Yifan Zhang, Yifan Luo and Andrew Chi-Chih Yao},
  year = {2024},
  doi = {10.48550/arxiv.2401.09003},
  eprint = {2401.09003},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.09003}
}

References

38 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Azerbayev, Z.; Schoelkopf, H.; Paster, K.; Santos, M. D.; McAleer, S.; Jiang, A. Q.; Deng, J.; Biderman, S.; and Welleck, S. 2023. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.10631.
  2. Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  3. Bi, X.; Chen, D.; Chen, G.; Chen, S.; Dai, D.; Deng, C.; Ding, H.; Dong, K.; Du, Q.; Fu, Z.; et al. 2024. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. arXiv preprint arXiv:2401.02954. find
  4. Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T. J.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.;… — Language Models are Few-Shot Learners
  5. Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y.; Lundberg, S.; et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  6. Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; de Oliveira Pinto, H. P.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; Ray, A.; Puri, R.; Krueger, G.; Petrov, M.; Khlaaf, H.; Sastry, G.; Mishkin, P.; Chan, B.; Gray, S.; Ryder, N.; Pavlov, M.; Power, A.; Kaiser, L.; Bavarian, M.; Winter, C.… — Evaluating Large Language Models Trained on Code
  7. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168.
  8. Computer, T. 2023. RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset. find
  9. Cossu, A.; Tuytelaars, T.; Carta, A.; Passaro, L.; Lomonaco, V.; and Bacciu, D. 2022. Continual pre-training mitigates forgetting in language and vision. arXiv preprint arXiv:2205.09357. — [arXiv:2205.09357](https://arxiv.org/abs/2205.09357) find
  10. Fu, Y.; Peng, H.; Sabharwal, A.; Clark, P.; and Khot, T. 2022. Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720.
  11. Gao, L.; Madaan, A.; Zhou, S.; Alon, U.; Liu, P.; Yang, Y.; Callan, J.; and Neubig, G. 2022. PAL: Program-aided Language Models. arXiv preprint arXiv:2211.10435. find
  12. Gou, Z.; Shao, Z.; Gong, Y.; Yang, Y.; Huang, M.; Duan, N.; Chen, W.; et al. 2023. ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving. arXiv preprint arXiv:2309.17452.
  13. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021a. Measuring Massive Multitask Language Understanding. arXiv preprint arXiv:2009.03300.
  14. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021b. Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS.
  15. Huang, J.; Gu, S. S.; Hou, L.; Wu, Y.; Wang, X.; Yu, H.; and Han, J. 2022a. Large language models can self-improve. arXiv preprint arXiv:2210.11610.
  16. Huang, W.; Abbeel, P.; Pathak, D.; and Mordatch, I. 2022b. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. ArXiv, abs/2201.07207. Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G…
  17. Lewkowycz, A.; Andreassen, A. J.; Dohan, D.; Dyer, E.; Michalewski, H.; Ramasesh, V. V.; Slone, A.; Anil, C.; Schlag, I.; Gutman-Solo, T.; Wu, Y.; Neyshabur, B.; Gur-Ari, G.; and Misra, V. 2022. Solving Quantitative Reasoning Problems with Language Models. In Oh, A. H.; Agarwal, A.; Belgrave, D.; an…
  18. Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2023. Let’s Verify Step by Step. arXiv preprint arXiv:2305.20050.
  19. Liu, B.; Bubeck, S.; Eldan, R.; Kulkarni, J.; Li, Y.; Nguyen, A.; Ward, R.; and Zhang, Y. 2023. TinyGSM: achieving¿ 80% on GSM8k with small language models. arXiv preprint arXiv:2312.09241. Meurer, A.; Smith, C. P.; Paprocki, M.; Čertı́k, O.; Kirpichev, S. B.; Rocklin, M.; Kumar, A.; Ivanov, S.; Mo… find
  20. OpenAI. 2023. GPT-4 Technical Report. ArXiv, abs/2303.08774.
  21. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training language models to follow instruction…
  22. Park, J. S.; O’Brien, J.; Cai, C. J.; Morris, M. R.; Liang, P.; and Bernstein, M. S. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 1–22. find
  23. Paster, K. 2023a. Testing Language Models on a Held-Out High School National Finals Exam. https://huggingface.co/ datasets/keirp/hungarian national hs finals exam. ↗
  24. Paster, K. 2023b. Testing Language Models on a Held-Out High School National Finals Exam. https://huggingface.co/ datasets/keirp/hungarian national hs finals exam. Paster, K.; Dos Santos, . M.; Azerbayev, Z.; and Ba, . J. 2023. OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text. arXi… ↗
  25. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019. Language Models are Unsupervised Multitask Learners. find
  26. OpenAI Blog. Rajbhandari, S.; Rasley, J.; Ruwase, O.; and He, Y. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, 1–16. find
  27. IEEE. Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford alpaca. ↗
  28. Team, G.; Anil, R.; Borgeaud, S.; Wu, Y.; Alayrac, J.-B.; Yu, J.; Soricut, R.; Schalkwyk, J.; Dai, A. M.; Hauth, A.; et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  29. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; Bikel, D.; Blecher, L.; Ferrer, C. C.; Chen, M.; Cucurull, G.; Esiobu, D.; Fernandes, J.; Fu, J.; Fu, W.; Fuller, B.; Gao, C.; Goswami, V.; Goyal, N.; Hartshorn, A.; Hossei… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  30. Touvron, H.; Martin, L.; Stone, K. R.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; Bikel, D. M.; Blecher, L.; Ferrer, C. C.; Chen, M.; Cucurull, G.; Esiobu, D.; Fernandes, J.; Fu, J.; Fu, W.; Fuller, B.; Gao, C.; Goswami, V.; Goyal, N.; Hartshorn, A. S… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  31. Uesato, J.; Kushman, N.; Kumar, R.; Song, F.; Siegel, N.; Wang, L.; Creswell, A.; Irving, G.; and Higgins, I. 2022. Solving math word problems with process- and outcome-based feedback. arXiv:2211.14275.
  32. Wang, P.; Li, L.; Shao, Z.; Xu, R.; Dai, D.; Li, Y.; Chen, D.; Wu, Y.; and Sui, Z. 2023a. Math-Shepherd: A Label-Free Step-by-Step Verifier for LLMs in Mathematical Reasoning. arXiv preprint arXiv:2312.08935.
  33. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q. V.; Chi, E. H.; Narang, S.; Chowdhery, A.; and Zhou, D. 2023b. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations.
  34. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E.; Le, Q.; and Zhou, D. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903.
  35. Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
  36. Yu, L.; Jiang, W.; Shi, H.; Yu, J.; Liu, Z.; Zhang, Y.; Kwok, J. T.; Li, Z.; Weller, A.; and Liu, W. 2023. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models. arXiv preprint arXiv:2309.12284.
  37. Yuan, Z.; Yuan, H.; Li, C.; Dong, G.; Tan, C.; and Zhou, C. 2023. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825.
  38. Yue, X.; Qu, X.; Zhang, G.; Fu, Y.; Huang, W.; Sun, H.; Su, Y.; and Chen, W. 2023. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2024-augmenting-math-word-problems-iterative/