Elamur Research⌕
☰
home / papers / 2024 / AI-Assisted Generation of Difficult Math Questions

AI-Assisted Generation of Difficult Math Questions

Paper · 2024 · Vedant Shah, Dingli Yu, Kaifeng Lyu et al.

Summary

LLM training now treats mathematical reasoning as a core capability, but public sources of hard problems are largely used up. Human-written questions are expensive, and purely model-generated ones tend to be repetitive or merely tedious. The authors propose a human-in-the-loop pipeline. A strong LLM first lists the core skills behind problems in the MATH dataset. It then receives random pairs of skills from different subareas and must write a question that requires both. Combining unrelated skills pushes both models and people beyond familiar problem patterns. Multi-turn prompting has the model draft a question, attempt a solution, check the question's validity and produce a final answer. Human experts then select promising items, revise them and supply reference solutions.

The resulting evaluation set, MATH², is markedly harder than MATH for every model tested. Using its items as few-shot examples also improves MATH scores more than using original MATH examples. The approach offers a scalable way to build fresh benchmarks as existing ones saturate and get overfitted.

Key points

  • Skill pairing:
    • Skills come from MATH using the LLM-metacognition method of Didolkar et al. (2024).
    • Each question pairs one pre-algebra/algebra skill with a skill sampled from another MATH section.
  • Five-stage pipeline:
    1. Skill-pair validation.
    2. Question generation.
    3. A deliberately "defeatist" solution attempt.
    4. Question validation (correctness, rigor, clarity).
    5. A final solution using in-context prompting and majority voting.
    • Generation used GPT-4, Claude and Gemini.
  • Human review: Annotators were computer-science graduate students. They often picked questions that models answered wrongly but with many correct ideas, and they made items less brute-force-solvable.
  • Squared difficulty: A model's accuracy on MATH² is roughly the square of its MATH accuracy (e.g., 0.5 → about 0.25). This suggests each item genuinely needs two distinct skills. o1-preview and DeepSeek-R1 distilled models showed the smallest drops.
  • Availability and limitations: The data is available at math-squared.github.io. The main limitation is cost, from heavy API use of frontier models plus manual verification.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Princeton UniversityPrinceton · United States137 works · 53,849 citations🎓Université de MontréalMontreal · Canada96 works · 92,262 citations🎓University of Colorado BoulderBoulder · United States76 works · 51,396 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

Cite

Cite as Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Jiatong Yu, Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora and Anirudh Goyal. 2024. AI-Assisted Generation of Difficult Math Questions. arXiv:2407.21009.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.21009 · arXiv 2407.21009

@misc{shah2024ai,
  title = {{AI-Assisted Generation of Difficult Math Questions}},
  author = {Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Jiatong Yu, Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora and Anirudh Goyal},
  year = {2024},
  doi = {10.48550/arxiv.2407.21009},
  eprint = {2407.21009},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.21009}
}

References

37 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024.
  2. Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. find
  3. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  4. Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  5. Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  6. Gupta, H., Scaria, K., Anantheswaran, U., Verma, S., Parmar, M., Sawant, S. A., Mishra, S., and Baral, C. Targen: Targeted data generation with large language models. arXiv preprint arXiv:2310.17876, 2023. — [arXiv:2310.17876](https://arxiv.org/abs/2310.17876) find
  7. Anthropic. Claude 3.5 sonnet model card addendum. https://www.anthropic.com, 2024. Accessed: November 28, 2024. ↗
  8. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset, 2021.
  9. Arora, S. and Goyal, A. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936, 2023. — [arXiv:2307.15936](https://arxiv.org/abs/2307.15936) find
  10. Honovich, O., Scialom, T., Levy, O., and Schick, T. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689, 2022.
  11. Bowman, S. and etal. Measuring progress on scalable oversight for large language models, 2022.
  12. Huang, Y., Liu, X., Gong, Y., Gou, Z., Shen, Y., Duan, N., and Chen, W. Key-point-driven data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.02333, 2024.
  13. Chan, X., Wang, X., Yu, D., Mi, H., and Yu, D. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094, 2024.
  14. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., La… find
  15. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  16. Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024a.
  17. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626, 2023.
  18. Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024b. find
  19. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stanford, CA, 2000. Morgan Kaufmann. find
  20. Toshniwal, S., Moshkov, I., Narenthiran, S., Gitman, D., Jia, F., and Gitman, I. Openmathinstruct-1: A 1.8 million math instruction tuning dataset. arXiv preprint arXiv:2402.10176, 2024.
  21. Li, C., Wang, W., Hu, J., Wei, Y., Zheng, N., Hu, H., Zhang, Z., and Peng, H. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706, 2024.
  22. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  23. Lu, Z., Zhou, A., Ren, H., Wang, K., Shi, W., Pan, J., Zhan, M., and Li, H. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms. arXiv preprint arXiv:2402.16352, 2024.
  24. Trinh, T. H., Wu, Y., Le, Q. V., He, H., and Luong, T. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476–482, 2024.
  25. Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229, 2024.
  26. Wei, J., Wang, X., Liu, Q., Yang, B., Dong, X., Huang, H., and Wang, W. Chain-of-thought prompting elicits reasoning in large language models. arXiv, abs/2201.11903, 2022. URL https://doi.org/10. 48550/arXiv.2201.11903.
  27. OpenAI. Gpt-4 technical report, 2023.
  28. OpenAI. Openai o1 system card. https://cdn.openai.com/o1-system-card-20241205.pdf, 2024. ↗
  29. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. find
  30. Patel, A., Raffel, C., and Callison-Burch, C. Datadreamer: A tool for synthetic data generation and reproducible llm workflows. arXiv preprint arXiv:2402.10379, 2024. find
  31. Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023.
  32. Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
  33. Yu, Y., Zhuang, Y., Zhang, J., Meng, Y., Ratner, A. J., Krishna, R., Shen, J., and Zhang, C. Large language model as attributed training data generator: A tale of diversity and bias. Advances in Neural Information Processing Systems, 36, 2024.
  34. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
  35. Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023.
  36. Team, G. Gemini: A family of highly capable multimodal models, 2023.
  37. Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Slack, D., Lyu, Q., et al. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332, 2024.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shah-2024-ai-assisted-generation-difficult-math/