Elamur Research⌕
☰
home / papers / 2025 / Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation

Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation

Paper · 2025 · Sadegh Mahdavi, Muchen Li, Kaiwen Liu et al.

Summary

Training and testing LLMs on Olympiad-level mathematics is hard for two reasons. Expert-written problems with correct solutions are scarce and expensive to produce. Public benchmarks also tend to leak into pre-training corpora, which inflates reported scores. The authors address both problems with the Art of Problem Solving (AoPS) community forum, where users post competition problems and discuss solutions. They built an automated pipeline that uses open-source LLMs to find problems in noisy forum threads, pick out final answers and turn user solutions into clean step-by-step write-ups. Qwen 2.5 72B handles the rewriting step. The result is AoPS-Instruct, a fine-tuning corpus of more than 600K QA pairs. The newest timestamped items form LiveAoPSBench, a benchmark that keeps refreshing as new posts appear. Fine-tuning several models on AoPS-Instruct raised their scores on OlympiadBench, Omni-Math and LiveAoPSBench. Many LLMs scored worse on newer forum problems than on older ones, which suggests memorisation. The work offers a scalable, renewable source of hard math data and a more trustworthy way to evaluate models.

Key points

  • AoPS-Instruct has about 652K Olympiad-level QA pairs, and LiveAoPSBench holds roughly 3.8K timestamped evaluation items. The benchmark excludes earlier problems via substring matching, and it is published at livemathbench.github.io. The paper appeared at ICML 2025.
  • Every tested model scored lower on 2024 problems than on 2023 ones. Relative drops ranged from about 4% (Qwen2.5-Math-7B-Ins) to about 23.6% (Llama-3.2-1B-Ins).
  • Model rankings on LiveAoPSBench closely match those on OlympiadBench, with a Spearman correlation of 0.989.
  • In ablations, rewriting raw forum solutions always improved accuracy, and stronger rewriters such as Qwen-2.5-72B gave larger gains than Llama-3.1-70B. AoPS-Instruct overlaps with other math datasets by at most about 14.1%.
  • The limitations are that the data is text-only, which hurts diagram-heavy geometry, and that proof-based problems cannot yet be evaluated automatically.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputational Theory and Mathematics · 252 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Sadegh Mahdavi, Muchen Li, Kaiwen Liu, Christos Thrampoulidis, Leonid Sigal and Renjie Liao. 2025. Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation. arXiv:2501.14275.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.14275 · arXiv 2501.14275

@misc{mahdavi2025leveraging,
  title = {{Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation}},
  author = {Sadegh Mahdavi, Muchen Li, Kaiwen Liu, Christos Thrampoulidis, Leonid Sigal and Renjie Liao},
  year = {2025},
  doi = {10.48550/arxiv.2501.14275},
  eprint = {2501.14275},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.14275}
}

References

46 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. AIMO. The aimo prize. https://aimoprize.org, November 2023. ↗
  3. AOPS. 2023 amc 12a, and 12b problems. https://artofproblemsolving.com/ wiki/index.php/2023_AMC_12A_Problems, https://artofproblemsolving.com/wiki/ index.php/2023_AMC_12B_Problems, 2023. ↗
  4. AOPS. 2024 aime community page. https://artofproblemsolving.com/community/ c3370201_2024_aime, 2024. ↗
  5. Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S. M., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=4WnqRR915j. ↗
  6. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
  7. Impact Statement This paper aims to advance the field of mathematical reasoning. It is important to note that AoPS-Instruct and LiveAoPSBench contain no personal information. Additionally, we have not identified any societal consequences that require special attention in this context. Cobbe, K., Kos… — Training Verifiers to Solve Math Word Problems
  8. Ding, Y., Shi, X., Liang, X., Li, J., Zhu, Q., and Zhang, M. Unleashing reasoning capability of llms via scalable question synthesis from scratch. arXiv preprint arXiv:2410.18693, 2024. — [arXiv:2410.18693](https://arxiv.org/abs/2410.18693) find
  9. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783x qx q, 2024.
  10. Information Processing Systems, 2022. URL https://openreview.net/forum?id=IFXTZERXdM7. ↗
  11. Li, J., Beeching, E., Tunstall, L., Lipkin, B., Soletskyi, R., Costa Huang, S., Rasul, K., Yu, L., Jiang, A., Shen, Z., Qin, Z., Dong, B., Zhou, L., Fleureau, Y., Lample, G., and Polu, S. Numinamath. https://github.com/ project-numina/aimo-progress-prize/ blob/main/report/numina_dataset.pdf, 2024. ↗
  12. Gao, B., Song, F., Yang, Z., Cai, Z., Miao, Y., Dong, Q., Li, L., Ma, C., Chen, L., Xu, R., et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024.
  13. He, C., Luo, R., Bai, Y., Hu, S., Thai, Z., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., Liu, J., Qi, L., Liu, Z., and Sun, M. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.)…
  14. Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/ alpaca_eval, 5 2023. ↗
  15. Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022.
  16. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
  17. Meurer, A., Smith, C. P., Paprocki, M., Čertı́k, O., Kirpichev, S. B., Rocklin, M., Kumar, A., Ivanov, S., Moore, J. K., Singh, S., Rathnayake, T., Vig, S., Granger, B. E., Muller, R. P., Bonazzi, F., Gupta, H., Vats, S., Johansson, F., Pedregosa, F., Curry, M. J., Terrel, A. R., Roučka, v., Saboo… — [doi:10.7717/peerj-cs.103](https://doi.org/10.7717/peerj-cs.103) find
  18. Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
  19. Miao, S.-Y., Liang, C.-C., and Su, K.-Y. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 975–984, 2020. find
  20. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. find
  21. Mistral. Mathstral blog. https://mistral.ai/ news/mathstral/, 2024. ↗
  22. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
  23. Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830, 2024.
  24. Koncel-Kedziorski, R., Roy, S., Amini, A., Kushman, N., and Hajishirzi, H. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pp. 1152–1157, 2016. find
  25. Paster, K., Santos, M. D., Azerbayev, Z., and Ba, J. Openwebmath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=jKHmjlpViu. ↗
  26. Lewkowycz, A., Andreassen, A. J., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V. V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., and Misra, V. Solving quantitative reasoning problems with language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho,…
  27. Ying, H., Zhang, S., Li, L., Zhou, Z., Shao, Y., Fei, Z., Ma, Y., Hong, J., Liu, K., Wang, Z., et al. Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06332, 2024. find
  28. Selsam, D., de Moura, L., Buzzard, K., Barton, R., Liang, P., Loos, S., and Wiedijk, F. Imo grand challenge. https://imo-grand-challenge.github.io/, 2019. ↗
  29. Yu, L., Jiang, W., Shi, H., YU, J., Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=N8N0hgNDRt. ↗
  30. Shah, V., Yu, D., Lyu, K., Park, S., Yu, J., He, Y., Ke, N. R., Mozer, M., Bengio, Y., Arora, S., et al. Ai-assisted generation of difficult math questions. arXiv preprint arXiv:2407.21009, 2024. find
  31. Yue, X., Zheng, T., Zhang, G., and Chen, W. Mammoth2: Scaling instructions from the web. arXiv preprint arXiv:2405.03548, 2024.
  32. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
  33. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics, 2019. URL https://api.semanticscholar.org/CorpusID:159041722. ↗
  34. Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. find
  35. Team, Q. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https://qwenlm.github.io/blog/qwq-32b-preview/. ↗
  36. Tong, Y., Zhang, X., Wang, R., Wu, R., and He, J. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. Advances in Neural Information Processing Systems, 37:7821–7846, 2024. find
  37. Zhang, B., Zhou, K., Wei, X., Zhao, W. X., Sha, J., Wang, S., and Wen, J.-R. Evaluating and improving tool-augmented computation-intensive math reasoning. arXiv preprint arXiv:2306.02408, 2023a. — [arXiv:2306.02408](https://arxiv.org/abs/2306.02408) find
  38. Zhang, X., Li, C., Zong, Y., Ying, Z., He, L., and Qiu, X. Evaluating the performance of large language models on gaokao benchmark. 2023b.
  39. Zhu, Q., Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., Ma, S., et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024.
  40. Toshniwal, S., Moshkov, I., Narenthiran, S., Gitman, D., Jia, F., and Gitman, I. Openmathinstruct-1: A 1.8 million math instruction tuning dataset. arXiv preprint arXiv:2402.10176, 2024. find
  41. Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877, 2024.
  42. Trinh, T., Wu, Y., Le, Q., He, H., and Luong, T. Solving olympiad geometry without human demonstrations. Nature, 2024. doi: 10.1038/s41586-023-06747-5.
  43. Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024a. find
  44. Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024b.
  45. Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E., and Stoica, I. Rethinking benchmark and contamination for language models with rephrased samples. arXiv preprint arXiv:2311.04850, 2023.
  46. A. More details on LiveAoPSBench A.1. Evaluation pipeline statistics find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mahdavi-2025-leveraging-online-olympiad-level-math/