Elamur Research⌕
☰
home / papers / 2025 / Towards Robust Mathematical Reasoning

Towards Robust Mathematical Reasoning

Paper · 2025 · Thang Luong, Dawsen Hwang, Hoang H Nguyen et al.

Summary

Popular math benchmarks such as GSM8K, MATH and AIME are close to saturation, and most rely on matching a final answer, which can reward lucky guessing rather than sound reasoning. The Google DeepMind team behind Gemini Deep Think built IMO-Bench, an evaluation suite pitched at International Mathematical Olympiad difficulty. It has three parts: short-answer problems, proof-writing tasks, and a proof-grading task. Problems were checked by IMO medalists and mathematicians. Many were altered to resist memorization through paraphrasing, renamed objects, changed numbers, added distractors or equivalent reformulations. Scoring uses LLM-based autograders built on Gemini 2.5 Pro, and the authors show these agree closely with human experts. The IMO-gold Deep Think model clearly beats other systems, especially on proofs, where competing models perform far worse. The suite was used to guide development toward gold-level IMO 2025 performance and is publicly released. It gives researchers a harder and more reliable way to measure genuine mathematical reasoning rather than answer-finding alone.

Key points

  • IMO-AnswerBench contains 400 robustified Olympiad problems with short answers. It has 100 each in Algebra, Combinatorics, Geometry and Number Theory, spread across four levels from pre-IMO to IMO-Hard.
  • IMO-ProofBench contains 60 proof problems: a basic set of 30 and an advanced set of 30. The advanced set includes 18 new problems written by medalists, 6 modified IMO 2024 problems and 6 USAMO 2025 problems. Grading guidelines are included.
  • IMO-GradingBench contains 1000 human-graded solutions to advanced ProofBench problems. On this benchmark, o3 reached the top category accuracy (54.0%), and Deep Think had the lowest MAE (18.4%, where the best achievable is 3.9%).
  • Model results: Gemini Deep Think (IMO gold) scored 80.0% on AnswerBench, which is 6.9 points above the best non-Gemini model and 19.2 points above the best open-weight model. On advanced ProofBench it scored 65.7%, compared with 23.3% and 7.1% for those rivals.
  • Autograders: AnswerAutoGrader matches human judgments with 98.9% accuracy, and ProofAutoGrader correlates strongly with expert scores. Resources are available at imobench.github.io.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputational Theory and Mathematics · 252 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Thang Luong, Dawsen Hwang, Hoang H Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ahn, Junhwi Bae, Xingyou Song, Trieu Hoang Trinh, Quoc V Le and Junehyuk Jung. 2025. Towards Robust Mathematical Reasoning. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1794

@inproceedings{luong2025robust,
  title = {{Towards Robust Mathematical Reasoning}},
  author = {Thang Luong, Dawsen Hwang, Hoang H Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ahn, Junhwi Bae, Xingyou Song, Trieu Hoang Trinh, Quoc V Le and Junehyuk Jung},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1794},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1794}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Introducing claude 4. https://www.anthropic.com/news/claude-4. ↗
  2. Evan Chen. 2023. Guidance for problem captains: Or: how to write an olympiad rubric. https://goo.gle/imo-gold https://web.evanchen.cc/static/usemo/ captain-guidance-usemo.pdf. ↗
  3. Yuri Chervonyi, Trieu H. Trinh, Miroslav Olák, Xiaomeng Yang, Hoang Nguyen, Marcelo Menegali, Junehyuk Jung, Vikas Verma, Quoc V. Le, and Thang Luong. 2025. Gold-medalist performance in solving olympiad geometry with alphageometry2. Preprint, arXiv:2502.03544. find
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems.
  5. Deep Think team. 2025. Try deep think in the gemini app. https://blog.google/products/gemini/ gemini-2-5-deep-think/. ↗
  6. DeepSeek. 2025a. Deepseek-r1-0528 release. https://api-docs.deepseek.com/news/news250528. ↗
  7. DeepSeek. 2025b. Deepseek-v3-0324 release. https://api-docs.deepseek.com/news/news250325. ↗
  8. Jingxuan Fan, Sarah Martinson, Erik Y Wang, Kaylie Hausknecht, Jonah Brenner, Danxian Liu, Nianli Peng, Corey Wang, and Michael P Brenner. 2024. Hardmath: A benchmark dataset for challenging problems in applied mathematics. arXiv preprint arXiv:2410.09988. — [arXiv:2410.09988](https://arxiv.org/abs/2410.09988) find
  9. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, and 1 others. 2024. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985.
  10. Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, and 1 others. 2024. FrontierMath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint arXiv:2411.04872… find
  11. Google DeepMind. 2025. Gemini 2.5 pro. https://deepmind.google/models/gemini/pro/. ↗
  12. Aryan Gulati, Brando Miranda, Eric Chen, Emily Xia, Kai Fronsdal, Bruno de Moraes Dumont, and Sanmi Koyejo. 2024. Putnam-axiom: A functional and static benchmark for measuring higher level mathematical reasoning. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24. find
  13. Simeng Han, Stephen Xia, Grant Zhang, Howard Dai, Chen Liu, Lichang Chen, Hoang Huy Nguyen, Hongyuan Mei, Jiayuan Mao, and R. Thomas McCoy. 2025. Creativity or brute force? using brainteasers as a window into the problem-solving abilities of large language models. Preprint, arXiv:2505.10844. — [arXiv:2505.10844](https://arxiv.org/abs/2505.10844) find
  14. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.
  15. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset.
  16. Kaixuan Huang, Jiacheng Guo, Zihao Li, Xiang Ji, Jiawei Ge, Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, and 1 others. 2025. Math-perturb: Benchmarking llms’ math reasoning abilities against hard perturbations. arXiv preprint arXiv:2502.06453. find
  17. Yichen Huang and Lin F. Yang. 2025. Gemini 2.5 pro capable of winning gold at imo 2025. Preprint, arXiv:2507.15855. — [arXiv:2507.15855](https://arxiv.org/abs/2507.15855) find
  18. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, and 1 others. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787. find
  19. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  20. Thang Luong and Edward Lockhart. 2025. Advanced version of gemini with deep think officially achieves gold-medal standard at the international mathematical olympiad. https://goo.gle/imo-gold. ↗
  21. Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and 1 others. 2022. Lila: A unified benchmark for mathematical reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Langu…
  22. Moonshot AI. 2025. Kimi k2: Open agentic intelligence. https://moonshotai.github.io/ Kimi-K2/. ↗
  23. OpenAI. 2025a. Introducing gpt-5. https://openai.com/index/introducing-gpt-5/. ↗
  24. OpenAI. 2025b. Introducing openai o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/. ↗
  25. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094.
  26. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, and 1 others. 2025. Humanity’s last exam. arXiv preprint arXiv:2501.14249.
  27. Qwen Team. 2025. Qwen3: Think deeper, act faster. https://qwenlm.github.io/blog/qwen3/. ↗
  28. Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, and 1 others. 2024. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402.19450. find
  29. Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. 2024. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476–482. xAI. 2025. Grok 4. https://x.ai/news/grok-4. ↗
  30. Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. 2021. Minif2f: a cross-system benchmark for formal olympiad-level mathematics. arXiv preprint arXiv:2109.00110. \#\# Limitations

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luong-2025-robust-mathematical-reasoning/