Elamur Research⌕
☰
home / papers / 2025 / Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning

Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning

Paper · 2025 · Joykirat Singh, Akshay Nambi and Vibhav Vineet

Summary

Most benchmarks for math word problems (MWPs) check only whether an LLM reaches the right final answer. They largely ignore whether the model can spot and repair flawed reasoning, which matters for tutoring and other educational uses. The authors built MWP-MISTAKE, a benchmark that pairs each problem with correct and deliberately faulty solution chains. Problems come from SVAMP, GSM-8K, MATH, MATHBENCH and JEEBENCH and range from middle-school to college difficulty. Faulty chains come from two sources:

  • six programmatic perturbation rules modelled on typical student errors;
  • chain-of-thought outputs from small models that ended in wrong answers, checked by human annotators.

Twelve models were tested under few-shot prompting. These included O1, GPT-4o, GPT-4, GPT-3.5Turbo, Claude, Llama, Phi, Mixtral and math-tuned variants. Each model had to judge a given solution, fix it if needed, and give the final answer.

GPT-4o performed best, but all models had trouble detecting even simple mistakes and were weak at correcting them. Models often still reached correct answers, which points to memorization or data contamination. Scores dropped on the newer datasets. The work shows that answer accuracy alone overstates how well LLMs actually reason.

Key points

  • Dataset size: MWP-MISTAKE has 535 base questions and about 4,961 reasoning instances. It is split into a "Default" part (gold plus rule-based errors) and an "SLM reason" part (errors from Llama-2-7b-chat, Mixtral-8x7B and Phi-3-mini).
  • Perturbation rules: the six rules shuffle numbers, replace numbers with random values (0–100), swap operators, insert an irrelevant step, reorder steps, or delete a step. Each question therefore yields one correct and six altered chains.
  • Gold reasoning: where a source had no reference solutions, GPT-4 wrote them and humans verified them. Five volunteers validated the incorrect chains produced by the small models.
  • O1 results: O1 reaches about 90% accuracy on reasoning and final answers for complex tasks, yet remains weak at detecting mistakes.
  • Contamination and generalization: GPT-4's results on GSM-8K suggest contamination, and performance drops on the more recent MATHBENCH and JEEBENCH. The data and code are released at aka.ms/MWP-MISTAKE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMathematics · 177 worksApplied Mathematics · 46 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Joykirat Singh, Akshay Nambi and Vibhav Vineet. 2025. Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1313

@inproceedings{singh2025exposing,
  title = {{Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning}},
  author = {Joykirat Singh, Akshay Nambi and Vibhav Vineet},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1313},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1313}
}

References

42 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. OpenAI Platform. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Qin Cai, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen… — Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
  2. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024b. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.
  3. Anthropic. 2024. Introducing the next generation of Claude \\ Anthropic. find
  4. Daman Arora, Himanshu Gaurav Singh, and Mausam. 2023. Have llms advanced enough? a challenging problem solving benchmark for large language models. Preprint, arXiv:2305.15074. find
  5. Wei Chen and Zhiyuan Li. 2024. Octopus v4: Graph of language models. arXiv preprint arXiv:2404.19296. — [arXiv:2404.19296](https://arxiv.org/abs/2404.19296) find
  6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021a. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021b. Training Verifiers to Solve Math Word Problems. arXiv preprint. ArXiv:2110.14168 [cs].
  8. Aniruddha Deb, Neeva Oza, Sarthak Singla, Dinesh Khandelwal, Dinesh Garg, and Parag Singla. 2023. Fill in the blank: Exploring and enhancing llm capabilities for backward reasoning in math word problems. Preprint, arXiv:2310.01991. find
  9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Wensheng Gan, Zhenlian Qi, Jiayang Wu, and Jerry Chun-Wei Lin. 2023. Large language models in education: Vision and opportunities. Preprint, arXiv:2311.13160. — [arXiv:2311.13160](https://arxiv.org/abs/2311.13160) find
  11. Shahriar Golchin and Mihai Surdeanu. 2024. Time travel in llms: Tracing data contamination in large language models. Preprint, arXiv:2308.08493.
  12. Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. Roscoe: A suite of metrics for scoring step-by-step reasoning. Preprint, arXiv:2212.07919.
  13. Joy He-Yueya, Gabriel Poesia, Rose E. Wang, and Noah D. Goodman. 2023. Solving math word problems by combining language models with symbolic solvers. Preprint, arXiv:2304.09102. find
  14. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. Preprint, arXiv:2103.03874.
  15. Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. Preprint, arXiv:2303.05398.
  16. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. find
  17. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1152–1157. find
  18. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  19. Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen. 2024a. Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark. Preprint, arXiv:2405.12209. find
  20. Wentao Liu, Hanglei Hu, Jie Zhou, Yuyang Ding, Junsong Li, Jiayi Zeng, Mengliang He, Qin Chen, Bo Jiang, Aimin Zhou, and Liang He. 2024b. Mathematical language models: A survey. Preprint, arXiv:2312.07622. — [arXiv:2312.07622](https://arxiv.org/abs/2312.07622) find
  21. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct. arXiv preprint. ArXiv:2308.09583 [cs].
  22. Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984. find
  23. Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, and Jenia Jitsev. 2024. Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models. Preprint, arXiv:2406.02061. find
  24. Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2024. Is Self-Repair a Silver Bullet for Code Generation? arXiv preprint. ArXiv:2306.09896 [cs]. find
  25. OpenAI. a. Hello GPT-4o. OpenAI. b. Introducing ChatGPT. find
  26. OpenAI. 2024. Openai o1 system card. Preprint, arXiv:2412.16720.
  27. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, and Sam Altman. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.
  28. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191.
  29. Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta, and Aran Komatsuzaki. 2023. ARB: Advanced Reasoning Benchmark for Large Language Models. arXiv preprint. ArXiv:2307.13692 [cs]. find
  30. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347. find
  31. KV Aditya Srivatsa and Ekaterina Kochmar. 2024. What makes math word problems challenging for llms? Preprint, arXiv:2403.11369. find
  32. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, and Jiahui Yu. 2024. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805.
  33. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  34. Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. 2024. Evaluating mathematical reasoning beyond accuracy. arXiv preprint arXiv:2404.05692.
  35. Xin Xu, Tong Xiao, Zitong Chao, Zhenya Huang, Can Yang, and Yang Wang. 2024. Can llms solve longer math word problems better? Preprint, arXiv:2405.14804. — [arXiv:2405.14804](https://arxiv.org/abs/2405.14804) find
  36. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  37. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. Meta-Math: Bootstrap Your Own Mathematical Questions for Large Language Models. arXiv preprint. ArXiv:2309.12284 [cs].
  38. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. Preprint, arXiv:2308.01825.
  39. Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, and Jiaya Jia. 2023. Mr-gsm8k: a metareasoning benchmark for large language model evaluation. arXiv preprint arXiv:2312.17080. find
  40. Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. 2024a. A careful examination of large language model performance on grade school arithmetic. Preprint, arXiv:2405.00… find
  41. Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. 2024b. Small language models need strong verifiers to self-correct reasoning. Preprint, arXiv:2404.17140. find
  42. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv preprint. ArXiv:2306.05685 [cs].

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2025-exposing-achilles-heel-evaluating-llms/