Elamur Research⌕
☰
home / papers / 2025 / Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges

Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges

Paper · 2025 · Safal Shrestha, Minwu Kim and Keith Ross

Summary

Most math benchmarks for LLMs use small numbers and grade only the final answer. As a result, they cannot show whether a model failed because its reasoning was flawed or because it made a calculation slip. The authors address both problems. They release GSM-Ranges, a generator that converts 100 integer-only GSM8K test problems into Python templates. It refills each template with random values at six scales, from same-digit swaps up to the millions, while keeping intermediate values non-negative and limiting answer growth. They also propose a grader in which GPT-4o rewrites a model's solution as Python code. Running that code removes arithmetic slips, so a solution that is still wrong is counted as a logical error. Across several open and proprietary models, logical errors rose with number size by up to 14 percentage points, even though the required reasoning never changes. Arithmetic embedded in word problems was also much less reliable than the same operations asked on their own. The work matters because it reveals weak numerical generalization that small-number benchmarks hide.

Key points

  • Values below 1,000 make up 94.9% of numbers in GSM8K, 97.8% in SVAMP and 98.0% in MATH.
  • GSM-Ranges draws its base problems from the 1,319-question GSM8K test set. It excludes problems with division or non-integer answers and keeps 100 problems containing only one- and two-digit numbers. The six perturbation levels are same-digit, 100–1,000, 1,000–10,000, 10,000–100,000, 100,000–1,000,000 and 1,000,000–10,000,000.
  • With repeated sampling at temperature 0.8 and up to 48 passes, the recall of correct logic rose. The gap between level 1 and level 6 shrank to 2 problems or fewer per model, which suggests the correct reasoning is still within reach.
  • The reasoning model o3-mini kept low error rates at every level, but it produced more tokens as the numbers grew.
  • Arithmetic operations that models got wrong inside solutions were re-asked in isolation, and most were then solved correctly. For example, Llama 3.2 3B still failed only 4.2% of its level-1 cases and 8.6% of its level-2 cases. Code and data are available at github.com/minwukim/GSM-Ranges.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates125 works · 4,714 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputational Theory and Mathematics · 496 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Safal Shrestha, Minwu Kim and Keith Ross. 2025. Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges. arXiv:2502.08680.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.08680 · arXiv 2502.08680

@misc{shrestha2025mathematical,
  title = {{Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges}},
  author = {Safal Shrestha, Minwu Kim and Keith Ross},
  year = {2025},
  doi = {10.48550/arxiv.2502.08680},
  eprint = {2502.08680},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.08680}
}

References

37 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zain Ul Abedin, Shahzeb Qamar, Lucie Flek, and Akbar Karimi. 2025. Arithmattack: Evaluating robustness of llms to noisy context in math problem solving. arXiv preprint arXiv:2501.08203.
  2. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages…
  3. Avinash Anand, Mohit Gupta, Kritarth Prasad, Navya Singla, Sanjana Sanjeev, Jatin Kumar, Adarsh Raj Shivam, and Rajiv Ratn Shah. 2024. Mathify: Evaluating large language models on mathematical problem solving tasks. arXiv preprint arXiv:2404.13099. find
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  5. Guhao Feng, Kai Yang, Yuntian Gu, Xinyue Ai, Shengjie Luo, Jiacheng Sun, Di He, Zhenguo Li, and Liwei Wang. 2024. How numerical precision affects mathematical reasoning capabilities of llms. arXiv preprint arXiv:2410.13857. — [arXiv:2410.13857](https://arxiv.org/abs/2410.13857) find
  6. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner. 2024. Mathematical capabilities of chatgpt. Advances in neural information processing systems, 36.
  7. Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. 2023. Chain-of-thought hub: A continuous effort to measure large language models’ reasoning performance. arXiv preprint arXiv:2305.17306.
  8. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  9. Siyuan Guo, Aniket Didolkar, Nan Rosemary Ke, Anirudh Goyal, Ferenc Huszár, and Bernhard Schölkopf. 2024. Learning beyond pattern matching? assaying mathematical understanding in llms. arXiv preprint arXiv:2405.15485. — [arXiv:2405.15485](https://arxiv.org/abs/2405.15485) find
  10. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. In Advances in Neural Information Processing Systems, volume 34, pages 24241–24253.
  11. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. 2020. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701.
  12. Pengfei Hong, Deepanway Ghosal, Navonil Majumder, Somak Aditya, Rada Mihalcea, and Soujanya Poria. 2024. Caught in the quicksand of reasoning, far from agi summit: Evaluating llms’ mathematical and coding competency through ontology-guided interventions. arXiv preprint arXiv:2401.09395. find
  13. Ashish Hooda, Mihai Christodorescu, Miltiadis Allamanis, Aaron Wilson, Kassem Fawaz, and Somesh Jha. 2024. Do large code models understand programming concepts? a black-box approach. arXiv preprint arXiv:2402.05980. — [arXiv:2402.05980](https://arxiv.org/abs/2402.05980) find
  14. Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J. Su, Camillo J. Taylor, and Dan Roth. 2024. A peek into token bias: Large language models are not yet genuine reasoners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP),…
  15. Kuei-Chun Kao, Ruochen Wang, and Cho-Jui Hsieh. 2024. Solving for x and beyond: Can large language models solve complex math problems with more-than-two unknowns? arXiv preprint arXiv:2407.05134. find
  16. Ananya Kumar and Siddharth Jain. 2024. Investigating implicit bias in large language models: A large-scale study of over 50 llms. arXiv preprint arXiv:2410.12864. — [arXiv:2410.12864](https://arxiv.org/abs/2410.12864) find
  17. Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024a. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page…
  18. Xiaoyuan Li, Wenjie Wang, Moxin Li, Junrong Guo, Yang Zhang, and Fuli Feng. 2024b. Evaluating mathematical reasoning of large language models: A focus on error identification and correction. arXiv preprint arXiv:2406.00755. find
  19. Aman Madaan and Amir Yazdanbakhsh. 2022. Text and patterns: For effective chain of thought, it takes two to tango. Preprint, arXiv:2209.07686.
  20. Davide Maltoni and Matteo Ferrara. 2024. Arithmetic with language models: From memorization to computation. Neural Networks, 179:106550.
  21. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. Preprint, arXiv:2410.05229.
  22. OpenAI. 2024. Gpt-4 technical report. arXiv:2303.08774.
  23. Preprint, OpenAI. 2024. Gpt-4o system card. Published: August 8, 2024. Accessed: 2024-12-20.
  24. OpenAI. 2025. Introducing OpenAI o1. Accessed February 7, 2025. find
  25. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Associat…
  26. Jing Qian, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. 2023. Limitations of language models in arithmetic and symbolic induction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9285–9298, Toronto, Canada. Association for… find
  27. Qwen. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  28. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539–68551.
  29. Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu, and Lakshmivihari Mareedu. 2023. An independent evaluation of chatgpt on mathematical word problems (mwp). arXiv preprint arXiv:2302.13814.
  30. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 31210–31227.
  31. Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, and Mrinmaya Sachan. 2022. A causal framework to quantify the robustness of mathematical reasoning with language models. arXiv preprint arXiv:2210.12023.
  32. Lindia Tjuatja, Valerie Chen, Sherry Tongshuang Wu, Ameet Talwalkar, and Graham Neubig. 2023. Do llms exhibit human-like response biases? a case study in survey design. arXiv preprint arXiv:2311.04076.
  33. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24827–24837.
  34. Roy Xie, Chengxuan Huang, Junlin Wang, and Bhuwan Dhingra. 2024. Llm-resistant math word problem generation via adversarial attacks. arXiv preprint arXiv:2402.17916. — [arXiv:2402.17916](https://arxiv.org/abs/2402.17916) find
  35. Zhen Yang, Ming Ding, Qingsong Lv, Zhihuan Jiang, Zehai He, Yuyi Guo, Jinfeng Bai, and Jie Tang. 2023. Gpt can solve mathematical problems without a calculator. arXiv preprint arXiv:2309.03241.
  36. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do large language models perform in arithmetic tasks? arXiv preprint arXiv:2304.02015.
  37. Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, and Jiaya Jia. 2024. Mr-gsm8k: A meta-reasoning benchmark for large language model evaluation. Preprint, arXiv:2312.17080.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shrestha-2025-mathematical-reasoning-large-language-models/