Elamur Research
☰
home / papers / 2024 / GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2410.05229v2

Summary

Scores on GSM8K, the standard grade-school math benchmark, keep rising. The authors ask whether that reflects real reasoning or just familiarity with a fixed and possibly leaked test set. They build GSM-Symbolic by turning GSM8K problems into templates, where names, numbers and family terms become variables with constraints, so many fresh versions of each question can be generated. They test 25 open and closed LLMs on these versions. Results swing noticeably between versions of the same problem. Accuracy usually drops compared with the original GSM8K, which points to possible data contamination. Changing only the numbers hurts more than changing only the names. Adding more clauses makes problems harder, and both the error rate and the spread of results grow. A companion set, GSM-NoOp, adds a statement that seems relevant but plays no part in the solution, and this sharply lowers accuracy for every model. The authors conclude that current LLMs mostly match patterns from their training data rather than reason formally. This matters for anyone who relies on single-number benchmark scores.

Key points

  • The benchmark has 100 GSM8K-derived templates with 50 generated instances each. It treats performance as a distribution across instances instead of one accuracy number. Templates and data are public at github.com/apple/ml-gsm-symbolic.
  • Harder variants (GSM-M1, GSM-P1, GSM-P2) remove or add clauses. Accuracy falls and variance rises as problems get longer.
  • On GSM-NoOp, accuracy drops by up to 65% for strong models. Few-shot examples, even ones with similar distractors, do not fix the problem.
  • An ablation with Gemma2-9b-it finds arithmetic accuracy near 97–99% on numbers up to 5 digits. Calculation mistakes therefore do not explain the drops.
  • An appendix responds to a statistical critique by Ivanova et al. (2025). The authors argue that an ideal reasoner should get a reworded question right whenever it got the original right, and wrong whenever it got the original wrong, rather than behave like a random coin-flip trial.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

209 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 207 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.05229 · arXiv 2410.05229

@misc{mirzadeh2024gsm,
  title = {{GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models}},
  author = {Mirzadeh, Iman and Alizadeh, Keivan and Shahrokhi, Hooman and Tuzel, Oncel and Bengio, Samy and Farajtabar, Mehrdad},
  year = {2024},
  doi = {10.48550/arxiv.2410.05229},
  eprint = {2410.05229},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.05229}
}

References

51 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra,… find our card
  2. Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap,… find our card
  3. Enric Boix-Adserà, Omid Saremi, Emmanuel Abbe, Samy Bengio, Etai Littwin, and Joshua M. Susskind. When can transformers reason with abstract symbols? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. find
  4. OpenReview.net, 2024. URL https://openreview.net/forum?id=STUGfUz8ob. ↗
  5. Ronald Brachman and Hector Levesque. Knowledge representation and reasoning. Elsevier, 2004. find
  6. François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. find
  7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. find our card
  8. David Cole. The Chinese Room Argument. In Edward N. Zalta and Uri Nodelman (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2024 edition, 2024. find
  9. Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A. Ortega. Neural networks and the chomsky hierarchy. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali,… find
  10. OpenReview.net, 2023. URL https://openreview.net/forum?id=WbxHAzkeQcn. ↗
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez,… find
  12. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality,… find our card
  13. Gilles E. Gignac and Eva T. Szodorai. Defining intelligence: Bridging the gap between human and artificial perspectives. Intelligence, 2024. URL https://api.semanticscholar.org/CorpusID:269015060. ↗
  14. Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, Deepak Gopinath, Dian Ang Yap, Dong Yin, Feng Nan, Floris Weers, Guoli Yin, Haoshuo Huang, Jianyu Wang, Jiarui Lu, John Peebles, Ke Ye, Mark Lee, Nan Du, Qibin Chen, Qu… — [doi:10.48550/arxiv.2407.21075](https://doi.org/10.48550/arxiv.2407.21075) find
  15. Pengfei Hong, Navonil Majumder, Deepanway Ghosal, Somak Aditya, Rada Mihalcea, and Soujanya Poria. Evaluating llms’ mathematical and coding competency through ontology-guided interventions. arXiv preprint arXiv:2401.09395, 2024. find
  16. Desi R Ivanova, Ilija Ilievski, and Momchil Konstantinov. Towards more rigorous evaluations of language models. In ICLR Blogposts 2025, 2025. URL https://iclr-blogposts. github.io/2025/blog/towards-more-rigorous-llm-evals/. https://iclr-blogposts.github.io/2025/blog/towards-more-rigorous-llm-evals/. ↗
  17. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  18. Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J. Su, Camillo J. Taylor, and Dan Roth. A peek into token bias: Large language models are not yet genuine reasoners. CoRR, abs/2406.11050, 2024. doi: 10.48550/ARXIV.2406.11050. URL https://doi.org/10.48550/arXiv.2406.11050… ↗
  19. Subbarao Kambhampati. Can large language models reason and plan? Annals of the New York Academy of Sciences, 1534:15 – 18, 2024. URL https://api.semanticscholar.org/CorpusID:268249961. ↗
  20. Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and machines, 17:391–444, 2007. find
  21. Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computat… find our card
  22. Zihao Li, Yuan Cao, Cheng Gao, Yihan He, Han Liu, Jason M. Klusowski, Jianqing Fan, and Mengdi Wang. One-layer transformer provably learns one-nearest neighbor in context. 2024b. URL https://api.semanticscholar.org/CorpusID:272307690. ↗
  23. Zhiyuan Liu, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. find
  24. OpenReview.net, 2024. URL https://openreview.net/forum?id=3EWTEy9MTM. ↗
  25. Gary F Marcus. The algebraic mind: Integrating connectionism and cognitive science. MIT press, 2003. find
  26. R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve, 2023. URL https://arxiv.org/abs/2309.13638. ↗
  27. Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacche… find
  28. Allen Newell and Herbert Simon. The logic theory machine–a complex information processing system. IRE Transactions on information theory, 2(3):61–79, 1956. find
  29. Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, and Jenia Jitsev. Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models. arXiv preprint arXiv:2406.02061, 2024. find
  30. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774. ↗ our card
  31. OpenAI. Learning to reason with large language models. https://openai.com/index/learning-to-reason-with-llms/, 2024. Accessed: 2024-09-29. ↗
  32. Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Elsevier, 2014. find
  33. Binghui Peng, Srini Narayanan, and Christos H. Papadimitriou. On limitations of the transformer architecture. CoRR, abs/2402.08164, 2024. doi: 10.48550/ARXIV.2402.08164. URL https: //doi.org/10.48550/arXiv.2402.08164. — [doi:10.48550/arxiv.2402.08164](https://doi.org/10.48550/arxiv.2402.08164) find
  34. Yasaman Razeghi, Adam Roberts, Colin Raffel, and Ariel Herbert-Voss. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.08904, 2022. find our card
  35. Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Ni… find
  36. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage?, 2023. find our card
  37. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), In… find our card
  38. PMLR, 2023. URL https://proceedings.mlr.press/v202/shi23a.html. ↗
  39. Saurabh Srivastava, Annarose M B, Anto P V, Shashank Menon, Ajay Sukumar, Adwaith Samod T, Alan Philipose, Stevin Prince, and Sooraj Thomas. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap, 2024. URL https://arxiv.org/abs/2402.19450. ↗
  40. Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schoelkopf, and Mrinmaya Sachan. A causal framework to quantify the robustness of mathematical reasoning with language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Associa… find
  41. Karthik Valmeekam, Alberto Olmo Hernandez, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can’t plan (A benchmark for llms on planning and reasoning about change). CoRR, abs/2206.10498, 2022. doi: 10.48550/ARXIV.2206.10498. URL https: //doi.org/10.48550/arXiv.2206.10498. find our card
  42. Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. Llms still can’t plan; can lrms? a preliminary evaluation of openai’s o1 on planbench. 2024. URL https://api. semanticscholar.org/CorpusID:272770270. find
  43. Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. find our card
  44. Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 11080–11090. find
  45. PMLR, 2021. URL http://proceedings.mlr.press/v139/weiss21a.html. ↗
  46. Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv preprint arXiv:2407.20311, 2024. find
  47. Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. In Edith Elkind (ed.), Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pp. 3365–3373. International Joint Conferenc… find
  48. Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. A careful examination of large language model performance on grade school arithmetic. CoRR, abs/2405.00332, 2024. do… find
  49. Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Joshua M. Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? A study in length generalization. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-1… find
  50. OpenReview.net, 2024. URL https://openreview.net/forum?id=AssIuHnmHX. ↗
  51. Nikola Zubic, Federico Soldà, Aurelio Sulser, and Davide Scaramuzza. Limits of deep learning: Sequence modeling through the lens of complexity theory. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=DhdqML3FdM. ↗