Elamur Research⌕
☰
home / papers / 2025 / ArithmAttack: Evaluating Robustness of LLMs to Noisy Context in Math Problem Solving

ArithmAttack: Evaluating Robustness of LLMs to Noisy Context in Math Problem Solving

Paper · 2025 · Zain Ul Abedin, Shahzeb Qamar, Lucie Flek et al.

Summary

The paper asks whether LLMs that solve arithmetic word problems still work when the input is cluttered with stray punctuation. The authors introduce ArithmAttack, a perturbation method adapted from the AEDA augmentation technique. It inserts random punctuation marks (. , ! ? ; :) at random positions in a math problem. The number of inserted marks is fixed at 10%, 30% or 50% of the problem's word count. No words are added, removed or altered, so the meaning is preserved; Universal Sentence Encoder rates the perturbed texts as fully similar to the originals.

Eight open models with 1.5B–8B parameters were tested on the GSM8K and MultiArith test sets using Zero-Shot CoT prompting. The evaluation reports accuracy and Attack Success Rate (ASR), the share of initially correct answers that become wrong after perturbation. Every model lost accuracy, and the losses grew with the amount of noise. Llama models were the most accurate and the most robust, while Zephyr was the weakest. The work shows that even trivial, meaning-neutral formatting noise can derail LLM reasoning. This matters for real-world inputs that are messy or poorly formatted.

Key points

  • Data and models: Experiments use the GSM8K test set (~1.32k problems) and the MultiArith test set (180 problems). The models are Mistral-7B-Instruct-v0.2, Mathstral-7b-v0.1, Llama-3-8B-Instruct, Llama-3.1-8B-Instruct, Gemma-2-2b-it, Zephyr-7b-beta, Qwen2.5-1.5B-Instruct and DeepSeek-R1-Distill-Llama-8B.
  • Best on GSM8K: Llama3.1 had the highest accuracy, falling from 82.25% clean to 77.02% at 50% noise. It also had the lowest ASR there (12.53%).
  • Best on MultiArith: Llama3 had the lowest ASR (7.79%), and DeepSeek was close behind (8.28%).
  • Weakest model: Zephyr was the least robust, with an ASR of about 75–77%. Its GSM8K accuracy dropped from 23.27% to 10.08% at 50% noise.
  • Other patterns: Mathstral was more robust than its general-purpose sibling Mistral (GSM8K ASR 19.81% vs. 39.69%). For all models except Zephyr, noise hurt more on the harder GSM8K dataset than on MultiArith. A manual check of answer extraction found low miss rates for most models, with notable exceptions for Mistral on GSM8K (9%) and Zephyr on MultiArith (12.8%).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of BonnBonn · Germany10 works · 5,751 citations🔬Lamarr Institute for Machine Learning and Artificial IntelligenceSankt Augustin · Germany6 works · 121 citations🔬Bonn Aachen International Center for Information TechnologyBonn · Germany2 works · 13 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Zain Ul Abedin, Shahzeb Qamar, Lucie Flek and Akbar Karimi. 2025. ArithmAttack: Evaluating Robustness of LLMs to Noisy Context in Math Problem Solving. arXiv:2501.08203.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.08203 · arXiv 2501.08203

@misc{abedin2025arithmattack,
  title = {{ArithmAttack: Evaluating Robustness of LLMs to Noisy Context in Math Problem Solving}},
  author = {Zain Ul Abedin, Shahzeb Qamar, Lucie Flek and Akbar Karimi},
  year = {2025},
  doi = {10.48550/arxiv.2501.08203},
  eprint = {2501.08203},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.08203}
}

References

26 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vansh Agrawal, Pratham Singla, Amitoj Singh Miglani, Shivank Garg, and Ayush Mangal. 2024. Give me a hint: Can llms take a hint to solve math problems? In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24.
  2. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder for english. In Proceedings of the 2018 conference on empirical methods in natural language processing: system demo…
  3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  4. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  5. Steffen Eger and Yannik Benz. 2020. From hero to zéroe: A benchmark of low-level adversarial attacks. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 78… find
  6. Brian Formento, Chuan-Sheng Foo, Luu Anh Tuan, and See Kiong Ng. 2023. Using punctuation as an adversarial attack on deep learning-based nlp systems: An empirical study. In Findings of the association for computational linguistics: EACL 2023, pages 1–34. find
  7. Brian Formento, See-Kiong Ng, and Chuan-Sheng Foo. 2021. Special symbol attacks on nlp systems. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE. find
  8. Esther Gan, Yiran Zhao, Liying Cheng, Yancan Mao, Anirudh Goyal, Kenji Kawaguchi, Min-Yen Kan, and Michael Shieh. 2024. Reasoning robustness of llms to adversarial typographical errors. arXiv preprint arXiv:2411.05345. find
  9. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  10. Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving google’s perspective api built for detecting toxic comments. arXiv preprint arXiv:1702.08138. — [arXiv:1702.08138](https://arxiv.org/abs/1702.08138) find
  11. Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 37–42.
  12. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  13. Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021. AEDA: An easier data augmentation technique for text classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2748–2754, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  14. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  15. Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages…
  16. Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743–1752.
  17. Amrutesh Saraf, Pooja Kamat, Shilpa Gite, Satish Kumar, and Ketan Kotecha. 2024. Towards robust automated math problem solving: a survey of statistical and deep learning approaches. Evolutionary Intelligence, pages 1–38. find
  18. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR.
  19. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  20. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023. Zephyr: Direct distillation of lm alignment. Preprint, arXiv… find
  21. Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. 2021. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (R… find
  22. Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, and Chi Wang. 2024. Mathchat: Converse to tackle challenging math problems with llm agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.
  23. Roy Xie, Chengxuan Huang, Junlin Wang, and Bhuwan Dhingra. 2024. Adversarial math word problem generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5075–5093. find
  24. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  25. Zihao Zhou, Qiufeng Wang, Mingyu Jin, Jie Yao, Jianan Ye, Wei Liu, Wei Wang, Xiaowei Huang, and Kaizhu Huang. 2024. Mathattack: Attacking large language models towards math solving ability. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19750–19758. find
  26. Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, et al. 2023. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv eprints, pages arXiv–2306. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abedin-2025-arithmattack-evaluating-robustness-llms-noisy/