Elamur Research
☰
home / papers / 2024 / Cutting Through the Noise: Boosting LLM Performance on Math Word Problems

Cutting Through the Noise: Boosting LLM Performance on Math Word Problems

Summary

The paper studies how large language models solving math word problems are thrown off by irrelevant numbers in the problem text. The authors use a two-step prompting chain: a model first invents an unrelated physical quantity (such as volume, temperature or speed) with start and end values, then weaves it into the problem so that the solution and answer stay unchanged; constraints ensure the new variable has a different unit and adds no information about the original quantities. This yields PROBLEMATHIC, a dataset of paired clean and noisy problems at two difficulty levels (simple additive problems and complex ones with nested operations), with hand-written noisy test items, plus GSM-8K-Adv, a noisy variant of GSM8K. Distractors cut accuracy by about a quarter on average across models. Fine-tuning open models on noisy examples with explanations that point out the irrelevant variables improves accuracy on noisy problems by about 8%. The main finding is that models tend to use every number in the problem mechanically.

How Mova uses it

Effectiveness in Mova

Measured: SVAMP 57.3% → 52.7% (identical in two runs), i.e. a loss, so the idea was not adopted. Diagnosis: the artificial distractor always sits in the same position, so the model learns the position rather than the principle; in the paper, distractors are tailored to the problem type (similar object, same topic).

👨‍🔬💥

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2406.15444

Summary

Large language models (LLMs) are good at solving math word problems (MWP). In real-world settings, however, problem texts often contain irrelevant numbers, and these lead models away from the correct line of reasoning. The authors study this weakness systematically. To do so, they built a chain of two prompts. First, the LLM picks an unrelated physical quantity, such as volume, temperature or speed, and sets its initial and final values. Then the model embeds this quantity in the problem text so that the solution and the answer stay the same. Three rules ensure that the new variable is unrelated to the existing ones, has a different unit of measurement and adds no new information about the original quantities. This produced the PROBLEMATHIC dataset, which contains pairs of regular and noisy problems at two difficulty levels. The authors also built GSM-8K-Adv, a noisy version of GSM-8K. On average, the noise lowers model accuracy by about a quarter. Fine-tuning on noisy examples with explanations that point out the irrelevant variables improves results by about 8%. The work shows that models tend to mechanically use all the numbers in the problem statement. It also offers an inexpensive way to measure this flaw and reduce it.

Key points

  • PROBLEMATHIC has two subsets. Simple contains problems with up to three linear operations, addition and subtraction only (1313 problems for training and 60 for testing). Complex contains problems with nested operations, multiplication, division and proportions (613 for training and 73 for testing). The problems are drawn from several well-known MWP corpora. The authors wrote the noisy test problems by hand to avoid data leakage. The explanations for the training examples were generated by GPT-4.
  • On noisy problems, model accuracy drops: Gemini-1.5 Pro loses 16.07%, Claude-3.5 Sonnet loses 22.93% and Qwen-1.5 loses 41.53%. The average relative drop is about 26%.
  • Fine-tuning on noisy examples yields a gain of 8.19% on the Simple test problems and 7.07% on Complex. The main text names the fine-tuned models as Llama-2 (7B, 13B) and Mistral (7B), while the abstract names Qwen-2 and Mistral.
  • On GSM-8K-Adv, Mistral Large, Llama-2 and Llama-3 lose about 4% accuracy on average. According to the abstract, the drop reaches 24% in the worst case.
  • Quality was checked by humans: they rated 310 noisy problems (10% of the total) on a four-point scale. The manual test perturbations are divided into four types (A–D), depending on whether the answer concerns the added object and whether it concerns the added number. The dataset is published on Hugging Face, and the code on GitHub.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2406.15444

@misc{anantheswaran2024cutting,
  title = {{Cutting Through the Noise: Boosting LLM Performance on Math Word Problems}},
  author = {Anantheswaran, Ujjwala and Gupta, Himanshu and Scaria, Kevin and Verma, Shreyas and Baral, Chitta and Mishra, Swaroop},
  year = {2024},
  eprint = {2406.15444},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.15444}
}

References

43 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. find our card
  2. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges, 2024. 01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu,… find our card
  3. Mistral AI. Au large, Apr 2024. URL https://mistral.ai/news/mistral-large/. ↗
  4. AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/ blob/main/MODEL\_CARD.md. ↗
  5. Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024. URL https://api. semanticscholar.org/CorpusID:268232499. find
  6. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… find our card
  7. Yu-Chu Chang, Xu Wang, Jindong Wang, Yuanyi Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Weirong Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qian Yang, and Xingxu Xie. A survey on evaluation of large language models. ArXiv, abs/2307.03109, 2023. URL https://api.semantics… ↗ our card
  8. Chaitanya Dwivedi, Shima Nofallah, Maryam Pouryahya, Janani Iyer, Kenneth Leidal, Chuhan Chung, Timothy Watkins, Andrew Billin, Robert Myers, John Abel, and Ali Behrooz. Multi stain graph fusion for multimodal integration in pathology. In Proceedings of the IEEE/CVF Conference on Computer Vision and… find
  9. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, N… find our card
  10. Vedant Gaur and Nikunj Saunshi. Reasoning in large language models through symbolic math word problems. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 5889–5903, Toronto, Canada, July 2023. Association for Compu… find
  11. Rynaa Grover, Jayant Sravan Tamarapalli, Sahiti Yerramilli, and Nilay Pande. Huemanity: Probing fine-grained visual perception in mllms. arXiv preprint arXiv:2506.03194, 2025. URL https: //arxiv.org/abs/2506.03194. — [arXiv:2506.03194](https://arxiv.org/abs/2506.03194) find
  12. Himanshu Gupta, Neeraj Varshney, Swaroop Mishra, Kuntal Kumar Pal, Saurabh Arjun Sawant, Kevin Scaria, Siddharth Goyal, and Chitta Baral. “john is 50 years old, can his son be 65?” evaluating nlp models’ understanding of feasibility. In Proceedings of the 17th Conference of the European Chapter of t… find
  13. Shima Imani, Liang Du, and Harsh Shrivastava. MathPrompter: Mathematical reasoning using large language models. In Sunayana Sitaram, Beata Beigman Klebanov, and Jason D Williams (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)… find our card
  14. Aishwarya Jadhav, Jeffery Cao, Abhishree Shetty, Urvashi Kumar, Aditi Sharma, Ben Sukboontip, Jayant Tamarapalli, Jingyi Zhang, and Aniruddh Koul. Ai guide dog: Egocentric path prediction on smartphone. Proceedings of the AAAI Symposium Series, 5(1):220–227, May 2025. doi: 10.1609/ aaaiss.v5i1.35591… find
  15. Vidhi Jain, Jayant Sravan Tamarapalli, Sahiti Yerramilli, and Yonatan Bisk. Maea: Multimodal attribution for embodied ai. arXiv preprint arXiv:2307.13850, 2023. URL https://arxiv. org/abs/2307.13850. — [arXiv:2307.13850](https://arxiv.org/abs/2307.13850) find
  16. Pratik Jayarao and Arpit Sharma. Retraining distilbert for a voice shopping assistant by using universal dependencies, 2021. URL https://arxiv.org/abs/2103.15737. — [arXiv:2103.15737](https://arxiv.org/abs/2103.15737) ↗
  17. Pratik Jayarao and Aman Srivastava. Intent detection for code-mix utterances in task oriented dialogue systems. In 2018 International Conference on Electrical, Electronics, Communication, Computer, and Optimization Techniques (ICEECCOT), pp. 583–587, 2018. doi: 10.1109/ICEECCOT43722. 2018.9001577. — [doi:10.1109/iceeccot43722](https://doi.org/10.1109/iceeccot43722) find
  18. Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2021–2031, Copenhagen, Denmark, September 2017. Associ… find our card
  19. Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L’elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée La… find
  20. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  21. Vivek Kumar, Rishabh Maheshwary, and Vikram Pudi. Adversarial examples for evaluating math word problem solvers. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2705–2712, Punta Cana, Domi… find
  22. Vivek Kumar, Rishabh Maheshwary, and Vikram Pudi. Practice makes a solver perfect: Data augmentation for math word problem solvers. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association… find
  23. Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. Learning to automatically solve algebra word problems. In Kristina Toutanova and Hua Wu (eds.), Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 271–281, Baltimore, Ma… find
  24. Ankur Mallick, Chaitanya Dwivedi, Bhavya Kailkhura, Gauri Joshi, and T. Yong-Jin Han. Deep kernels with probabilistic embeddings for small-data learning. In Cassio de Campos and Marloes H. Maathuis (eds.), Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume… find
  25. Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. LILA: A unified benchmark for mathematical reasoning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of th… find our card
  26. Aitor Ormazabal, Che Zheng, Cyprien de Masson d’Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Y… find
  27. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 202… find our card
  28. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco J R Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, Alhussein Fawzi, Josh Grochow, Andrea Lodi, Jean-Baptiste Mouret, Talia Ringer, and Tao Yu. Mathemati… find our card
  29. Subhro Roy and Dan Roth. Solving general arithmetic word problems. ArXiv, abs/1608.01413, 2016. URL https://api.semanticscholar.org/CorpusID:560565. ↗ our card
  30. Subhro Roy and Dan Roth. Mapping to declarative knowledge for word problem solving. Transactions of the Association for Computational Linguistics, 6:159–172, 2018. doi: 10.1162/tacl\_a\_00012. URL https://aclanthology.org/Q18-1012. ↗ our card
  31. Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta, and Aran Komatsuzaki. Arb: Advanced reasoning benchmark for large language models. ArXiv, abs/2307.13692, 2023. URL https: //api.semanticscholar.org/CorpusID:260155126. find
  32. Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu, and Lakshmivihari Mareedu. An independent evaluation of chatgpt on mathematical word problems (mwp), 2023. find
  33. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Huai hsin Chi, Nathanael Scharli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, 2023. URL https: //api.semanticscholar.org/CorpusID:25645977… find our card
  34. Jayant Sravan Tamarapalli, Rynaa Grover, Nilay Pande, and Sahiti Yerramilli. Countqa: How well do mllms count in the wild? arXiv preprint arXiv:2508.06585, 2025. URL https: //arxiv.org/abs/2508.06585. find
  35. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. find our card
  36. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. find
  37. Roy Xie, Chengxuan Huang, Junlin Wang, and Bhuwan Dhingra. Adversarial math word problem generation, 2024. URL https://arxiv.org/abs/2402.17916. ↗
  38. Xilie Xu, Keyi Kong, Ninghao Liu, Li zhen Cui, Di Wang, Jingfeng Zhang, and Mohan S. Kankanhalli. An llm can fool itself: A prompt-based adversarial attack. ArXiv, abs/2310.13345, 2023. URL https://api.semanticscholar.org/CorpusID:264406064. — [arXiv:2310.13345](https://arxiv.org/abs/2310.13345) ↗
  39. Sahiti Yerramilli, Jayant Sravan Tamarapalli, Jonathan Francis, and Eric Nyberg. Attribution regularization for multimodal paradigms. arXiv preprint arXiv:2404.02359, 2024a. URL https://arxiv.org/abs/2404.02359. — [arXiv:2404.02359](https://arxiv.org/abs/2404.02359) ↗
  40. Sahiti Yerramilli, Jayant Sravan Tamarapalli, Tanmay Girish Kulkarni, Jonathan Francis, and Eric Nyberg. Semantic augmentation in images using language. arXiv preprint arXiv:2404.02353, 2024b. URL https://arxiv.org/abs/2404.02353. — [arXiv:2404.02353](https://arxiv.org/abs/2404.02353) ↗
  41. Sahiti Yerramilli, Nilay Pande, Rynaa Grover, and Jayant Sravan Tamarapalli. Geochain: Multimodal chain-of-thought for geographic reasoning. arXiv preprint arXiv:2506.00785, 2025. URL https: //arxiv.org/abs/2506.00785. find
  42. Zihao Zhou, Qiufeng Wang, Mingyu Jin, Jie Yao, Jianan Ye, Wei Liu, Wei Wang, Xiaowei Huang, and Kaizhu Huang. Mathattack: Attacking large language models towards math solving ability. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19750–19758, March 2024. ISSN 2159-5399. doi:… — [doi:10.1609/aaai.v38i17.29949](https://doi.org/10.1609/aaai.v38i17.29949) find
  43. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. ArXiv, abs/2307.15043, 2023. URL https://api. semanticscholar.org/CorpusID:260202961. find our card