Elamur Research⌕
☰
home / papers / 2025 / Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passes

Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passes

Paper · 2025 · Bryan R Christ, Zachary Gottesman, Jonathan Kropko et al.

Summary

The paper asks whether mathematical reasoning in large language models is tied to a separable set of parameters, and whether that set can be located cheaply. Existing forward-pass localization methods fall short. LAPE does not find math-related neurons. Wanda finds important weights, but they overlap heavily with weights that other tasks rely on. The authors propose MathNeuro, which scores each weight in attention and MLP layers by its absolute value multiplied by the norm of its input activations, summed over samples. It then takes the top-K% of weights for math inputs and removes any that also rank in the top-K% for a non-math task such as RACE or MMLU. No gradients are needed.

Tested on five LLMs between 1B and 8B parameters, pruning the selected weights eliminates math ability. The effect on non-math tasks is about the same as pruning random weights. Multiplying the selected weights by a small constant improves math scores without changing other behavior. These results suggest math skill can be targeted for improvement without side effects.

Key points

  • Scaling MathNeuro-selected weights raises accuracy by 4–17% on GSM8K and by 5–35% on MATH, for both pretrained and instruction-tuned models.
  • The method is data-efficient: most of its effect is preserved when importance is computed from a single sample.
  • For Llama 3.2 1B IT, the math-specific weights make up only 1.51% (MMLU filter) or 1.84% (RACE filter) of all parameters. They are spread fairly evenly across decoder blocks.
  • Weights identified from GSM8K generalize to unseen tasks:
    • Pruning them destroys performance on the math benchmarks EGSM, MATH and MathQA.
    • Performance stays largely intact on HellaSwag, MuTual, PIQA, WikiText and WinoGrande.
  • Code is available at github.com/bryanchrist/MathNeuro. The authors did not run a full sweep of the scaling factor; smaller models appeared to benefit from larger factors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Bryan R Christ, Zachary Gottesman, Jonathan Kropko and Thomas Hartvigsen. 2025. Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passes. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1209

@inproceedings{christ2025math,
  title = {{Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passes}},
  author = {Bryan R Christ, Zachary Gottesman, Jonathan Kropko and Thomas Hartvigsen},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1209},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1209}
}

References

52 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. 2024. find
  2. Many-Shot In-Context Learning. arXiv preprint. ArXiv:2404.11018 [cs].
  3. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large Language Models for Mathematical Reasoning: Progresses and Challenges. arXiv preprint. ArXiv:2402.00157 [cs].
  4. Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. arXiv preprint. ArXiv:1905.13319 [cs].
  5. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2018. Identifying and Controlling Important Neurons in Neural Machine Translation. arXiv preprint. ArXiv:1811.01157 [cs]. find
  6. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about Physical Commonsense in Natural Language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439. Number: 05.
  7. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. 2024. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv preprint. ArXiv:2407.21787 [cs].
  8. Ting-Yun Chang, Jesse Thomason, and Robin Jia. 2024. Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1… find
  9. Bryan R Christ, Jonathan Kropko, and Thomas Hartvigsen. 2024. MATHWELL: Generating Educational Math Word Problems Using Teacher Annotations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11914– 11938, Miami, Florida, USA. Association for Computational Linguistics. find
  10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint. ArXiv:2110.14168 [cs].
  11. Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. MuTual: A Dataset for Multi-Turn Dialogue Reasoning. arXiv preprint. ArXiv:2004.04494 [cs].
  12. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge Neurons in Pretrained Transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493– 8502, Dublin, Ireland. Association for Computatio… find
  13. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. 2018. What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models. find
  14. Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. 2020. Analyzing Redundancy in Pretrained Transformer Models. find
  15. Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, Peng Shi, Wenpeng Yin, and Rui Zhang. 2023. Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning. arXiv preprint. ArXiv:2311.03748 [cs]. find
  16. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… find
  17. Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipalli, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. 2024. LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement. find
  18. Yongqi Leng and Deyi Xiong. 2024. Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neurons. arXiv preprint. ArXiv:2407.06488 [cs].
  19. Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. 2024a. Common 7B Language Models Already Possess Strong Math Capabilities. arXiv preprint. ArXiv:2403.04706 [cs]. find
  20. Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024b. GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers. arXiv preprint. ArXiv:2402.19255 [cs].
  21. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks Are All You Need II: phi-1.5 technical report. arXiv preprint. ArXiv:2309.05463 [cs].
  22. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. Finding Neurons in a Haystack: Case Studies with Sparse Probing. find
  23. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2024. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning. arXiv preprint. ArXiv:2308.08747 [cs]. find
  24. Michael Hanna, Ollie Liu, and Alexandre Variengien. 2023. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36:76033–76060.
  25. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer Sentinel Mixture Models. _eprint: 1609.07843.
  26. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring Massive Multitask Language Understanding. arXiv preprint. ArXiv:2009.03300 [cs].
  27. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring Mathematical Problem Solving With the MATH Dataset. arXiv preprint. ArXiv:2103.03874 [cs].
  28. Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. 2024. On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons. arXiv preprint. ArXiv:2404.02431 [cs].
  29. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding Comprehension Dataset From Examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785– 794, Copenhagen, Denmark. Association for Computational Lin…
  30. MetaAI. 2024a. Introducing Llama 3.1: Our most capable models to date. find
  31. MetaAI. 2024b. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. find
  32. Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. 2024. Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics. arXiv preprint. ArXiv:2410.21272. find
  33. Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. 2023. Task-Specific Skill Localization in Fine-tuned Language Models. In Proceedings of the 40th International Conference on Machine Learning, pages 27011–27033. PMLR. ISSN: 2640-3498. find
  34. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. Learning to Generate Reviews and Discovering Sentiment.
  35. Daking Rai and Ziyu Yao. 2024. An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs. arXiv preprint. ArXiv:2406.12288 [cs]. find
  36. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. WinoGrande: an adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. find
  37. Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Mat… find
  38. Peiyang Song, Kaiyu Yang, and Anima Anandkumar. 2024. Towards Large Language Models as Copilots for Theorem Proving in Lean. arXiv preprint. ArXiv:2404.12534 [cs]. find
  39. Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023. A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7035–7052, Singapore. Association… find
  40. Xavier Suau, Pieter Delobelle, Katherine Metcalf, Armand Joulin, Nicholas Apostoloff, Luca Zappella, and Pau Rodríguez. 2024. Whispering Experts: Neural Interventions for Toxicity Mitigation in Language Models. arXiv preprint. ArXiv:2407.12824 [cs]. — [arXiv:2407.12824](https://arxiv.org/abs/2407.12824) find
  41. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2023. A Simple and Effective Pruning Approach for Large Language Models. arXiv preprint. ArXiv:2306.11695 [cs]. find
  42. Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models. arXiv preprint. ArXiv:2402.16438 [cs].
  43. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton T… find
  44. Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. 2022. Finding Skill Neurons in Pre-trained Transformer-based Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11132–11152, Abu Dhabi, United Arab Emirates.…
  45. Yifan Wei, Xiaoyan Yu, Yixuan Weng, Huanhuan Ma, Yuanzhe Zhang, Jun Zhao, and Kang Liu. 2024. Does Knowledge Localization Hold True? Surprising Differences Between Entity and Relation Perspectives in Language Models. arXiv preprint. ArXiv:2409.00617 [cs]. — [arXiv:2409.00617](https://arxiv.org/abs/2409.00617) find
  46. Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. 2024. DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data. arXiv preprint. ArXiv:2405.14333 [cs].
  47. Ji Xin, Jimmy Lin, and Yaoliang Yu. 2019. What Part of the Neural Network Does This? Understanding LSTMs by Measuring and Dissecting Neurons. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce… find
  48. Fangcong Yin, Xi Ye, and Greg Durrett. 2024. LoFiT: Localized Fine-tuning on LLM Representations. arXiv preprint. ArXiv:2406.01563 [cs]. — [arXiv:2406.01563](https://arxiv.org/abs/2406.01563) find
  49. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? arXiv preprint. ArXiv:1905.07830 [cs]. find
  50. Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu-ming Cheung, Xinmei Tian, Xu Shen, and Jieping Ye. 2024. Interpreting and Improving Large Language Models in Arithmetic Calculation. arXiv preprint. ArXiv:2409.01659. find
  51. Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. How do Large Language Models Handle Multilingualism? arXiv preprint. ArXiv:2402.18815 [cs].
  52. Fangwei Zhu, Damai Dai, and Zhifang Sui. 2025. Language Models Encode the Value of Numbers Linearly. In Proceedings of the 31st International Conference on Computational Linguistics, pages 693–709, Abu Dhabi, UAE. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/christ-2025-math-neurosurgery-isolating-language-models/