Elamur Research⌕
☰
home / papers / 2024 / MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning

MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning

Paper · 2024 · Debrup Das, Debopriyo Banerjee, Somak Aditya et al.

Summary

This paper examines whether connecting a large language model to external tools helps with difficult mathematics, not just simple arithmetic. The authors present MathSensei, a modular framework based on the Chameleon architecture. It chains several modules in sequence. Four of them run on the LLM itself: a module that retrieves relevant knowledge (KR), one that writes Python programs (PG), one that refines that code (CR), and one that produces the final solution (SG). Others call external APIs: Bing Web Search (BS) and WolframAlpha (WA). Using gpt-3.5-turbo as the base model, they run extensive ablations on which modules to include and in what order. The main benchmark is MATH, with GSM-8K, AQuA-RAT and MMLU-Math as additional evaluations. The best pipeline, PG+WA+SG, outperforms chain-of-thought prompting by a wide margin on MATH. The gains shrink as problems get easier, and adding tools barely helps on grade-school word problems. Off-the-shelf tool planners (Plan-And-Solve, REACT) fail to beat the best fixed pipeline. This suggests that mathematical tool use needs its own planning methods, and that tools are most valuable for knowledge-heavy, computation-heavy problems.

Key points

  • Best configuration PG+WA+SG reaches 47.6% accuracy on MATH, which is 13.5 points above gpt-3.5-turbo with CoT.
  • On Intermediate Algebra this setup beats GPT-4 with CoT by 11.6 points. On Precalculus, WA+PG+SG raises accuracy from 26.7% (GPT-4 CoT) to 28.9%.
  • Gains are smaller on AQuA-RAT (+2.4) and MMLU-Math (+3.3). On GSM-8K, extra modules give little benefit, and adding BS even hurts performance.
  • Web search (BS) retrieves more useful knowledge than the LLM-based KR. WA+BS+SG outperforms PG+SG, so program-guided solving is not universally best.
  • Code and data are released at github.com/Debrup-61/MathSensei.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology KharagpurKharagpur · India26 works · 741 citations🏢RakutenTokyo · Japan2 works · 137 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputational Theory and Mathematics · 496 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Debrup Das, Debopriyo Banerjee, Somak Aditya and Ashish Kulkarni. 2024. MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.naacl-long.54 · arXiv 2402.17231

@inproceedings{das2024mathsensei,
  title = {{MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning}},
  author = {Debrup Das, Debopriyo Banerjee, Somak Aditya and Ashish Kulkarni},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.54},
  eprint = {2402.17231},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.54}
}

References

39 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and et al. 2020. Language Models are Few-Shot Learners. In Proceedings of the 34th Conference on Neural Information Processing Systems, volume 33…
  3. Jiaao Chen, Xiaoman Pan, Dian Yu, Kaiqiang Song, Xiaoyang Wang, Dong Yu, and Jianshu Chen. 2023a. Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models.
  4. Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. 2023b. ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs.
  5. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023c. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. find
  6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, and et al. Gehrmann. 2024. PaLM: scaling language modeling with pathways. J. Mach. Learn. Res., 24(1):1– 113.
  7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems.
  8. Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, and et al. 2022. A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level. Proceedings of the National Academy of Sciences, 119(32):1–10. find
  9. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate.
  10. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-Based Prompting for Multi-step Reasoning. In Proceedings of the 11th International Conference on Learning Representations, ICLR ’23, pages 1–15.
  11. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. PAL: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23, pages 10764–10799. find
  12. Yiduo Guo, Yaobo Liang, Chenfei Wu, Wenshan Wu, Dongyan Zhao, and Nan Duan. 2023. Learning to Program with Natural Language. find
  13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring Massive Multitask Language Understanding. In Proceedings of the 9th International Conference on Learning Representations, ICLR ’21, pages 1–27.
  14. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring Mathematical Problem Solving With the MATH Dataset. In Proceedings of the 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Roun…
  15. Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards Reasoning in Large Language Models: A Survey.
  16. Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing Algebraic Word Problems into Equations. Transactions of the Association for Computational Linguistics, 3:585–597. find
  17. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
  18. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate.
  19. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL ’17’, pages 158–167.… find
  20. OpenAI. 2023. GPT-4 Technical Report.
  21. Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. ART: Automatic multi-step reasoning and tool-use for large language models.
  22. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. find
  23. Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Proceedings of the 37th Conference on Neural Information Processing Systems, NIPS ’…
  24. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, and et al. 2023. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, pages 1–95.
  25. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. In Findings of the Association for Computational Linguistics, A…
  26. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023a. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. In Proceedings of the 37th Conference on Neural Information Processing Systems, NIPS ’23, pages 1–32.
  27. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo…
  28. Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. 2023b. A Survey of Deep Learning for Mathematical Reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL ’23, pages 14605–14631. find
  29. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In Proceedings of the 36th Conference on Neural Information Processing Systems, NIPS ’22, pages 1–14.
  30. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with… find
  31. Yuanzhen Xie, Tao Xie, Mingxiong Lin, WenTao Wei, Chenglin Li, Beibei Kong, Lei Chen, Chengxiang Zhuo, Bo Hu, and Zang Li. 2023. OlaGPT: Empowering LLMs With Human-like Problem-Solving Abilities. find
  32. Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers. In Proceedings of the 58th Annual Meeting of the Association Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christoph… find
  33. Runzhe Yang and Karthik Narasimhan. 2023. The Socratic Method for Self-Discovery in Large Language Models. find
  34. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP ’18, pages 2369–2380. find
  35. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Proceedings of the 37th Conference on Neural Information Processing Systems, NIPS ’23, pages 1–14. find
  36. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. ReAct: Synergizing Reasoning and Acting in Language Models.
  37. Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models.
  38. Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023. Progressive-Hint Prompting Improves Reasoning in Large Language Models.
  39. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2024. Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification. In Proceedings of the 12th International Conference on Le…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/das-2024-mathsensei-tool-augmented-large-language/