Elamur Research⌕
☰
home / papers / 2023 / Design of Chain-of-Thought in Math Problem Solving

Design of Chain-of-Thought in Math Problem Solving

Paper · 2023 · Zhanming Jie, Trung Quoc Luong, Xinbo Zhang et al.

Summary

The paper asks which chain-of-thought (CoT) format best helps language models solve math word problems. The authors, from ByteDance Research, compare natural language CoT with three program-based styles. The self-describing program (SDP) uses variable names drawn from the question. The non-describing program (NDP) uses only abstract names such as v1 and v2. The comment-describing program (CDP), a new design, pairs abstract names with short natural-language comments. Each program style is written in both Python and Wolfram Language. Training CoTs for GSM8K, MathQA and SVAMP were produced semi-automatically: retrieval-based few-shot prompting generated candidates, execution checked them, and humans fixed the remainder. Models of 6.7B and 30B parameters were then fine-tuned and evaluated with majority voting and reward-model reranking. Program CoTs generally beat natural language, and Python beat Wolfram. SDP performed best overall because its sampled solutions are more varied, which helps voting and reranking. The best 30B setup clearly outperforms few-shot GPT-3.5-turbo. These results give practical guidance on choosing a programming language and coding style for CoT.

Key points

  • With reward-model reranking, the 30B model beats few-shot GPT-3.5-turbo by roughly 2.9 points on GSM8K, 18 on MathQA and 8 on SVAMP.
  • With reward-weighted voting at 30B, Python SDP scores 79.5 (GSM8K), 77.9 (MathQA) and 86.0 (SVAMP). Natural language CoT reaches 71.2, 72.8 and 77.0.
  • NDP clearly underperforms SDP and CDP, so some natural language in program CoTs matters. Mixing different CoT types further improves results.
  • Annotation pipeline:
    • started from 20 hand-written seed examples per program style;
    • retrieved similar examples using OpenAI "text-embedding-ada-002" embeddings;
    • verified generated programs by executing them;
    • repeated the loop 3–5 times before manual fixes.
  • MathQA was cleaned of instances with wrong answers. Datasets and code are released at github.com/lqtrung1998/mwp_cot_design.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢ByteDanceBeijing · China54 works · 4,494 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksMathematics · 233 worksComputational Theory and Mathematics · 496 worksApplied Mathematics · 73 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Zhanming Jie, Trung Quoc Luong, Xinbo Zhang, Xiaoran Jin and Hang Li. 2023. Design of Chain-of-Thought in Math Problem Solving. arXiv:2309.11054.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.11054 · arXiv 2309.11054

@misc{jie2023design,
  title = {{Design of Chain-of-Thought in Math Problem Solving}},
  author = {Zhanming Jie, Trung Quoc Luong, Xinbo Zhang, Xiaoran Jin and Hang Li},
  year = {2023},
  doi = {10.48550/arxiv.2309.11054},
  eprint = {2309.11054},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.11054}
}

References

41 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 9(1):147–169, 1985. URL https://www.cs.toronto.edu/ ˜hinton/absps/cogscibm.pdf. ↗
  2. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of NAACL, 2019. URL https://arxiv.org/abs/1905.13319. ↗
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of NeurIPS, 2020. URL https://arxiv.org/abs/ 2005.14165. ↗
  4. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022. URL https://arxiv.org/abs/2211.12588. ↗
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. ↗
  6. Julian Coda-Forno, Marcel Binz, Zeynep Akata, Matthew Botvinick, Jane X Wang, and Eric Schulz. Meta-in-context learning in large language models. arXiv preprint arXiv:2305.12907, 2023. URL https://arxiv.org/abs/2305.12907. — [arXiv:2305.12907](https://arxiv.org/abs/2305.12907) ↗
  7. Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, et al. A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceedings of the National… — [arXiv:2112.15594](https://arxiv.org/abs/2112.15594) find
  8. Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. In Proceedings of EMNLP, 2017. URL https://aclanthology.org/W17-4912. ↗
  9. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. In Proceedings of ICLR, 2023. URL https://arxiv.org/abs/ 2210.00720. ↗
  10. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided language models. In Proceedings of ICML, 2023. URL https://arxiv.org/abs/2211.10435. ↗
  11. Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of ACL, 2021. URL https://aclanthology.org/2021. acl-long.295/. ↗
  12. Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. Roscoe: A suite of metrics for scoring step-by-step reasoning. In Proceedings of ICLR, 2022. URL https://arxiv.org/abs/2212.07919. ↗
  13. Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1,000 examples. arXiv preprint arXiv:2212.06713, 2022. URL https://arxiv.org/abs/2212.06713. — [arXiv:2212.06713](https://arxiv.org/abs/2212.06713) ↗
  14. Joy He-Yueya, Gabriel Poesia, Rose E Wang, and Noah D Goodman. Solving math word problems by combining language models with symbolic solvers. arXiv preprint arXiv:2304.09102, 2023. URL https://arxiv.org/abs/2304.09102. ↗
  15. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Proceedings of Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2…
  16. Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398, 2023. URL https://arxiv.org/abs/ 2303.05398. ↗
  17. Zhanming Jie, Jierui Li, and Wei Lu. Learning to reason deductively: Math word problem solving as complex relation extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5944–5955, 2022.
  18. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In Proceedings of ICLR, 2023. URL https://arxiv.org/abs/2210.02406. ↗
  19. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022. URL https://arxiv.org/abs/ 2203.07814. ↗
  20. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023. URL https://arxiv.org/abs/2305.20050. ↗
  21. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, 2022. URL https://a…
  22. Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. A survey of deep learning for mathematical reasoning. In Proceedings of ACL, 2023. URL https://arxiv.org/abs/ 2212.10535. ↗
  23. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. In Proceedings of NAACL, 2022. URL https://arxiv.org/abs/2110.15943. ↗
  24. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. URL https://arxiv.o… ↗
  25. OpenAI. GPT-4 technical report, 2023. URL https://arxiv.org/abs/2303.08774. ↗
  26. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? In Proceedings of NAACL, 2021. URL https://arxiv.org/abs/2103. 07191. ↗
  27. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. URL https://arxiv.org/… ↗
  28. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. URL https://arxiv.org/abs/2302. 04761. ↗
  29. Minghuan Tan, Lei Wang, Lingxiao Jiang, and Jing Jiang. Investigating math word problems using pretrained multilingual language models. In Proceedings of the 1st Workshop on Mathematical Natural Language Processing (MathNLP), pp. 7–16, 2022. find
  30. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022. URL https://arxiv.org/abs/2211. 09085. ↗
  31. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. URL https://arxiv.org/abs/2302.13971… ↗
  32. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. URL https://arxiv.org/abs/2307.09288. ↗
  33. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022. URL https://arxiv.org/abs/2211.14275. ↗
  34. Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995. find
  35. Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint arXiv:2307.10635, 2023a. URL https://arxiv.org/abs… ↗
  36. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In Proceedings of ICLR, 2023b. URL https://arxiv.org/abs/2203.11171. ↗
  37. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 8 2022a. URL https://arxiv.org/abs/2206.07682. ↗
  38. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of NeurIPS, 2022b. URL https://arxiv.org/abs/2201.11903. ↗
  39. Stephen Wolfram. An Elementary Introduction to the Wolfram Language. Wolfram Media, Inc.; 3rd edition, 2015. URL https://www.wolfram.com/language/ elementary-introduction/3rd-ed/. ↗
  40. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921, 2023a. URL https://arxiv.org/pdf/2308.079… ↗
  41. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. In Proceedings of ICLR, 2023b. URL https://arxiv.org/abs/2205.10625. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jie-2023-design-chain-thought-math-problem/