Elamur Research⌕
☰
home / papers / 2024 / The Impact of Reasoning Step Length on Large Language Models

The Impact of Reasoning Step Length on Large Language Models

Summary

Chain of Thought (CoT) prompting helps large language models reason, but it is unclear which part of a CoT prompt drives the gains. The authors test whether the number of reasoning steps is the decisive factor. In controlled experiments, they stretch or shorten the rationales in CoT demonstrations without adding or removing actual knowledge. For few-shot prompting, they lengthen Auto-CoT and Manual-CoT examples with filler-style strategies such as restating the question or verifying intermediate states. For zero-shot prompting, they add an instruction asking the model to reason in more steps. Across eight reasoning datasets and several OpenAI models, longer chains consistently raised accuracy, and compressed chains dropped performance to roughly zero-shot level. Rationales with wrong intermediate values still helped if they kept their length. The benefit depended on the task, with harder problems gaining most. These results give practitioners a simple, measurable lever for designing prompts and add to the evidence on why CoT works.

Key points

  • To expand rationales, the authors have the model reflect on a word, reread the question, restate the current state, and apply "Self-Verification" and "Make Equation". All of them add steps without adding new information.
  • Models tested were text-davinci-002, GPT-3.5-turbo-1106 and GPT-4. On GSM8K, the weakest model (text-davinci-002) gained the most, and GPT-4 showed no performance drop from added steps.
  • With GPT-3.5-turbo-1106, adding steps raised Auto-CoT accuracy on GSM8K from 65.8 to 78.8 and on MultiArith from 94 to 97.2. Manual-CoT on GSM8K rose from 64.7 to 70.1.
  • Altering the numbers in demonstration questions and rationales, so that they became factually wrong, changed accuracy only slightly. For example, Manual-CoT on MultiArith went from 93.5 to 92.9.
  • In few-shot CoT, accuracy grew roughly linearly with step count. Simple tasks needed fewer steps, while complex math tasks benefited from longer chains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

84 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-acl.108

@inproceedings{jin2024impact,
  title = {{The Impact of Reasoning Step Length on Large Language Models}},
  author = {Jin, Mingyu and Yu, Qinkai and Shu, Dong and Zhao, Haiyan and Hua, Wenyue and Meng, Yanda and Zhang, Yongfeng and Du, Mengnan},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.108},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.108}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. find our card
  3. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-based prompting for multi-step reasoning. Preprint, arXiv:2210.00720. find our card
  4. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. find
  5. Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, and Yongfeng Zhang. 2024. Exploring concept depth: How large language models acquire knowledge at different layers? Preprint, arXiv:2404.07066. — [arXiv:2404.07066](https://arxiv.org/abs/2404.07066) find
  6. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large language models are zero-shot reasoners. Preprint, arXiv:2205.11916. find our card
  7. Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597. find
  8. Yingcong Li, Kartik Sreenivasan, Angeliki Giannou, Dimitris Papailiopoulos, and Samet Oymak. 2023. Dissecting chain-of-thought: A study on compositional in-context learning of mlps. arXiv preprint arXiv:2305.18869. — [arXiv:2305.18869](https://arxiv.org/abs/2305.18869) find
  9. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167. find
  10. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. Preprint, arXiv:2301.13379. find our card
  11. Aman Madaan and Amir Yazdanbakhsh. 2022. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint arXiv:2209.07686. find
  12. William Merrill and Ashish Sabharwal. 2023. The expressive power of transformers with chain of thought. Preprint, arXiv:2310.07923. find
  13. R OpenAI. 2023. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2:13. find our card
  14. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. find
  15. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094. find our card
  16. Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find our card
  17. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are emergent abilities of large language models a mirage? arXiv preprint arXiv:2304.15004. find our card
  18. Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Synthetic prompting: Generating chain-of-thought demonstrations for large language models. Preprint, arXiv:2302.00618. — [arXiv:2302.00618](https://arxiv.org/abs/2302.00618) find
  19. Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song-Chun Zhu, Yitao Liang, and Muhan Zhang. 2023. Large language models are in-context semantic reasoners rather than symbolic reasoners. arXiv preprint arXiv:2305.14825. find
  20. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. Preprint, arXiv:2203.11171. find our card
  21. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837. find our card
  22. Skyler Wu, Eric Meng Shen, Charumathi Badrinath, Jiaqi Ma, and Himabindu Lakkaraju. 2023. Analyzing chain-of-thought prompting in large language models via gradient-based feature attributions. arXiv preprint arXiv:2307.13339. find
  23. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Preprint, arXiv:2305.10601. find
  24. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jin-2024-impact-reasoning-step-length-large/