Automatic Chain of Thought Prompting in Large Language Models
Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2210.03493
Summary
Chain-of-thought (CoT) prompting helps large language models solve multi-step problems. The best-performing variant, Manual-CoT, needs people to write example questions with worked reasoning, and each task needs its own set. This paper replaces that manual work with Auto-CoT. The authors first show that simple automation does not work well. If you retrieve questions similar to the test question and let Zero-Shot-CoT ("Let's think step by step") write their rationales, the model's reasoning errors tend to repeat across the demonstrations. Their fix is to make the demonstration questions diverse. Auto-CoT encodes the questions of a dataset with Sentence-BERT and groups them into clusters. It then takes one representative question from each cluster, has Zero-Shot-CoT generate its reasoning chain, and filters these chains with simple heuristics. Experiments used GPT-3 (text-davinci-002, 175B) on ten arithmetic, commonsense and symbolic reasoning benchmarks. Auto-CoT matched or beat Manual-CoT on every one of them. This matters because good CoT prompting becomes practical for any task without hand-written demonstrations, and each dataset gets its own prompts.
Key points
- Benchmarks: MultiArith, GSM8K, AddSub, AQuA, SingleEq, SVAMP (arithmetic); CSQA, StrategyQA (commonsense); Last Letter Concatenation, Coin Flip (symbolic). Only unlabeled test questions are used, with no training set.
- With GPT-3, Auto-CoT vs Manual-CoT: MultiArith 92.0 vs 91.7, GSM8K 47.9 vs 46.9, CSQA 74.4 vs 73.5, Coin Flip 99.9 vs 97.2. On StrategyQA both scored 65.4.
- With Codex, Auto-CoT stays competitive. It is ahead on GSM8K (62.8 vs 59.4) and AddSub (91.9 vs 84.6) but behind on MultiArith (93.2 vs 96.8).
- Diverse sampling makes the method robust to bad demonstrations: on MultiArith, accuracy barely drops even when half the demonstrations have wrong answers. Sampling from the test question's own cluster is much less robust.
- A bootstrapping variant, Auto-CoT*, handles questions that arrive in batches of 30. From the second batch on, it performs about as well as Manual-CoT. Code is at github.com/amazon-research/auto-cot.
Cited by
- Many-Shot In-Context Learning
- Quiet-STaR: Language models can teach themselves to think before speaking
- A Survey on In-context Learning
- A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
- Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
- Exploring Human-Like Translation Strategy with Large Language Models
- Large Language Models Can Self-Improve
- Towards Reasoning in Large Language Models: A Survey
- TrustLLM: Trustworthiness in Large Language Models
- Challenges and Applications of Large Language Models
- Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
- Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
- Deductive Verification of Chain-of-Thought Reasoning
- Query Rewriting in Retrieval-Augmented Large Language Models
- DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction
- Reasoning with Language Model Prompting: A Survey
- Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
- A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
- Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
- Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration
- Large Language Models are Better Reasoners with Self-Verification
- Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
- Multimodal Chain-of-Thought Reasoning in Language Models
- A Survey of Large Language Models
- AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Cite
@misc{zhang2022automatic,
title = {{Automatic Chain of Thought Prompting in Large Language Models}},
author = {Zhang, Zhuosheng and Zhang, Aston and Li, Mu and Smola, Alex},
year = {2022},
doi = {10.48550/arxiv.2210.03493},
eprint = {2210.03493},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2210.03493}
}References
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
- Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzho… find our card
- Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen… find
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… find
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022), 2022a. URL https://arxiv.org/abs/2201.11903. ↗ our card
- Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022), 2022. URL https://arxiv.org/abs/2205.11916. ↗ our card
- Subhro Roy and Dan Roth. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743–1752, Lisbon, Portugal, 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1202. URL https://aclanthology.org/D1… ↗ our card
- Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… find
- Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051, 2021. doi: 10.18653/v1/2021.emnlp-ma… find our card