Elamur Research
☰
home / papers / 2022 / Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought

Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2210.01240

Summary

High accuracy on reasoning benchmarks does not show how large language models reach their answers. They may recall facts from pretraining or exploit shortcuts instead of reasoning. Saparov and He address this by checking the chain-of-thought (CoT) itself, step by step, rather than only the final label. They build PrOntoQA, a synthetic QA dataset inspired by ProofWriter. Each example comes from an ontology defined in first-order logic, together with a unique proof built by chaining modus ponens. The text is generated from simple templates, so a model's CoT can be parsed back into logical forms and compared with the gold proof. The authors test InstructGPT (text-davinci-002) and several GPT-3 variants, varying proof depth and whether the ontology is true, false or fictional. Individual deduction steps are usually valid, even with made-up concepts. Proof planning is the weak point: when several valid steps are available, models tend to commit greedily to one and can drift off the correct path. This failure mode is hidden by label-only evaluation.

Key points

  • Dataset design: PrOntoQA examples contain a context, a query, a CoT and a True/False label. Ontologies can be true, false (counterfactual) or fictional, such as "wumpus" and "rompus", and proof depth is controlled by the number of hops.
  • Models tested: text-ada-001, text-babbage-001, text-curie-001, davinci, text-davinci-001 and text-davinci-002.
  • Effect of world knowledge: models are misled by distractor steps much less often when the ontology matches real-world facts. Pretraining knowledge therefore shapes how well they reason.
  • Self-consistency barely helps: with 40 samples at temperature 0.7, valid-proof accuracy was 0.56 versus 0.545 for the baseline (5 hops, fictional ontology, 100 examples). Incorrect proofs often received higher probability than correct ones.
  • Depth-first-search demonstrations barely help: in-context examples showing a depth-first search reached 0.55, again not a significant gain. Code, data and model outputs are available at github.com/asaparov/prontoqa.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

198 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 189 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2210.01240 · arXiv 2210.01240

@misc{saparov2022language,
  title = {{Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought}},
  author = {Saparov, Abulhair and He, He},
  year = {2022},
  doi = {10.48550/arxiv.2210.01240},
  eprint = {2210.01240},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2210.01240}
}

References

37 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gabor Angeli and Christopher D. Manning. Naturalli: Natural logic inference for common sense reasoning. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar,… find
  2. ACL, 2014. doi: 10.3115/v1/d14-1059. URL https://doi.org/10.3115/v1/d14-1059. ↗
  3. Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay V. Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. CoRR, abs/2207.04901, 2022. doi: 10.48550/arXiv.2207.04901. URL https://doi.org/10.48550/a… find our card
  4. Gregor Betz. Critical thinking for language models. CoRR, abs/2009.07185, 2020. URL https: //arxiv.org/abs/2009.07185. find
  5. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive commonsense reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. find our card
  6. OpenReview.net, 2020. URL https://openreview.net/forum? id=Byg1v1HKDB. ↗
  7. Kaj Bostrom, Xinyu Zhao, Swarat Chaudhuri, and Greg Durrett. Flexible generation of natural language deductions. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Vi… find
  8. Kaj Bostrom, Zayne Sprague, Swarat Chaudhuri, and Greg Durrett. Natural language deduction through search over statement compositions. CoRR, abs/2201.06028, 2022. URL https:// arxiv.org/abs/2201.06028. find
  9. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  10. Jifan Chen, Eunsol Choi, and Greg Durrett. Can NLI models verify QA systems’ predictions? In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20… find
  11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… find
  12. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. ↗ our card
  13. Antonia Creswell and Murray Shanahan. Faithful reasoning using large language models. CoRR, abs/2208.14271, 2022. doi: 10.48550/arXiv.2208.14271. URL https://doi.org/10.48550/ arXiv.2208.14271. find
  14. Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. CoRR, abs/2205.09712, 2022. doi: 10.48550/ arXiv.2205.09712. URL https://doi.org/10.48550/arXiv.2205.09712. ↗
  15. Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. Language models show human-like content effects on reasoning. CoRR, abs/2207.07051, 2022. doi: 10.48550/arXiv.2207.07051. URL https://doi. org/10.48550/arXiv.2207.07051… find
  16. David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-Dickstein, Kevin Murphy, and Charles Sutton. Language model cascades. CoRR, abs/2207.10342, 2022. doi: 10.48550/arXiv. 2207.10342. URL https://doi.or… ↗
  17. Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. find
  18. OpenReview.net, 2019. URL https://openreview.net/forum?id= B1xY-hRctX. ↗
  19. Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq R. Joty, Alexander R. Fabbri,… find our card
  20. Pavan Kapanipathi, Ibrahim Abdelaziz, Srinivas Ravishankar, Salim Roukos, Alexander G. Gray, Ramón Fernandez Astudillo, Maria Chang, Cristina Cornelio, Saswati Dana, Achille Fokoue, Dinesh Garg, Alfio Gliozzo, Sairam Gurajada, Hima Karanam, Naweed Khan, Dinesh Khandelwal, Young-Suk Lee, Yunyao Li, F… find
  21. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. CoRR, abs/2206.14858… find our card
  22. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Associati… find our card
  23. Bill MacCartney and Christopher D. Manning. An extended model of natural logic. In Harry Bunt, Volha Petukhova, and Sander Wubben (eds.), Proceedings of the Eight International Conference on Computational Semantics, IWCS 2009, Tilburg, The Netherlands, January 7-9, 2009, pp. 140–156. Association for… find
  24. William Merrill, Ashish Sabharwal, and Noah A. Smith. Saturated Transformers are Constant-Depth Threshold Circuits. Transactions of the Association for Computational Linguistics, 10:843–856, 08 2022. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00493. URL https://doi.org/10.1162/ tacl\_a\_00493. find
  25. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training… find
  26. Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. CoRR, abs/2202.07206, 2022. URL https://arxiv.org/ abs/2202.07206. — [arXiv:2202.07206](https://arxiv.org/abs/2202.07206) ↗
  27. Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Pr… find
  28. Abulhair Saparov and Tom M. Mitchell. Towards general natural language understanding with probabilistic worldbuilding. Trans. Assoc. Comput. Linguistics, 10:325–342, 2022. doi: 10.1162/ tacl\\\_a\\\_00463. URL https://doi.org/10.1162/tacl\_a\_00463. ↗
  29. Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, A… find our card
  30. Karthik Valmeekam, Alberto Olmo Hernandez, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can’t plan (A benchmark for llms on planning and reasoning about change). CoRR, abs/2206.10498, 2022. doi: 10.48550/arXiv.2206.10498. URL https://doi. org/10.48550/arXiv.2206.10498. find our card
  31. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. CoRR, abs/2203.11171, 2022. doi: 10.48550/arXiv.2203.11171. URL https://doi.org/10.48550/arXiv.2203.11171. ↗ our card
  32. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903, 2022. URL https://arxiv.org/abs/2201.11903. ↗ our card
  33. Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hanna Hajishirzi, Yejin Choi, and Kyunghyun Cho. Naturalproofs: Mathematical theorem proving in natural language. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, Ne… find
  34. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track… find our card
  35. Edwin B Wilson. Probable inference, the law of succession, and statistical inference. J. Am. Stat. Assoc., 22(158):209, June 1927. find
  36. Hanlin Zhang, Yi-Fan Zhang, Li Erran Li, and Eric Xing. The impact of symbolic representations on in-context learning for few-shot reasoning. In Neuro Causal and Symbolic AI Workshop at NeurIPS 2022, Virtual Workshop, December 9, 2022, 2022a. URL https://openreview.net/ pdf?id=qLgQpeQX3x1. To appear… ↗
  37. Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. CoRR, abs/2205.11502, 2022b. doi: 10.48550/arXiv. 2205.11502. URL https://doi.org/10.48550/arXiv.2205.11502. — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) ↗