Elamur Research
☰
home / papers / 2023 / Faithful Chain-of-Thought Reasoning

Faithful Chain-of-Thought Reasoning

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.ijcnlp-main.20.pdf

Summary

Chain-of-Thought (CoT) prompting improves language models on complex reasoning. However, the explanation it produces is not causally tied to the predicted answer: a model can write a sensible-looking chain and then output an unrelated result. The authors address this lack of faithfulness with Faithful CoT, a two-stage pipeline. First, an LM (e.g., Codex, GPT-4) converts a natural-language question into a reasoning chain. The chain mixes NL comments, which split the problem into dependent subquestions, with code in a task-specific symbolic language such as Python, Datalog or PDDL. Second, a deterministic tool, such as an interpreter or a planner, executes that code to produce the answer. By construction, the chain is therefore the actual explanation of the output. The method was tested on 10 benchmarks covering math word problems, planning, multi-hop QA and relational inference. It beats standard CoT on nine of them and reaches a new few-shot state of the art on seven. The work suggests that interpretability and accuracy can reinforce each other, although the translation step itself remains opaque.

Key points

  • With Codex, Faithful CoT outperforms vanilla CoT on 9 of 10 datasets. The biggest relative accuracy gain, 21.4%, comes in Relational Inference, followed by 6.3% in MWP, 5.5% in Multi-hop QA and 3.4% in Planning.
  • Using GPT-4 and Codex, the approach sets new few-shot SOTA on 7 datasets and scores 95.0+ accuracy on 6 of them.
  • In a human evaluation, annotators judged reasoning chains fully correct in over 90% of cases for Sports, SayCan and MWP. The weakest scores were StrategyQA (66.7%), Date (87.9%) and CLUTRR (88.0%).
    • Many chains flagged as wrong were actually correct, including all such CLUTRR cases and 41.7% for Date.
    • In StrategyQA, faulty chains sometimes reached the right True/False answer by chance.
  • An ablation shows that removing the NL comments barely affects accuracy on many less knowledge-intensive datasets. The authors keep the comments for readability and possible user interaction.
  • OpenAI retired code-davinci-002 on March 23, 2023, which makes some results hard to reproduce. Code, data and prompts are released at github.com/veronica320/Faithful-COT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

228 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 217 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.ijcnlp-main.20 · arXiv 2301.13379

@inproceedings{lyu2023faithful,
  title = {{Faithful Chain-of-Thought Reasoning}},
  author = {Lyu, Qing and Havaldar, Shreya and Stein, Adam and Zhang, Li and Rao, Delip and Wong, Eric and Apidianaki, Marianna and Callison-Burch, Chris},
  booktitle = {Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.ijcnlp-main.20},
  eprint = {2301.13379},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.ijcnlp-main.20}
}

References

38 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jes-month, Nikhil… find our card
  2. Yoshua Bengio. 2019. FROM SYSTEM 1 DEEP LEARNING TO SYSTEM 2 DEEP LEARNING. find
  3. BIG-Bench collaboration. 2021. Beyond the Imitation Game: Measuring and extrapolating the capabilities of language models. find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… find
  6. Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, J… find our card
  7. Consistent Reasoning with Recursive Explanations. ArXiv:2205.11822 [cs]. find our card
  8. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. find our card
  9. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving Quantitative Reasoning Problems with Language Models. ArXiv:2206.14858… find our card
  10. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. ArXiv:2211.12588 [cs]. find our card
  11. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2022. On the Advance of Making Language Models Better Reasoners. ArXiv:2206.02336 [cs]. — [arXiv:2206.02336](https://arxiv.org/abs/2206.02336) find
  12. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. ArXiv:2110.14168 [cs]. find our card
  13. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167. Associati… find
  14. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. PAL: Program-aided Language Models. ArXiv:2211.10435 [cs]. find
  15. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. find
  16. Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining Explanations: An Overview of Interpretability of Machine Learning. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pages 80–89. find
  17. L. A. Harrington, M. D. Morley, A. Šcedrov, and S. G. Simpson. 1985. Harvey Friedman’s Research on the Foundations of Mathematics. Elsevier. Google-Books-ID: 2plPRR4LDxIC. find
  18. Bernease Herman. 2019. The Promise and Peril of Human Evaluation for Model Interpretability. arXiv:1711.07414 [cs, stat]. ArXiv: 1711.07414. — [arXiv:1711.07414](https://arxiv.org/abs/1711.07414) find
  19. Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205. Association for Computational Linguistics. find our card
  20. Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022. find
  21. Maieutic Prompting: Logically Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984, Online. Association for Com… find
  22. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show Your Work: Scratchpads for Intermediate Computation with Language Models. ArXiv:2112.00114 [cs]. find our card
  23. OpenAI. 2023. Gpt-4 technical report. find our card
  24. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Associat… find our card
  25. Silviu Pitis, Michael R Zhang, Andrew Wang, and Jimmy Ba. 2023. Boosted prompt ensembles for large language models. arXiv preprint arXiv:2304.05970. — [arXiv:2304.05970](https://arxiv.org/abs/2304.05970) find
  26. Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. 2020. Learning to deceive with attention-based explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4782– 4793, Online. Association for Computational Linguisti… find
  27. Jing Qian, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. 2022. Limitations of Language Models in Arithmetic and Symbolic Induction. ArXiv:2208.05051 [cs]. find
  28. Marco Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “why should I trust you?”: Explaining the predictions of any classifier. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, KDD ’16, pages 97–101, New York, NY,… find our card
  29. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. 2022. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. ArXiv:2205.10625 [cs]. find our card
  30. Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743–1752, Lisbon, Portugal. Association for Computational Linguistics. find our card
  31. Gabriele Sarti, Nils Feldhus, Ludwig Sickert, and Oskar van der Wal. 2023. Inseq: An interpretability toolkit for sequence generation models. arXiv preprint arXiv:2302.13942. find
  32. Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan… find
  33. Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020. Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 180–186. Association for Computing Machinery, New York, NY, USA… find
  34. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. find our card
  35. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ArXiv:2203.11171 [cs]. find our card
  36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. find our card
  37. Kayo Yin and Graham Neubig. 2022. Interpreting language models with contrastive explanations. arXiv preprint arXiv:2202.10419. find
  38. Hanlin Zhang, Ziyang Li, Jiani Huang, Mayur Naik, and Eric Xing. 2022. Improved logical reasoning of language models via differentiable symbolic programming. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022. find