Elamur Research
☰
home / papers / 2023 / Measuring Faithfulness in Chain-of-Thought Reasoning

Measuring Faithfulness in Chain-of-Thought Reasoning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2307.13702

Summary

Step-by-step "Chain-of-Thought" (CoT) prompting makes LLMs more accurate. It is less clear whether the written reasoning actually explains how the model reaches its answer. The authors, mostly at Anthropic, built a set of interventions to test this. They cut the CoT short, inserted an error into it, reworded it, or replaced it entirely with meaningless dots. Then they checked whether the model's final answer changed. Each test targets one way the reasoning could fail to be faithful: rationalisation after the fact, hidden extra computation, or information hidden in the wording (a kind of steganography). The main experiments used a 175B-parameter RLHF-tuned assistant model on eight multiple-choice benchmarks. How much the model relied on its reasoning varied widely by task. Filler text gave no accuracy gain, and paraphrasing did not hurt it. On most tasks, larger and more capable models depended less on their stated reasoning. This matters for interpretability and safety: CoT can be a trustworthy explanation, but only under suitable choices of task and model size.

Key points

  • Benchmarks: ARC Challenge, ARC Easy, AQuA, HellaSwag, LogiQA, MMLU, OpenBookQA and TruthfulQA. For each question, 100 CoT samples were drawn with nucleus sampling (p = 0.95, temperature 0.8), then split into sentences with the NLTK punkt tokenizer. Chains averaged about 4 steps.
  • Accuracy: CoT helped on seven of the eight tasks, with AQuA gaining the most. HellaSwag got worse, dropping by about 4.7 points.
  • Early-answering test: on ARC Easy, ARC Challenge and OpenBookQA, the reasoning changed the final answer under 10% of the time. On AQuA it changed the answer over 60% of the time, so the reasoning there is largely not post-hoc. How much a task relied on its CoT did not track how much CoT improved accuracy on that task.
  • Filler and paraphrase tests: swapping the CoT for ellipses gave no benefit. Paraphrased reasoning performed about as well as the original. Together these argue against extra test-time compute or encoded wording being the source of CoT's gains.
  • Model size: faithfulness often got worse as models grew (inverse scaling). Models were also less faithful on easier versions of addition problems. If faithful reasoning matters, a smaller model may be the better choice.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

217 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 208 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.13702 · arXiv 2307.13702

@misc{lanham2023measuring,
  title = {{Measuring Faithfulness in Chain-of-Thought Reasoning}},
  author = {Lanham, Tamera and Chen, Anna and Radhakrishnan, Ansh and Steiner, Benoit and Denison, Carson and Hernandez, Danny and Li, Dustin and Durmus, Esin and Hubinger, Evan and Kernion, Jackson and Lukovsiute, Kamile and Nguyen, Karina and Cheng, Newton and Joseph, Nicholas and Schiefer, Nicholas and Rausch, Oliver and Larson, Robin and McCandlish, Sam and Kundu, Sandipan and Kadavath, Saurav and Yang, Shannon and Henighan, Thomas and Maxwell, Timothy and Telleen-Lawton, Timothy and Hume, Tristan and Hatfield-Dodds, Zac and Kaplan, Jared and Brauner, Jan and Bowman, Samuel R. and Perez, Ethan},
  year = {2023},
  doi = {10.48550/arxiv.2307.13702},
  eprint = {2307.13702},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.13702}
}

References

44 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andreas, J. Language models as agent models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5769–5779, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology .org/2022.findings-emnlp.423. find
  2. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C.… find
  3. Bird, S., Loper, E., and Klein, E. Natural Language Processing with Python. O’Reilly Media, Inc., USA, 2009. find
  4. Bowman, S. R., Hyun, J., Perez, E., Chen, E., Pettit, C., Heiner, S., Lukošiūtė, K., Askell, A., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Olah, C., Amodei, D., Amodei, D., Drain, D., Li, D., Tran-Johnson, E., Kernion, J., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndouss… find
  5. Branwen, G., 01 2023. URL https://www.lesswron g.com/posts/bwyKCQD7PFWKhELMr/by-def ault-gpts-think-in-plain-sight?comme ntId=zfzHshctWZYo8JkLe. ↗
  6. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… find
  7. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30.… find
  8. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1803.05457, 2018. find
  9. Creswell, A. and Shanahan, M. Faithful reasoning using large language models. arXiv preprint 2208.14271, 2022. find
  10. Creswell, A., Shanahan, M., and Higgins, I. Selection-inference: Exploiting large language models for interpretable logical reasoning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=3Pf3 Wg6o-A4. ↗
  11. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint 2305.14325, 2023. find our card
  12. Dua, D., Gupta, S., Singh, S., and Gardner, M. Successive prompting for decomposing complex questions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1251–1265, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.… find
  13. Ganguli, D., Askell, A., Schiefer, N., Liao, T. I., Lukošiūtė, K., Chen, A., Goldie, A., Mirhoseini, A., Olsson, C., Hernandez, D., Drain, D., Li, D., Tran-Johnson, E., Perez, E., Kernion, J., Kerr, J., Mueller, J., Landau, J., Ndousse, K., Nguyen, K., Lovitt, L., Sellitto, M., Elhage, N., Mercad… find our card
  14. Gao, L. Shapley value attribution in chain of thought. ht tps://www.lesswrong.com/posts/FX5Jmf tqL2j6K8dn4/shapley-value-attribution -in-chain-of-thought, 04 2023. find
  15. Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., and Yang, G.-Z. Xai&\#x2014;explainable artificial intelligence. Science Robotics, 4(37):eaay7120, 2019. doi: 10.1126/scirobotics.aay7120. URL https: //www.science.org/doi/abs/10.1126/sc irobotics.aay7120. — [doi:10.1126/scirobotics.aay7120](https://doi.org/10.1126/scirobotics.aay7120) find
  16. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://open review.net/forum?id=d7KBjmI3GmQ. find our card
  17. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rygG QyrFvH. ↗
  18. Holzinger, A., Biemann, C., Pattichis, C. S., and Kell, D. B. What do we need to build explainable ai systems for the medical domain? arXiv preprint 1712.09923, 2017. — [arXiv:1712.09923](https://arxiv.org/abs/1712.09923) find
  19. Jacovi, A. and Goldberg, Y. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4198– 4205, Online, July 2020. Association for Computational Linguistics. doi: 10.… find our card
  20. Lanham, T. Externalized reasoning oversight: a research direction for language model alignment, 08 2022. URL https://www.lesswrong.com/posts/FRRb 6Gqem8k69ocbi/externalized-reasoning -oversight-a-research-direction-for. ↗
  21. Li, S., Chen, J., Shen, Y., Chen, Z., Zhang, X., Li, Z., Wang, H., Qian, J., Peng, B., Mao, Y., Chen, W., and Yan, X. Explanations from large language models make small reasoners better. arXiv preprint 2210.06726, 2022. find
  22. Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214– 3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 1… find our card
  23. URL https://aclanthology.org/2022.ac l-long.229. Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: L… ↗
  24. Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y., and Zhang, Y. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Bessiere, C. (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 3622–3628. International… find our card
  25. Lyu, Q., Havaldar, S., Stein, A., Zhang, L., Rao, D., Wong, E., Apidianaki, M., and Callison-Burch, C. Faithful chain-of-thought reasoning. arXiv preprint 2301.13379, 2023. find our card
  26. Madaan, A. and Yazdanbakhsh, A. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint 2209.07686, 2022. find
  27. McKenzie, I. R., Lyzhov, A., Pieler, M., Parrish, A., Mueller, A., Prabhu, A., McLean, E., Kirtland, A., Ross, A., Liu, A., Gritsevskiy, A., Wurgaft, D., Kauffman, D., Recchia, G., Liu, J., Cavanagh, J., Weiss, M., Huang, S., Droid, T. F., Tseng, T., Korbak, T., Shen, X., Zhang, Y., Zhou, Z., Kim, N… find
  28. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–2391, Brussels, Belgium, October-November 2018. Associatio… find
  29. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training, 2018. URL https://s3-us-west-2.a mazonaws.com/openai-assets/research-c overs/language-unsupervised/language \_understanding\_paper.pdf. ↗
  30. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners, 2019. find
  31. Radhakrishnan, A., Nguyen, K., Kaplan, J., Brauner, J., Bowman, S. R., and Perez, E. Question decomposition improves the faithfulness of model-generated reasoning. arXiv preprint (released concurrently), 2023. find
  32. Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1:206–215, 05 2019. doi: 10.1038/s42256-019-0048-x. find our card
  33. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.… find
  34. Turpin, M., Michael, J., Perez, E., and Bowman, S. R. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint 2305.04388, 2023. find our card
  35. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, v… find
  36. Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., and Lim, E.-P. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint 2305.04091, 2023. find
  37. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. Rationale-augmented ensembles in language models. arXiv preprint 2207.00747, 2022. find
  38. Wei, J., Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F., Chi, E., Le, Q. V., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Syst… find our card
  39. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint 2305.10601, 2023a. find
  40. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023b. URL https://openreview.net/forum?id=WE\_v luYUL-X. ↗ our card
  41. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791– 4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi:… find
  42. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q. V., and Chi, E. H. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrev… find our card
  43. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P. F., and Irving, G. Fine-tuning language models from human preferences. CoRR, abs/1909.08593, 2019. URL http://arxiv.org/abs/1909.08593. ↗
  44. Figure 9. Probability of producing the same answer under paraphrasing as the unmodified reasoning sample. The dotted baselines are the probability that two IID reasoning samples would come to the same answer by chance alone. A. Adding Mistakes & Paraphrasing Prompts We present the prompts used for t… find