Elamur Research⌕
☰
home / papers / 2026 / Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization

Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization

Paper · 2026 · Kerem Zaman and Shashank Srivastava

Summary

The paper questions a popular argument that chain-of-thought (CoT) reasoning cannot serve as an explanation of LLM behaviour. That argument mostly relies on the Biasing Features test. In this test, a hint is inserted into the prompt, and if the hint changes the answer but the CoT never mentions it, the CoT is labelled unfaithful. The authors argue that this test mistakes incompleteness for unfaithfulness. Any natural-language account of distributed transformer computation has to be a lossy summary.

They first reproduce the pessimistic Biasing Features results on three multi-hop QA datasets with small instruction-tuned models. They then re-examine the flagged CoTs in three ways:

  • scoring them with two alternative faithfulness metrics;
  • sampling more CoTs per example using a new faithful@k metric;
  • applying Causal Mediation Analysis to check whether non-verbalizing CoTs still carry the hint's influence.

All three analyses weaken the claim that CoTs are unfaithful, and the findings extend to larger and reasoning-specialized models. The authors do not claim that CoTs are always faithful. Their point is that hint-based checks should be complemented by corruption-based and causal methods, and that training models just to verbalize hints could reward the wrong goal.

Key points

  • Setup: The experiments use OpenbookQA, StrategyQA and ARC-Easy with Llama-3-8B-Instruct, Llama-3.2-3B-Instruct and gemma-3-4b-it. There are three hint types (Professor, Metadata, Black Squares). Hint verbalization is judged by gpt-oss-20b through DSPy, which agrees with manual annotation in 80% of cases.
  • Baseline result: Biasing Features labels at least 80% of answer-flipping cases as unfaithful in every model, dataset and hint combination.
  • Metric disagreement: Filler Tokens and FUR judge many of these "unfaithful" CoTs to be faithful, and for some models this applies to more than half of them.
  • faithful@k: With a larger inference-time budget, the chance that at least one sampled CoT verbalizes the hint rises to as much as 90%. This growth is strong for Professor hints but stays nearly flat for Metadata and Black Squares.
  • Causal mediation: Causal Mediation Analysis shows significant indirect effects, meaning CoTs that never mention the hint still partly mediate its effect on the answer. The authors also note a limitation: the judge has low precision (36%) and recall (31%), but a stricter lexical check does not change the conclusions. Code is available at github.com/KeremZaman/IsCotExplainability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of North Carolina at Chapel HillChapel Hill · United States67 works · 16,474 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksInterpretability · 130 works

Cite

Cite as Kerem Zaman and Shashank Srivastava. 2026. Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2217

@inproceedings{zaman2026is,
  title = {{Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization}},
  author = {Kerem Zaman and Shashank Srivastava},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2217},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2217}
}

References

45 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. OpenAI Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Hai-Biao Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sébastien Bubeck, Cheng Chang, Kai Chen, and 105 others. 2025. gpt-oss-120b…
  2. Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. 2025. Chain-of-thought reasoning in the wild is not always faithful. arXiv preprint arXiv:2503.08679. find
  3. Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. 2023. Faithfulness Tests for Natural Language Explanations. ArXiv, abs/2305.18029.
  4. Fazl Barez, Tung-Yu Wu, Iván Arcuschin, Michael Lan, Vincent Wang, Noah Siegel, Nicolas Collignon, Clement Neo, Isabelle Lee, Alasdair Paren, Adel Bibi, Robert Trager, Damiano Fornasiere, John Yan, Yanai Elazar, and Yoshua Bengio. 2025. Chain-of-Thought Is Not Explainability. Preprint, alphaXiv:2025… find
  5. Steven Bird and Edward Loper. 2004. NLTK: The Natural Language Toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain. Association for Computational Linguistics.
  6. Aaron Chan, Shaoliang Nie, Liang Tan, Xiaochang Peng, Hamed Firooz, Maziar Sanjabi, and Xiang Ren. 2022. FRAME: Evaluating Rationale-Label Consistency Metrics for Free-Text Rationales. find
  7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 34 others. 2021. Evaluating L…
  8. Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson E. Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vladimir Mikulik, Sam Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. 2025. Reasoning Models Don’t Always Say What They Think. ArXiv, abs/2505.05410… find
  9. James Chua and Owain Evans. 2025. Are DeepSeek R1 And Other Reasoning Models More Faithful? find
  10. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1.
  11. Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretable Machine Learning. arXiv: Machine Learning.
  12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others.… — The Llama 3 Herd of Models
  13. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. find
  14. Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. 2023. AI control: Improving safety despite intentional subversion. arXiv preprint arXiv:2312.06942. — [arXiv:2312.06942](https://arxiv.org/abs/2312.06942) find
  15. Alon Jacovi and Yoav Goldberg. 2020. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? In Annual Meeting of the Association for Computational Linguistics.
  16. Gemma Team Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram’e, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  17. Adam Karvonen and Samuel Marks. 2025. Robustly Improving LLM Fairness in Realistic Settings via Interpretability. ArXiv, abs/2506.10922. — [arXiv:2506.10922](https://arxiv.org/abs/2506.10922) find
  18. Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-Search-Predict: Composing Retrieval and Language Models for Knowledge-Intensive NLP. arXiv preprint arXiv:2212.14024.
  19. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines.
  20. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  21. Tomasz Korbak, Mikita Balesni, Eliza beth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, Scott Emmons, Owain Evans, David Farhi, Ryan Greenblatt, Dan Hendrycks, Marius Hobbhahn, Evan Hubinger, Geoffrey Irving, Erik Jenner, and 22 others. 2025. Chai… find
  22. Tamera Lanham. 2022. Externalized reasoning oversight: a research direction for language model alignment. In AI Alignment Forum. find
  23. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
  24. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson E. Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, John Kernion, Kamil.e Lukovsiut.e, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, an… — Measuring Faithfulness in Chain-of-Thought Reasoning
  25. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. find
  26. Aengus Lynch, Benjamin Wright, Caleb Larson, Stuart J Ritchie, Soren Mindermann, Evan Hubinger, Ethan Perez, and Kevin Troy. 2025. Agentic misalignment: Peter Hase and Mohit Bansal. 2020. Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? In Annual Meeting o…
  27. How llms could be insider threats. arXiv preprint arXiv:2510.05179. — [arXiv:2510.05179](https://arxiv.org/abs/2510.05179) find
  28. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association fo… find
  29. Richard Ngo, Lawrence Chan, and Sören Mindermann. 2024. The Alignment Problem from a Deep Learning Perspective. In The Twelfth International Conference on Learning Representations. find
  30. Nostalgebraist. 2020. Interpreting gpt: The logit lens. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens. Less-Wrong blog post. ↗
  31. Letitia Parcalabescu and Anette Frank. 2023. On Measuring Faithfulness or Self-consistency of Natural Language Explanations. In Annual Meeting of the Association for Computational Linguistics. find
  32. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others. 2019.… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  33. Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. 2024. Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15012– 15032, Miami, Florida, USA. Association for Computat…
  34. Judea Pearl. 2001. Direct and Indirect Effects. Probabilistic and Causal Inference. find
  35. Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. 2023. Large Language Model Alignment: A Survey. ArXiv, abs/2309.15025.
  36. Noah Siegel, Oana-Maria Camburu, Nicolas Manfred Otto Heess, and María Pérez-Ortiz. 2024. The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Models. ArXiv, abs/2404.03189. find
  37. Miles Turpin, Andy Arditi, Marvin Li, Joe Benton, and Julian Michael. 2025. Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning. ArXiv, abs/2506.22777. — [arXiv:2506.22777](https://arxiv.org/abs/2506.22777) find
  38. Miles Turpin, Julian Michael, Ethan Perez, and Sam Bowman. 2023. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. ArXiv, abs/2305.04388.
  39. Martin Tutek, Fateme Hashemi Chaleshtori, Ana Marasovic, and Yonatan Belinkov. 2025. Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9946–9971, Suzhou, China. Association for Compu…
  40. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  41. Sarah Wiegreffe, Ana Marasović, and Noah A. Smith. 2020. Measuring Association Between Labels and Free-Text Rationales. In Conference on Empirical Methods in Natural Language Processing.
  42. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  43. Kerem Zaman and Shashank Srivastava. 2025. A Causal Lens for Evaluating Faithfulness Metrics. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29413–29437, Suzhou, China. Association for Computational Linguistics. find
  44. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning. ArXiv, abs/2404.05868.
  45. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: efficient execution of structured language model programs. In Proceedings of the 38th International Conference…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaman-2026-is-chain-thought-really-not/