Elamur Research⌕
☰
home / papers / 2025 / Thought Anchors: Which LLM Reasoning Steps Matter?

Thought Anchors: Which LLM Reasoning Steps Matter?

Paper · 2025 · Paul C. Bogdan, Uzay Macar, Neel Nanda et al.

Summary

Standard interpretability tools usually examine one forward pass, which suits reasoning models poorly because they produce long chains of thought that feed back into later generation. The authors propose treating the individual sentence as the unit of analysis. Their main black-box method resamples the continuation from each sentence about 100 times, keeps only semantically different replacements, and measures how the distribution of final answers shifts. Two white-box methods add to this. One finds "receiver" attention heads that focus on a few earlier "broadcasting" sentences. The other suppresses attention to a sentence and measures the KL divergence in later token logits. The experiments use DeepSeek R1-Distill Qwen-14B on hard MATH problems. A small number of sentences, called thought anchors, turn out to steer the trajectory of the reasoning, and these are mostly planning or uncertainty-management steps. The three methods give broadly consistent results. This matters because it offers a principled way to locate the decisive steps in a reasoning trace, which can help debug reasoning failures without relying on whether the CoT text is faithful.

Key points

  • Data and setup: The main model is DeepSeek R1-Distill Qwen-14B (48 layers, temperature 0.6, top-p 0.95), with an extra check on R1-Distill-Llama-8B. The authors screened 1,000 MATH problems, 10 attempts each, and kept 20 that were solved 25–75% of the time. They generated one correct and one incorrect trace per problem, for 40 traces averaging about 144 sentences and about 4,200 tokens.
  • Case study (MATH Problem 4682, bits in hex 66,666): Sentence 13 sharply raises accuracy. In it, the model decides to convert the number to decimal, which leads to the correct answer of 19 rather than 20. The older forced-answer baseline misses this sentence's importance.
  • Choice of granularity: Resampling at the sentence level recovers token-level patterns at much lower cost, while resampling whole paragraphs loses resolution.
  • Sentence categories and links: Sentences are labelled with an eight-category taxonomy adapted from Venhoff et al. (2025). Influence between pairs of sentences is scored by whether a later sentence still appears in resampled traces, using a cosine similarity threshold above 0.8.
  • Agreement between methods and tooling: The attention-suppression and resampling matrices correlate positively in 90% of traces (mean r = .20). The correlation rises to r = .34 for sentence pairs fewer than five sentences apart. An interactive visualizer is available at thought-anchors.com.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Paul C. Bogdan, Uzay Macar, Neel Nanda and Arthur Conmy. 2025. Thought Anchors: Which LLM Reasoning Steps Matter?. arXiv:2506.19143.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2506.19143 · arXiv 2506.19143

@misc{bogdan2025thought,
  title = {{Thought Anchors: Which LLM Reasoning Steps Matter?}},
  author = {Paul C. Bogdan, Uzay Macar, Neel Nanda and Arthur Conmy},
  year = {2025},
  doi = {10.48550/arxiv.2506.19143},
  eprint = {2506.19143},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.19143}
}

References

31 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. Chain-of-thought reasoning in the wild is not always faithful, 2025. URL https://arxiv.org/abs/2503.08679. ↗
  2. Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation, 2025. URL https://arxiv.org/abs/2503.11926. ↗
  3. Eric Bigelow, Ari Holtzman, Hidenori Tanaka, and Tomer Ullman. Forking paths in neural text generation, 2024. URL https://arxiv.org/abs/2412.07961. — [arXiv:2412.07961](https://arxiv.org/abs/2412.07961) ↗
  4. Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. Reasoning models don’t always say what they think, 2025. — Reasoning Models Don't Always Say What They Think
  5. DeepSeek. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948. ↗
  6. Leo Gao. Shapley value attribution in chain of https://www.lesswrong.com/posts/FX5JmftqL2j6K8dn4/ shapley-value-attribution-in-chain-of-thought. thought, 2023. ↗
  7. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks, 2021. URL https://arxiv.org/abs/2106.02997. ↗
  8. Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. Causal abstraction: A theoretical foundation for mechanistic interpretability, 2025. URL https://arxiv.org/abs/2301.04709. ↗
  9. Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. Roscoe: A suite of metrics for scoring step-by-step reasoning, 2023. URL https://arxiv.org/abs/2212.07919. ↗
  10. Goodfire. Under the hood of a reasoning model. https://www.goodfire.ai/blog/ under-the-hood-of-a-reasoning-model, 2025. Accessed: May 15, 2025. ↗
  11. Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, 2023. URL https://arxiv.org/abs/2305.00586. ↗
  12. Stefan Heimersheim and Jett Janiak. A circuit for python docstrings in a 4-layer attention-only transformer, 2023. https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/ a-circuit-for-python-docstrings-in-a-4-layer-attention-only. ↗
  13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  14. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874. ↗
  15. Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, et al. Chain of thought monitorability: A new and fragile opportunity for ai safety. arXiv preprint arXiv:2507.11473, 2025. find
  16. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Sa… — Measuring Faithfulness in Chain-of-Thought Reasoning
  17. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023b.
  18. Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persi… find
  19. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025. URL https://arxiv.org/abs/2501.19393. ↗
  20. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2021.
  21. Chris Olah. Mechanistic interpretability, variables, and the importance of interpretable bases. https://www.transformer-circuits.pub/2022/mech-interp-essay, 2022. ↗
  22. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distill.00024.001. — [doi:10.23915/distill.00024.001](https://doi.org/10.23915/distill.00024.001) find
  23. OpenAI. o1: Introducing our first reasoning model. https://openai.com/o1/, 2024. Accessed: 2025-05-15. ↗
  24. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019.
  25. Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm, 2021.
  26. Rohin Shah, Alex Irpan, Alexander Matt Turner, Anna Wang, Arthur Conmy, David Lindner, Jonah Brown-Cohen, Lewis Ho, Neel Nanda, Raluca Ada Popa, Rishub Jain, Rory Greig, Samuel Albanie, Scott Emmons, Sebastian Farquhar, Sébastien Krier, Senthooran Rajamanoharan, Sophie Bridgers, Tobi Ijitoye, Tom Ev… — [arXiv:2504.01849](https://arxiv.org/abs/2504.01849) find
  27. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting, 2023. URL https://arxiv.org/abs/2305.04388. ↗
  28. Constantin Venhoff, Iván Arcuschin, Philip Torr, Arthur Conmy, and Neel Nanda. Understanding reasoning in thinking language models via steering vectors. In Workshop on Reasoning and Planning for Large Language Models, 2025. find
  29. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022. URL https://arxiv.org/abs/2211.00593. ↗
  30. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903. ↗
  31. Skyler Wu, Eric Meng Shen, Charumathi Badrinath, Jiaqi Ma, and Himabindu Lakkaraju. Analyzing chain-of-thought prompting in large language models via gradient-based feature attributions, 2023. URL https://arxiv.org/abs/2307.13339. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bogdan-2025-thought-anchors-which-llm-reasoning/