Elamur Research⌕
☰
home / papers / 2025 / Towards Understanding Distilled Reasoning Models: A Representational Approach

Towards Understanding Distilled Reasoning Models: A Representational Approach

Paper · 2025 · David D. Baek and Max Tegmark

Summary

Distilling reasoning traces from strong models such as DeepSeek-R1 into smaller LLMs works well in practice, but little is known about what distillation changes inside a model. Baek and Tegmark use model diffing to study this. They train a sparse crosscoder jointly on Qwen base models and their DeepSeek-R1-Distill-Qwen counterparts (1.5B, 7B, 14B). The training data mix reasoning traces from OpenThoughts-114k with general text from RedPajama. To separate shared features from model-specific ones, they compare each feature's decoder norms across the two models.

Most features turn out to be shared. The features specific to the distilled models, however, capture recognizable reasoning behaviours, such as self-correction (often marked by "wait") and checking computations. Four reasoning categories are examined in depth. Ablation and steering along these directions can push a distilled model into over-thinking or into terser, incisive reasoning. A feature-geometry analysis suggests that larger distilled models form more structured representations, which appears linked to more successful distillation. The work offers an interpretability lens on distillation, relevant to transparency and trust in reasoning models.

Key points

  • Crosscoder setup: 32,768 features, trained to reconstruct the residual stream at the half-depth layer of each model.
  • Training data: 200M tokens from open-thoughts/OpenThoughts-114k (DeepSeek-R1 math, science and code traces) plus 200M tokens from togethercomputer/RedPajama-Data-1T-Sample.
  • Measuring uniqueness: features are scored by a relative decoder norm and its normalized form (NRN). A value near 0.5 indicates a shared feature, and a value near 1 indicates a feature unique to the distilled model. The NRN distribution peaks at shared features with exponentially thinning tails, and the 14B pair shows the highest average NRN.
  • Feature labelling and reasoning types: GPT-4o-mini annotated the 100 top-ranked and 100 bottom-ranked features. The four reasoning types studied are self-reflection, deductive, alternative and contrastive reasoning. In one example, steering makes the distilled model overthink a trivial "5-1" question.
  • Geometry analysis: representations are evaluated with a parallelogram (analogy) loss on word-pair relations: English–Spanish/German/French, present–past and singular–plural. Activations are projected via PCA to 2, 5, 10 and 20 dimensions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,948 worksEfficient models & compression · 328 works

Cite

Cite as David D. Baek and Max Tegmark. 2025. Towards Understanding Distilled Reasoning Models: A Representational Approach. arXiv:2503.03730 · ICLR 2025 Workshop on Building Trust in Language Models and Applications.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.03730 · arXiv 2503.03730

@misc{baek2025understanding,
  title = {{Towards Understanding Distilled Reasoning Models: A Representational Approach}},
  author = {David D. Baek and Max Tegmark},
  year = {2025},
  doi = {10.48550/arxiv.2503.03730},
  eprint = {2503.03730},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.03730}
}

References

38 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. David D Baek, Yuxiao Li, and Max Tegmark. Generalization from starvation: Hints of universality in llm knowledge graph learning. arXiv preprint arXiv:2410.08255, 2024. — [arXiv:2410.08255](https://arxiv.org/abs/2410.08255) find
  3. David D Baek, Ziming Liu, Riya Tyagi, and Max Tegmark. Harmonic loss trains interpretable ai models. arXiv preprint arXiv:2502.01628, 2025. — [arXiv:2502.01628](https://arxiv.org/abs/2502.01628) find
  4. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah… find
  5. Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733, 2016.
  6. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. find
  7. Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are linear. arXiv preprint arXiv:2405.14860, 2024. — [arXiv:2405.14860](https://arxiv.org/abs/2405.14860) find
  8. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024. find
  9. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  10. Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023. find
  11. Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  12. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
  13. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024.
  14. Ole Jorgensen, Dylan Cope, Nandi Schoots, and Murray Shanahan. Improving activation steering in language models with mean-centring. arXiv preprint arXiv:2312.03813, 2023. — [arXiv:2312.03813](https://arxiv.org/abs/2312.03813) find
  15. Subhash Kantamneni and Max Tegmark. Language models use trigonometry to do addition. arXiv preprint arXiv:2502.00873, 2025.
  16. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  17. Belinda Z Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint arXiv:2106.00737, 2021. find
  18. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147, 2024.
  19. Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread, 2024. find
  20. Ziming Liu, Eric J Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01117, 2022.
  21. Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023.
  22. Eric J Michaud, Isaac Liao, Vedang Lad, Ziming Liu, Anish Mudide, Chloe Loughridge, Zifan Carl Guo, Tara Rezaei Kheirkhah, Mateja Vukelić, and Max Tegmark. Opening the ai black box: program synthesis via mechanistic interpretability. arXiv preprint arXiv:2402.05110, 2024. — [arXiv:2402.05110](https://arxiv.org/abs/2402.05110) find
  23. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024–001, 2020. find
  24. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. Transformer Circuits Thread, 2022.
  25. Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. arXiv preprint arXiv:2406.01506, 2024. — [arXiv:2406.01506](https://arxiv.org/abs/2406.01506) find
  26. Gonçalo Paulo, Stepan Shabalin, and Nora Belrose. Transcoders beat sparse autoencoders for interpretability. arXiv preprint arXiv:2501.18823, 2025. — [arXiv:2501.18823](https://arxiv.org/abs/2501.18823) find
  27. Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668, 2018. — [arXiv:1802.05668](https://arxiv.org/abs/1802.05668) find
  28. Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435, 2024. — [arXiv:2407.14435](https://arxiv.org/abs/2407.14435) find
  29. Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. Memory injections: Correcting multi-hop reasoning failures during inference in transformer-based language models. arXiv preprint arXiv:2309.05605, 2023. find
  30. Harshay Shah, Sung Min Park, Andrew Ilyas, and Aleksander Madry. Modeldiff: A framework for comparing learning algorithms. In International Conference on Machine Learning, pp. 30646– 30688. PMLR, 2023. find
  31. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, et al. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. transformer circuits thread, 2024. find
  32. Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15213, 2023.
  33. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248, 2023.
  34. Teun van der Weij, Massimo Poesio, and Nandi Schoots. Extending activation steering to broad skills and multiple behaviours. arXiv preprint arXiv:2403.05767, 2024. — [arXiv:2403.05767](https://arxiv.org/abs/2403.05767) find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.
  37. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024. — [arXiv:2402.13116](https://arxiv.org/abs/2402.13116) find
  38. Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36, 2024. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baek-2025-towards-understanding-distilled-reasoning-models/