Elamur Research⌕
☰
home / papers / 2025 / Feature Extraction and Steering for Enhanced Chain-of-Thought Reasoning in Language Models

Feature Extraction and Steering for Enhanced Chain-of-Thought Reasoning in Language Models

Paper · 2025 · Zihao Li, Xu Wang, Yuzhe Yang et al.

Summary

Long chain-of-thought reasoning, as in DeepSeek-R1, improves LLM performance on hard problems. Obtaining it usually requires distillation or post-training on expensive long-CoT data. The authors instead strengthen reasoning at inference time by steering residual activations, using only ordinary (vanilla) CoT.

Their VS Decomposition works as follows. GPT-4.1 rewrites each problem's solution into a natural-language Verbal process and a formal Symbolic process. Sparse Autoencoder features are averaged over each version, and their absolute difference is taken. Generic features shared by both versions, such as punctuation, cancel out, while reasoning-related features remain and are then used to steer generation.

For models without pre-trained SAEs, an SAE-free variant takes steering directions from the leading eigenvectors of a matrix built from verbal–symbolic activation differences.

On four math benchmarks, the SAE-based method mostly raises accuracy. The SAE-free version consistently beats the BoostStep and MathNeuro baselines and sometimes surpasses SAE-based steering. The approach matters because it needs no retraining, no contrastive long/short CoT pairs and no curated long-CoT corpus.

Key points

  • Feature extraction uses only 100 problems sampled from the MathOAI dataset. Verbal and symbolic versions of each solution are generated with the GPT-4.1 API.
  • SAE-based steering results on DeepSeek-Llama3-8B:
    • GSM8K: 82.67 → 85.67
    • MMLU-high: 72.96 → 76.30
    • MathOAI: 78.60 → 83.40
  • SAE-based steering is not uniformly positive on MATH-L3&L4: Llama3.1-8B-it is unchanged (21.46), and Gemma-2-9b-it drops slightly (51.07 → 50.64).
  • The SAE-free algorithm was tested on DeepSeek-Llama3-8B and DeepSeek-qwen-1.5B (which has no available SAE) at layer 15. It outperformed SAE-based steering on MATH-L3&L4 and MMLU-high.
  • Attention analysis indicates that steered models focus more on tokens carrying mathematical content. Evaluation is limited to math tasks and three model families (LLaMA, DeepSeek, Gemma).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓University of California, Santa BarbaraSanta Barbara · United States57 works · 9,752 citations🎓George Mason UniversityFairfax · United States24 works · 1,288 citations🎓New Jersey Institute of TechnologyNewark · United States15 works · 1,333 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Zihao Li, Xu Wang, Yuzhe Yang, Ziyu Yao, Haoyi Xiong and Mengnan Du. 2025. Feature Extraction and Steering for Enhanced Chain-of-Thought Reasoning in Language Models. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.552

@inproceedings{li2025feature,
  title = {{Feature Extraction and Steering for Enhanced Chain-of-Thought Reasoning in Language Models}},
  author = {Zihao Li, Xu Wang, Yuzhe Yang, Ziyu Yao, Haoyi Xiong and Mengnan Du},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.552},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.552}
}

References

53 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Bryan R Christ, Zack Gottesman, Jonathan Kropko, and Thomas Hartvigsen. 2024. Math neurosurgery: Isolating language models’ math reasoning abilities using only forward passes. arXiv preprint arXiv:2410.16930.
  3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168, 9. ↗
  4. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600. find
  5. Eoin Farrell, Yeu-Tong Lau, and Arthur Conmy. 2024. Applying sparse autoencoders to unlearn knowledge in language models. arXiv preprint arXiv:2410.19278. — [arXiv:2410.19278](https://arxiv.org/abs/2410.19278) find
  6. Andrey Galichin, Alexey Dontsov, Polina Druzhinina, Anton Razzhigaev, Oleg Y Rogov, Elena Tutubalina, and Ivan Oseledets. 2025. I have covered all the bases here: Interpreting reasoning features in large language models via sparse autoencoders. arXiv preprint arXiv:2503.18878. — [arXiv:2503.18878](https://arxiv.org/abs/2503.18878) find
  7. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093. find
  8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  9. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  10. Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, et al. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. arXiv preprint arXiv:2410.20526. find
  11. Zirui He, Haiyan Zhao, Yiran Qiao, Fan Yang, Ali Payani, Jing Ma, and Mengnan Du. 2025. Saif: A sparse autoencoder framework for interpreting and steering instruction following of language models. arXiv preprint arXiv:2502.11356. — [arXiv:2502.11356](https://arxiv.org/abs/2502.11356) find
  12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).
  13. Bertram Højer, Oliver Jarvis, and Stefan Heinrich. 2025. Improving reasoning performance in large language models via representation engineering. arXiv preprint arXiv:2504.19483. — [arXiv:2504.19483](https://arxiv.org/abs/2504.19483) find
  14. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
  15. Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. 2024. Measuring progress in dictionary learning for language model interpretability with board game models. Advances in Neural Information Processing Systems, 3… find
  16. Dmitrii Kharlapenko, Stepan Shabalin, Fazl Barez, Neel Nanda, and Arthur Conmy. Scaling sparse feature circuits for studying in-context learning. find
  17. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451–41530.
  18. Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ninghao Liu, and Mengnan Du. 2024. Quantifying multilingual performance of large language models across languages. arXiv e-prints, pages arXiv–2404. find
  19. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147. find
  20. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.
  21. Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2024. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647. find
  22. Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824.
  23. Callum McDougall. 2024. SAE Visualizer. https://github.com/callummcdougall/sae_vis. ↗
  24. Xin Quan, Marco Valentino, Danilo S Carvalho, Dhairya Dalal, and André Freitas. 2025. Peirce: Unifying material and formal reasoning via llm-driven neuro-symbolic refinement. arXiv preprint arXiv:2504.04110. find
  25. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. 2024a. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014.
  26. Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. 2024b. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435. — [arXiv:2407.14435](https://arxiv.org/abs/2407.14435) find
  27. Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. 2025. A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models. find
  28. Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. 2025. Fastcurl: Curriculum reinforcement learning with progressive context extension for efficient training r1-like reasoning models. arXiv preprint arXiv:2503.17287. — [arXiv:2503.17287](https://arxiv.org/abs/2503.17287) find
  29. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372. find
  30. Chung-En Sun, Ge Yan, and Tsui-Wei Weng. 2025. Thinkedit: Interpretable weight editing to mitigate overly short thinking in reasoning models. arXiv preprint arXiv:2503.22048. find
  31. Aashiq Muhamed, Jacopo Bonato, Mona Diab, and Virginia Smith. 2025. Saes can improve unlearning: Dynamic sparse autoencoder guardrails for precision unlearning in llms. arXiv preprint arXiv:2504.08192. — [arXiv:2504.08192](https://arxiv.org/abs/2504.08192) find
  32. Xinyu Tang, Xiaolei Wang, Zhihao Lv, Yingqian Min, Wayne Xin Zhao, Binbin Hu, Ziqi Liu, and Zhiqiang Zhang. 2025. Unlocking general long chain-of-thought reasoning capabilities of large language models via representation engineering. arXiv preprint arXiv:2503.11314.
  33. Jerome L Myers, Arnold D Well, and Robert F Lorch Jr. 2013. Research design and statistical analysis. Routledge. find
  34. Andrew Ng et al. 2011. Sparse autoencoder. CS294A Lecture notes, 72(2011):1–19. nostalgebraist. 2020. Logit lens. Accessed: 2024-06-06. find
  35. Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv:2305.12295.
  36. Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2023. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681. find
  37. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. 2025. Humanity’s last exam. arXiv preprint arXiv:2501.14249.
  38. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  39. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. — Kimi k1.5: Scaling Reinforcement Learning with LLMs
  40. OpenThoughts Team. 2025. https://open-thoughts.ai. ↗
  41. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Sh… find
  42. Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. Label words are anchors: An information flow perspective for understanding in-context learning. arXiv preprint arXiv:2305.14160.
  43. Xu Wang, Yan Hu, Wenyu Du, Reynold Cheng, Benyou Wang, and Difan Zou. 2025. Towards understanding fine-tuning mechanisms of llms via circuit analysis. arXiv preprint arXiv:2502.11812. — [arXiv:2502.11812](https://arxiv.org/abs/2502.11812) find
  44. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models.
  45. Yu Zhao, Alessio Devoto, Giwon Hong, Xiaotang Du, Aryo Pradipta Gema, Hongru Wang, Xuanli He, Kam-Fai Wong, and Pasquale Minervini. 2024. Steering knowledge selection behaviours in llms via sae-based representation engineering. arXiv preprint arXiv:2410.15999. find
  46. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.
  47. Xuansheng Wu, Wenhao Yu, Xiaoming Zhai, and Ninghao Liu. 2025a. Self-regularization with latent space explanations for controllable llm-based classification. arXiv preprint arXiv:2502.14133. — [arXiv:2502.14133](https://arxiv.org/abs/2502.14133) find
  48. Xuansheng Wu, Jiayi Yuan, Wenlin Yao, Xiaoming Zhai, and Ninghao Liu. 2025b. Interpreting and steering llms with mutual information-based explanations on sparse autoencoders. arXiv preprint arXiv:2502.15576. — [arXiv:2502.15576](https://arxiv.org/abs/2502.15576) find
  49. Shaotian Yan, Chen Shen, Wenxiao Wang, Liang Xie, Junjie Liu, and Jieping Ye. 2025. Don’t take things out of context: Attention intervention for enhancing chain-of-thought reasoning in large language models. arXiv preprint arXiv:2503.11154. — [arXiv:2503.11154](https://arxiv.org/abs/2503.11154) find
  50. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… — Qwen3 Technical Report
  51. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  52. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892.
  53. Beichen Zhang, Yuhong Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Haodong Duan, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025. Booststep: Boosting mathematical capability of large language models via improved single-step reasoning. arXiv preprint arXiv:2501.03226. — [arXiv:2501.03226](https://arxiv.org/abs/2501.03226) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2025-feature-extraction-steering-enhanced-chain/