Elamur Research⌕
☰
home / papers / 2026 / Knowledge Vector of Logical Reasoning in Large Language Models

Knowledge Vector of Logical Reasoning in Large Language Models

Paper · 2026 · Zixuan Wang and Yuanyuan Lei

Summary

The paper asks whether the three classic forms of logical reasoning (deductive, inductive and abductive) correspond to linear directions in an LLM's activation space, and how these directions relate to each other. For each type, the authors build contrastive pairs: a strong prompt that produces a correct answer and a weak prompt that produces a wrong one. They average residual-stream activations over the generated tokens and train a linear probe, whose weights serve as the reasoning vector. Steering the model along these vectors improves results, so the linear assumption holds. However, the three vectors are nearly orthogonal, which conflicts with cognitive-science accounts in which these reasoning forms interact. The authors therefore propose a complementary subspace-constrained refinement. A complementary loss lets each vector draw on knowledge from the other two, and a subspace constraint loss keeps each vector inside its own SAE-derived subspace. Refined vectors beat both the unsteered model and single-vector steering across models. Interpretability analyses show which features are shared across reasoning types and which are specific to one.

Key points

  • Data and metrics: JustLogic (deductive), DEER (inductive) and ART (abductive). Deductive and abductive tasks are scored by accuracy, and inductive by METEOR.
  • Models: Llama-3.1-8B-it and Gemma-2-9B-it, steered at layer 13, plus GPT-OSS-20B (a Mixture-of-Experts model) to test a larger, different architecture.
  • Pairwise cosine similarities between the naively extracted vectors are close to zero, meaning the reasoning types occupy largely separate directions. Refinement builds each type's subspace from SAE features via QR decomposition.
  • Example gains for Llama-3.1-8B-it under greedy decoding (unsteered → mono steering → complementary steering):
    • deductive accuracy: 48.95 → 55.22 → 56.46;
    • inductive METEOR: 26.36 → 27.13 → 27.55.
  • Mechanistic findings:
    • After refinement, deductive and inductive features move closer together, while abductive features become more specialized.
    • Attention patching shows that core heads are preserved, activations become more concentrated, and new heads emerge (e.g., layer 17 head 24 for deductive reasoning).
    • Token-level span analysis links deduction to connectives like "therefore", induction to quantifiers, and abduction to words like "likely".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of FloridaGainesville · United States33 works · 364 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Zixuan Wang and Yuanyuan Lei. 2026. Knowledge Vector of Logical Reasoning in Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2021

@inproceedings{wang2026knowledge,
  title = {{Knowledge Vector of Logical Reasoning in Large Language Models}},
  author = {Zixuan Wang and Yuanyuan Lei},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2021},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2021}
}

References

60 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samir Abdaljalil, Hasan Kurban, Khalid Qaraqe, and Erchin Serpedin. 2025. Theorem-of-thought: A multi-agent framework for abductive, deductive, and inductive reasoning in language models. arXiv preprint arXiv:2506.07106. find
  2. Kingma DP Ba J Adam and 1 others. 2014. A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 1412(6).
  3. Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644. find
  4. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  5. Suhana Bedi, Yixing Jiang, Philip Chung, Sanmi Koyejo, and Nigam Shah. 2025. Fidelity of medical reasoning in large language models. JAMA Network Open, 8(8):e2526021–e2526021. find
  6. Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219.
  7. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2019. Abductive commonsense reasoning. arXiv preprint arXiv:1908.05739.
  8. Chengkun Cai, Xu Zhao, Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, and Lei Li. 2025. The role of deductive and inductive reasoning in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers…
  9. Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2025. Separating tongue from thought: Activation patching reveals language-agnostic concept representations in transformers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics…
  10. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, and 1 others. 2022. Toy models of superposition. arXiv preprint arXiv:2209.10652. find
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  12. Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. 2023. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. In Thirty-seventh Conference on Neural Information Processing Systems. find
  13. Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji. 2024. Word embeddings are steers for language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16410–16430.
  14. Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, and 1 others. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. arXiv preprint arXiv:2410.20526. find
  15. Stefan Heimersheim and Neel Nanda. 2024. How to use and interpret activation patching. arXiv preprint arXiv:2404.15255. — [arXiv:2404.15255](https://arxiv.org/abs/2404.15255) find
  16. Michael K Chen, Xikun Zhang, and Dacheng Tao. 2025. Justlogic: A comprehensive benchmark for evaluating deductive reasoning in large language models. arXiv preprint arXiv:2501.14851.
  17. Evan Heit and Caren M Rotello. 2010. Relations between inductive reasoning and deductive reasoning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(3):805. find
  18. Kewei Cheng, Jingfeng Yang, Haoming Jiang, Zhengyang Wang, Binxuan Huang, Ruirui Li, Shiyang Li, Zheng Li, Yifan Gao, Xian Li, Bing Yin, and Yizhou Sun. 2024. Inductive or deductive? rethinking the fundamental reasoning abilities of llms. Preprint, arXiv:2408.00114. find
  19. Keith J Holyoak and Robert G Morrison. 2013. The Oxford handbook of thinking and reasoning. Oxford University Press. find
  20. Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2023. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318–16352. find
  21. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600. find
  22. Zhenglin Hua, Jinghan He, Zijun Yao, Tianxu Han, Haiyun Guo, Yuheng Jia, and Junfeng Fang. 2025. Steering lvlms via sparse autoencoder for hallucination mitigation. arXiv preprint arXiv:2505.16146. find
  23. Philip N Johnson-Laird. 2010. Mental models and human reasoning. Proceedings of the National Academy of Sciences, 107(43):18243–18250. find
  24. Ole Jorgensen, Dylan Cope, Nandi Schoots, and Murray Shanahan. 2023. Improving activation steering in language models with mean-centring. arXiv preprint arXiv:2312.03813. — [arXiv:2312.03813](https://arxiv.org/abs/2312.03813) find
  25. Jiabao Kang, Xinye Li, Liyan Xu, Qingbin Liu, Xi Chen, Zhiying Tu, Dianhui Chu, and Dianbo Sui. 2025. Exploring deductive and inductive reasoning capabilities of large language models in procedural planning. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 16320–16341. find
  26. Connor Kissane, robertzk, Arthur Conmy, and Neel Nanda. 2024. SAEs (usually) Transfer Between Base and Chat Models. https://www.lesswrong.com/posts/fmwk6qxrpW8d4jvbd/saes-usually-transfer-between-base-and-chat-models. ↗
  27. Long Hei Matthew Lam, Ramya Keerthy Thatikonda, and Ehsan Shareghi. 2024. A closer look at tool-based logical reasoning with llms: The choice of tool matters. In Proceedings of the 22nd Annual Workshop of the Australasian Language Technology Association, pages 41–63. find
  28. Chunyang Li, Weiqi Wang, Tianshi Zheng, and Yangqiu Song. 2025. Patterns over principles: The fragility of inductive reasoning in llms under noisy observations. arXiv preprint arXiv:2502.16169. find
  29. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147. find
  30. Johnny Lin and Joseph Bloom. 2023. Neuronpedia: Interactive reference and tooling for analyzing neural networks. Software available from neuronpedia. org. find
  31. Samuel Marks, Adam Karvonen, and Aaron Mueller. 2024. Dictionary learning. https://github.com/ saprmarks/dictionary_learning. GitHub repository. ↗
  32. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372. find
  33. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022b. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229. find
  34. Aashiq Muhamed, Jacopo Bonato, Mona T Diab, and Virginia Smith. 2025. Saes can improve unlearning: Dynamic sparse autoencoder guardrails for precision unlearning in llms. In ICML 2025 Workshop on Reliable and Responsible Foundation Models. find
  35. Neel Nanda. 2023. Actually, othello-gpt has a linear emergent world representation. https://www.neelnanda.io/mechanistic-interpretability/othello. ↗
  36. Neel Nanda and Joseph Bloom. 2022. Transformerlens. https://github.com/TransformerLensOrg/ TransformerLens. ↗
  37. Duy Nguyen, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. 2025. Multi-attribute steering of language models via targeted intervention. arXiv preprint arXiv:2502.12446.
  38. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925.
  39. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  40. Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. 2023. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3806–3824.
  41. Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658.
  42. Charles Sanders Peirce. 1934. Collected papers of charles sanders peirce, volume 5. Harvard University Press. find
  43. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522. find
  44. Ted Sanders and Ninne Stukker. 2012. Causal connectives in discourse: A cross-linguistic perspective. Journal of pragmatics, 44(2):131–137. find
  45. Daniel Scalena, Gabriele Sarti, and Malvina Nissim. 2024. Multi-property steering of large language models with dynamic activation composition. arXiv preprint arXiv:2406.17563. find
  46. Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. 2025. A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models. arXiv preprint arXiv:2503.05613. find
  47. Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. 2024. Improving instruction-following in language models through activation steering. arXiv preprint arXiv:2410.12877. — [arXiv:2410.12877](https://arxiv.org/abs/2410.12877) find
  48. Daniel Tan, David Chanin, Aengus Lynch, Brooks Paige, Dimitrios Kanoulas, Adrià Garriga-Alonso, and Robert Kirk. 2024. Analysing the generalisation and reliability of steering vectors. Advances in Neural Information Processing Systems, 37:139179– 139212. find
  49. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  50. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, and 1 others. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.
  51. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248.
  52. A Aniket Vashishtha, Qirun Dai, Hongyuan Mei, Amit Sharma, Chenhao Tan, and Hao Peng. 2025. Executable counterfactuals: Improving llms’ causal reasoning through code. arXiv preprint arXiv:2510.01539. find
  53. Constantin Venhoff, Iván Arcuschin, Philip Torr, Arthur Conmy, and Neel Nanda. 2025. Understanding reasoning in thinking language models via steering vectors. arXiv preprint arXiv:2506.18167. — [arXiv:2506.18167](https://arxiv.org/abs/2506.18167) find
  54. Danqing Wang, Jianxin Ma, Fei Fang, and Lei Li. 2024. Typedthinker: Diversify large language model reasoning with typed thinking. arXiv preprint arXiv:2410.01952. — [arXiv:2410.01952](https://arxiv.org/abs/2410.01952) find
  55. Tianlong Wang, Xianfeng Jiao, Yinghao Zhu, Zhongzhi Chen, Yifan He, Xu Chu, Junyi Gao, Yasha Wang, and Liantao Ma. 2025a. Adaptive activation steering: A tuning-free llm truthfulness improvement method for diverse hallucinations categories. In Proceedings of the ACM on Web Conference 2025, pages 256… find
  56. Xu Wang, Zihao Li, Benyou Wang, Yan Hu, and Difan Zou. 2025b. Model unlearning via sparse autoencoder subspace guided projections. arXiv preprint arXiv:2505.24428.
  57. Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. 2024. Language models as inductive reasoners. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 209–225.
  58. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations.
  59. Fred Zhang and Neel Nanda. 2023. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXiv:2309.16042. find
  60. Jason Zhang and Scott W Viteri. Uncovering latent chain of thought vectors in large language models. In Workshop on Neural Network Weights as a New Data Modality. Datasets find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-knowledge-vector-logical-reasoning-large/