Elamur Research⌕
☰
home / papers / 2026 / Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics

Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics

Paper · 2026 · Ziwen Xu, Chenyan WU, Hengyu Sun et al.

Summary

Methods for controlling LLM behaviour are usually studied separately, with different assumptions and evaluation setups. These include local weight fine-tuning, LoRA, and activation steering. The authors show that all three can be written as one affine update of a linear layer, where a multiplier scales a change to the weights and/or bias. LoRA changes only the weights through low-rank factors, steering vectors change only the bias, and local weight tuning changes both. On this basis they define two quantities on a common log-odds scale, computed from paired positive and negative completions. Preference is the pull toward a target concept, and utility is coherent, task-valid output. Across all three methods, raising the multiplier increases preference but reduces utility in a regular, fittable way. They explain this with an activation-manifold hypothesis: interventions move representations along concept directions, but utility drops when activations are pushed off the region where valid generation happens. Guided by this, they propose SPLIT, a training objective that optimizes preference while protecting utility. SPLIT outperforms SFT, REPS and DiffMean baselines, which makes steering more predictable and less damaging.

Key points

  • The unified formula is h_{i+1} = (W + m₁ΔW)h_i + (b + m₂Δb). Parameter cost is about d_in×d_out + d_out for local weight tuning, d_in×r + r×d_out for LoRA, and only d_out for a steering vector.
  • Closed-form relations between the multiplier m and both preference and utility log-odds fit the observed curves with high R².
  • Evaluation uses three benchmarks:
    • Psychopathy, scored by accuracy.
    • PowerSeeking, scored by an LLM judge on a 0–4 scale.
    • AxBench, scored by concept score and a harmonic mean of concept, instruction and fluency.
  • In one reported setting with vector-form steering on Psychopathy, SPLIT reaches 98% accuracy. This compares with 97% for SFT, 88% for REPS, 55% for DiffMean and 50% for the unsteered model.
  • Code is released in EasyEdit (zjunlp/EasyEdit, examples/SPLIT.md). The authors note the experiments cover attribute-level control and fixed multipliers only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China168 works · 15,386 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Ziwen Xu, Chenyan WU, Hengyu Sun, Haiwen Hong, Mengru Wang, Yunzhi Yao, Longtao Huang, Hui Xue, Shumin Deng, Zhixuan Chu, Huajun Chen and Ningyu Zhang. 2026. Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1463

@inproceedings{xu2026why,
  title = {{Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics}},
  author = {Ziwen Xu, Chenyan WU, Hengyu Sun, Haiwen Hong, Mengru Wang, Yunzhi Yao, Longtao Huang, Hui Xue, Shumin Deng, Zhixuan Chu, Huajun Chen and Ningyu Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1463},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1463}
}

References

54 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lukasz Bartoszcze, Sarthak Munshi, Bryan Sukidi, Jennifer Yen, Zejia Yang, David Williams-King, Linh Le, Kosi Asuzu, and Carsten Maple. 2025. Representation engineering for large-language models: Survey and research challenges. CoRR, abs/2502.17601. — [arXiv:2502.17601](https://arxiv.org/abs/2502.17601) find
  2. Amrita Bhattacharjee, Shaona Ghosh, Traian Rebedea, and Christopher Parisien. 2024. Towards inference-time category-wise safety steering for large language models. CoRR, abs/2410.01174. — [arXiv:2410.01174](https://arxiv.org/abs/2410.01174) find
  3. Eric J. Bigelow, Daniel Wurgaft, YingQiao Wang, Noah D. Goodman, Tomer D. Ullman, Hidenori Tanaka, and Ekdeep Singh Lubana. 2025. Belief dynamics reveal the dual nature of in-context learning and activation steering. CoRR, abs/2511.00617. — [arXiv:2511.00617](https://arxiv.org/abs/2511.00617) find
  4. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah… find
  5. Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. 2024. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. In Advances in Neural Information Processing Systems 38: Annual Conference on Neu… find
  6. Huajun Chen. 2024. Large knowledge model: Perspectives and challenges. DATA INTELLIGENCE, 6(3):587–620. find
  7. Songlin Chen, Weicheng Wang, Xiaoliang Chen, Peng Lu, Zaiyan Yang, and Yajun Du. 2024. Llama-lora neural prompt engineering: A deep tuning framework for automatically generating chinese text logical reasoning thinking chains. DATA INTELLIGENCE, 6(2):375–408. find
  8. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. 2022. Delta tuning: A comprehensive study of param… — [arXiv:2203.06904](https://arxiv.org/abs/2203.06904) find
  9. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature machine intelligence, 5(3):220–235.
  10. Houcheng Jiang, Junfeng Fang, Ningyu Zhang, Guojun Ma, Mingyang Wan, Xiang Wang, Xiangnan He, and Tat-seng Chua. 2025. Anyedit: Edit any knowledge encoded in language models. arXiv preprint arXiv:2502.05628. — [arXiv:2502.05628](https://arxiv.org/abs/2502.05628) find
  11. Constanza Fierro and Fabien Roger. 2025. Steering language models with weight arithmetic. CoRR, abs/2511.05408. find
  12. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 548… find
  13. Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji. 2024. Word embeddings are steers for language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16410–16430.
  14. Peixuan Han, Cheng Qian, Xiusi Chen, Yuji Zhang, Denghui Zhang, and Heng Ji. 2025. Internal activation as the polar star for steering unsafe llm behavior. arXiv preprint arXiv:2502.01042. — [arXiv:2502.01042](https://arxiv.org/abs/2502.01042) find
  15. Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024. Lora+: Efficient low rank adaptation of large models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. find
  16. Rima Hazra, Sayan Layek, Somnath Banerjee, and Soujanya Poria. 2024. Safety arithmetic: A framework for test-time safety alignment of language models by steering parameters and activations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami,… find
  17. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022a. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  18. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022b. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  19. Yi Hu, Jiaqi Gu, Ruxin Wang, Zijun Yao, Hao Peng, Xiaobao Wu, Jianhui Chen, Muhan Zhang, and Liangming Pan. 2026. Towards a mechanistic understanding of large reasoning models: A survey Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias… find
  20. Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. 2024. Vera: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
  21. Tianhong Li and Kaiming He. 2025. Back to basics: Let denoising generative models denoise. arXiv preprint https://arxiv.org/pdf/2511.13720. ↗
  22. Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. 2025. A survey on lora of large language models. Frontiers of Computer Science, 19(7):197605. find
  23. Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. CoRR, abs/2310.06824.
  24. Erik Miehling, Michael Desmond, Karthikeyan Natesan Ramamurthy, Elizabeth M. Daly, Kush R. Varshney, Eitan Farchi, Pierre Dognin, Jesus Rios, Djallel Bouneffouf, Miao Liu, and Prasanna Sattigeri. 2025. Evaluating the prompt steerability of large language models. In Proceedings of the 2025 Conference… find
  25. Tomás Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 9-14, 2013, Westin Peachtree Plaza Hotel…
  26. Alexander Modell, Patrick Rubin-Delanchy, and Nick Whiteley. 2025. The origins of representation manifolds in large language models. CoRR, abs/2505.18235. — [arXiv:2505.18235](https://arxiv.org/abs/2505.18235) find
  27. Neel Nanda, Andrew Lee, and Martin Wattenberg. 2023. Emergent linear representations in world models of self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2023, Singapore, December 7, 2023, pages 16–3… find
  28. Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.
  29. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group…
  30. Joris Postmus and Steven Abreu. 2024. Steering large language models using conceptors: Improving addition-based activation engineering. CoRR, abs/2410.16314. — [arXiv:2410.16314](https://arxiv.org/abs/2410.16314) find
  31. Nate Rahn, Pierluca D’Oro, and Marc G. Bellemare. 2024. Controlling large language model agents with entropic activation steering. CoRR, abs/2406.00244. — [arXiv:2406.00244](https://arxiv.org/abs/2406.00244) find
  32. Carl Edward Rasmussen. 2004. Gaussian Processes in Machine Learning, pages 63–71. Springer Berlin Heidelberg, Berlin, Heidelberg. find
  33. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2024. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand,… find
  34. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Activation addition: Steering language models without optimization. arXiv eprints, pages arXiv–2308. find
  35. Teun van der Weij, Massimo Poesio, and Nandi Schoots. 2024. Extending activation steering to broad skills and multiple behaviours. CoRR, abs/2403.05767. — [arXiv:2403.05767](https://arxiv.org/abs/2403.05767) find
  36. Mengru Wang, Ziwen Xu, Shengyu Mao, Shumin Deng, Zhaopeng Tu, Huajun Chen, and Ningyu Zhang. 2025. Beyond prompt engineering: Robust behavior control in llms via steering target atoms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),…
  37. Mengru Wang, Yunzhi Yao, Ziwen Xu, Shuofei Qiao, Shumin Deng, Peng Wang, Xiang Chen, Jia-Chen Gu, Yong Jiang, Pengjun Xie, et al. 2024. Knowledge mechanisms in large language models: A survey and perspective. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7097–7135. find
  38. Tom Wollschläger, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan Günnemann, and Johannes Gasteiger. 2025. The geometry of refusal in large language models: Concept cones and representational independence. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver… find
  39. Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2025a. Axbench: Steering llms? even simple baselines outperform sparse autoencoders. arXiv preprint arXiv:2501.17148. find
  40. Daniel Scalena, Gabriele Sarti, and Malvina Nissim. 2024. Multi-property steering of large language models with dynamic activation composition. CoRR, abs/2406.17563. find
  41. Zhengxuan Wu, Qinan Yu, Aryaman Arora, Christopher D. Manning, and Christopher Potts. 2025b. Improved representation steering for language models. CoRR, abs/2505.20809. find
  42. Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. 2025. A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models. CoRR, abs/2503.05613. find
  43. Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Jiang Chang, Liang Zhao, Shangyan Zhou, Zhean Xu, Zhengyan Zhang, Wangding Zeng, Shengding Hu, Yuqing Wang, Jingyang Yuan, Lean Wang, and Wenfeng Liang. 2025. mhc: Manifold-constrained hyper-connectio… — [arXiv:2512.24880](https://arxiv.org/abs/2512.24880) find
  44. Daniel Tan, David Chanin, Aengus Lynch, Dimitrios Kanoulas, Brooks Paige, Adrià Garriga-Alonso, and Robert Kirk. 2024. Analyzing the generalization and reliability of steering vectors. CoRR, abs/2407.12404. — [arXiv:2407.12404](https://arxiv.org/abs/2407.12404) find
  45. Curt Tigges, Curt Tigges, Oskar Hollinsworth, Curt Tigges, Atticus Geiger, Atticus Geiger, Oskar Hollinsworth, Neel Nanda, Neel Nanda, Atticus Geiger, and Neel Nanda. 2023. Linear representations of sentiment in large language models. http://arxiv.org/abs/2310.15154. ↗
  46. Ziwen Xu, Shuxun Wang, Kewei Xu, Haoming Xu, Mengru Wang, Xinle Deng, Yunzhi Yao, Guozhou Zheng, Huajun Chen, and Ningyu Zhang. 2025. Easyedit2: An easy-to-use steering framework for editing large language models. CoRR, abs/2504.15133. find
  47. Wanli Yang, Fei Sun, Rui Tang, Hongyu Zang, Du Su, Qi Cao, Jingang Wang, Huawei Shen, and Xueqi Cheng. 2025. Fine-tuning done right in model editing. CoRR, abs/2509.22072. — [arXiv:2509.22072](https://arxiv.org/abs/2509.22072) find
  48. Yunzhi Yao, Jiaxin Qin, Ningyu Zhang, Haoming Xu, Yuqi Zhu, Zeping Yu, Mengru Wang, Yuqi Tang, Jia-Chen Gu, Shumin Deng, et al. 2025. Rethinking knowledge editing in reasoning era. Authorea Preprints. find
  49. Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 2…
  50. Hengyuan Zhang, Zhihao Zhang, Mingyang Wang, Zun-hai Su, Yiwei Wang, Qianli Wang, Shuzhou Yuan, Ercong Nie, Xufeng Duan, Qibo Xue, et al. 2026. Locate, steer, and improve: A practical survey of actionable mechanistic interpretability in large language models. arXiv preprint arXiv:2601.14004.
  51. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. Open-Review.net. find
  52. Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. 2024. Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… find
  53. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2).
  54. Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. 2023. Delta-lora: Fine-tuning high-rank parameters with the delta of low-rank matrices. CoRR, abs/2309.02411. — [arXiv:2309.02411](https://arxiv.org/abs/2309.02411) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-why-steering-works-unified-view/