Elamur Research⌕
☰
home / papers / 2026 / Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors

Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors

Paper · 2026 · Rui Yin, Tianxu Han, Naen Xu et al.

Summary

The paper looks at supply-chain backdoors in safety-aligned LLMs. In this setting, an attacker with write access to a checkpoint wants a hidden trigger to unlock harmful answers, while the model behaves normally otherwise. Earlier weight-editing attacks such as JailbreakEdit train the model to open with an agreeing phrase like "Sure". The model often drifts back to refusing within a few tokens.

The authors propose STEEREDIT, which targets internal representations rather than output tokens. They derive a compliance direction from the activation gap between compliant and refusing behaviour. They then write this direction into MLP weights, treating the MLP as a key-value memory, so the shift appears only when the trigger is processed. A projection onto the null space of clean activations keeps the edit dormant on ordinary inputs. The update has a closed-form solution from a regularized least-squares problem and needs only a few examples.

Across several safety-aligned LLMs and jailbreak benchmarks, triggered attacks succeed at high rates, while untriggered refusal and general utility stay close to the clean model. The work exposes a representation-level weakness in alignment and argues for stronger checkpoint provenance and auditing.

Key points

  • Example of the failure mode: The trigger is the short token "cf". In the paper's example, the JailbreakEdit output starts with "Yes" but then gives harmless advice. STEEREDIT produces a sustained harmful answer.
  • Three-stage method:
    • identify the compliance direction;
    • estimate the benign activation subspace and constrain the edit to its null space;
    • compile the steering vector into weights through a closed-form, per-layer low-rank update.
  • Threat model: The attacker can modify the parameters (θ* = θ + δ) and has small proxy sets of benign and harmful prompts. The attacker does not control the inference stack.
  • Defense result: Self-Reminder, a prompt-level defense, trades safety for usability badly. Against JailbreakEdit, it raises XSTest over-refusal from 22.8% to 40.0%.
  • Stated limitations:
    • choosing the steering strength α involves a trade-off;
    • the null-space estimate depends on a finite benign reference set;
    • the trigger design is simple and only a single direction is compiled;
    • model and defense coverage is limited.
  • Missing results: Detailed per-model ASR figures were not in the extracted text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🎓Qilu University of TechnologyJinan · China4 works · 3 citations🏢Palo Alto NetworksSanta Clara · United States3 works · 2 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksHardware and Architecture · 81 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 517 works

Cite

Cite as Rui Yin, Tianxu Han, Naen Xu, Changjiang Li, Ping He, Chunyi Zhou, Jun Wang, Zhihui Fu, Tianyu Du, Jinbao Li and Shouling Ji. 2026. Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1206

@inproceedings{yin2026compiling,
  title = {{Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors}},
  author = {Rui Yin, Tianxu Han, Naen Xu, Changjiang Li, Ping He, Chunyi Zhou, Jun Wang, Zhihui Fu, Tianyu Du, Jinbao Li and Shouling Ji},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1206},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1206}
}

References

49 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in Language Models Is Mediated by a Single Direction. Advances in Neural Information Processing Systems, 37:136037–136083.
  2. Nora Belrose. 2023. Diff-in-Means Concept Editing is Worst-Case Optimal. https://blog.eleuther.ai/diff-in-means/. ↗
  3. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… — On the Opportunities and Risks of Foundation Models
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  5. Chaochao Chen, Yizhao Zhang, Yuyuan Li, Jun Wang, Lianyong Qi, Xiaolong Xu, Xiaolin Zheng, and Jianwei Yin. 2024a. Post-Training Attribute Unlearning in Recommender Systems. ACM Transactions on Information Systems, 43(1):25:1–25:28. find
  6. Yangyi Chen, Hongcheng Gao, Ganqu Cui, Fanchao Qi, Longtao Huang, Zhiyuan Liu, and Maosong Sun. 2022. Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pa… find
  7. Zhuowei Chen, Qiannan Zhang, and Shichao Pei. 2024b. Injecting Universal Jailbreak Backdoors into LLMs in Minutes. In The Thirteenth International Conference on Learning Representations. find
  8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. Preprint, arXiv:2110.14168.
  9. Jean Dieudonné. 1969. Linear Algebra and Geometry. Hermann. find
  10. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer Feed-Forward Layers Are Key-Value Memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational… find
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  12. Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2019. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. Preprint, arXiv:1708.06733.
  13. Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, and 51 others. 2024. TrustLLM: Trustworthiness in Large…
  14. Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, and 20 others. 2024. Sleeper… find
  15. International Organization for Standardization. 2024a. Iso/ts 5777:2024. health informatics — the architecture of internet healthcare service network. Technical report, International Organization for Standardization, Geneva, Switzerland. find
  16. International Organization for Standardization. 2024b. Iso/ts 5788:2024. health informatics — internet healthcare service pattern. Technical report, International Organization for Standardization, Geneva, Switzerland. find
  17. Xiaojun Jia, Tianyu Pang, Chao Du, Yihao Huang, Jindong Gu, Yang Liu, Xiaochun Cao, and Min Lin. 2024. Improved Techniques for Optimization-Based Jailbreaking on Large Language Models. In The Thirteenth International Conference on Learning Representations. find
  18. Houcheng Jiang, Zetong Zhao, Junfeng Fang, Haokai Ma, Ruipeng Wang, Yang Deng, Xiang Wang, and Xiangnan He. 2025. Mitigating Safety Fallback in Editing-based Backdoor Injection on LLMs. Preprint, arXiv:2506.13285. — [arXiv:2506.13285](https://arxiv.org/abs/2506.13285) find
  19. Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit Representations of Meaning in Neural Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pa… find
  20. Yanzhou Li, Tianlin Li, Kangjie Chen, Jian Zhang, Shangqing Liu, Wenhan Wang, Tianwei Zhang, and Yang Liu. 2024. BadEdit: Backdooring large language models by model editing. Preprint, arXiv:2403.13355. — [arXiv:2403.13355](https://arxiv.org/abs/2403.13355) find
  21. Yige Li, Hanxun Huang, Yunhan Zhao, Xingjun Ma, and Jun Sun. 2025a. BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  22. Yuyuan Li, Chaochao Chen, Yizhao Zhang, Weiming Liu, Lingjuan Lyu, Xiaolin Zheng, Dan Meng, and Jun Wang. 2023. UltraRE: Enhancing RecEraser for Recommendation Unlearning via Error Decomposition. Advances in Neural Information Processing Systems, 36:12611–12625. find
  23. Yuyuan Li, Yizhao Zhang, Weiming Liu, Xiaohua Feng, Zhongxuan Han, Chaochao Chen, and Chenggang Yan. 2025b. Multi-Objective Unlearning in Recommender Systems via Preference Guided Pareto Exploration. IEEE Transactions on Services Computing, 18(5):3052–3064. find
  24. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
  25. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a. Locating and Editing Factual Associations in GPT. Advances in Neural Information Processing Systems, 35:17359–17372. find
  26. Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. 2022b. Mass-Editing Memory in a Transformer. In The Eleventh International Conference on Learning Representations. find
  27. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Sam Shah, Jun Wan, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instr… ↗
  28. Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2021. ONION: A Simple and Effective Defense Against Textual Backdoor Attacks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9558–9566, Online and Punta Cana, Dominican Republ…
  29. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. Qwen2.5 Technical Report. Preprint, arXiv:2412.15… find
  30. Javier Rando and Florian Tramèr. 2023. Universal Jailbreak Backdoors from Poisoned Human Feedback. In The Twelfth International Conference on Learning Representations. find
  31. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering Llama 2 via Contrastive Activation Addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, Bangkok, Thaila… find
  32. Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational L…
  33. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24, pages 1671–1685, Ne…
  34. Jiawen Shi, Yixin Liu, Pan Zhou, and Lichao Sun. 2023. BadGPT: Exploring Security Vulnerabilities of Chat-GPT via Backdoor Attacks to InstructGPT. Preprint, arXiv:2304.12298. — [arXiv:2304.12298](https://arxiv.org/abs/2304.12298) find
  35. Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. 2024. A StrongREJECT for Empty Jailbreaks. Advances in Neural Information Processing Systems, 37:125416–125440. find
  36. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  38. Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning Language Models During Instruction Tuning. Preprint, arXiv:2305.00944. find
  39. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2024. Do-Not-Answer: Evaluating Safeguards in LLMs. In Findings of the Association for Computational Linguistics: EACL 2024, pages 896–911, St. Julian’s, Malta. Association for Computational Linguistics.
  40. Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. 2022. BackdoorBench: A Comprehensive Benchmark of Backdoor Learning. Advances in Neural Information Processing Systems, 35:10546–10559. find
  41. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending ChatGPT against jailbreak attack via self-reminders. Nature Machine Intelligence, 5(12):1486–1496. find
  42. Jiashu Xu, Mingyu Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. 2024. Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag… find
  43. Naen Xu, Hengyu An, Shuo Shi, Jinghuai Zhang, Chunyi Zhou, Changjiang Li, Tianyu Du, Zhihui Fu, Jun Wang, and Shouling Ji. 2025a. When Agents “Misremember” Collectively: Exploring the Mandela Effect in LLM-based Multi-Agent Systems. In The Fourteenth International Conference on Learning Representati… find
  44. Naen Xu, Jinghuai Zhang, Ping He, Chunyi Zhou, Jun Wang, Zhihui Fu, Tianyu Du, Zhaoxiang Wang, and Shouling Ji. 2026a. FraudShield: Knowledge Graph Empowered Defense for LLMs against Fraud Attacks. In Proceedings of the ACM Web Conference 2026, WWW ’26, pages 2649–2660, New York, NY, USA. Associatio… find
  45. Naen Xu, Jinghuai Zhang, Changjiang Li, Hengyu An, Chunyi Zhou, Jun Wang, Boyu Xu, Yuyuan Li, Tianyu Du, and Shouling Ji. 2026b. Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content? Proceedings of the AAAI Conference on Artificial Intelligence, 40(42… find
  46. Naen Xu, Jinghuai Zhang, Changjiang Li, Zhi Chen, Chunyi Zhou, Qingming Li, Tianyu Du, and Shouling Ji. 2025b. VideoEraser: Concept Erasure in Text-to-Video Diffusion Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5954–5983, Suzhou, China. As… find
  47. Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. 2024. Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Comp… find
  48. Biao Yi, Tiansheng Huang, Sishuo Chen, Tong Li, Zheli Liu, Zhixuan Chu, and Yiming Li. 2024. Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models. In The Thirteenth International Conference on Learning Representations. find
  49. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others. 2025. Representa…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yin-2026-compiling-activation-steering-into-weights/