Elamur Research⌕
☰
home / papers / 2026 / Preventing Safety Drift in Large Language Models via Coupled Weight and Activation Constraints

Preventing Safety Drift in Large Language Models via Coupled Weight and Activation Constraints

Paper · 2026 · Songping Peng, Zhiheng Zhang, Daojian Zeng et al.

Summary

Aligned LLMs often lose their refusal behavior after fine-tuning, even when the training data is benign. Most existing defenses act at only one level: they either restrict weight updates or regularize internal activations. The authors linearize a layer's output change around the aligned model and show that it splits into a weight-driven term and an activation-driven term. Bounding only one term leaves the other free, which opens a bypass route to unsafe outputs.

Their method, CWAC (Coupled Weight and Activation Constraints), handles both terms during fine-tuning:

  • Weight constraint: updates are projected onto a safety subspace, computed in advance by applying SVD to covariance matrices of harmful-prompt inputs.
  • Activation constraint: safety-critical features are located with a sparse autoencoder whose training data combines harmful and benign examples. A regularizer then keeps those features close to their original values on held-out harmful prompts.

Across four open instruction-tuned models (7B–9B) and several downstream tasks, CWAC produced the lowest harmful scores among the compared methods while keeping task accuracy. This matters for teams that customize open LLMs and need safety to survive adaptation.

Key points

  • With a harmful data ratio of p=0.05, the harmful score was 72.70 for SFT, 38.50 for ASFT and 10.50 for CWAC. At p=0.5, CWAC rose to 30.78.
  • Averaged over harmful ratios, CWAC scored 18.73 versus 38.22 for ASFT. Its average accuracy was 94.35%, compared with 94.07% for plain SFT.
  • In the learning-rate sweep (1e-7 to 1e-3), CWAC degraded far less than the baselines. At 1e-4, its harmful score went from 35 to 68, while ASFT climbed from 47 to 82 and SFT from 53 to 80.
  • In ablations, the Weight-only and Activation-only variants each beat most baselines, including ASFT and SPPFT. The full coupled CWAC was best across all models and epochs.
  • Safety was evaluated with JailbreakBench, HarmBench, AdvBench and BeaverTails. The method needs white-box access to the model and a well-trained SAE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🔬Institute of AutomationBeijing · China70 works · 14,251 citations🎓National University of Defense TechnologyChangsha · China10 works · 931 citations

Also on the byline: Hunan Normal University.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Songping Peng, Zhiheng Zhang, Daojian Zeng, Lincheng Jiang and Xieping Gao. 2026. Preventing Safety Drift in Large Language Models via Coupled Weight and Activation Constraints. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.874

@inproceedings{peng2026preventing,
  title = {{Preventing Safety Drift in Large Language Models via Coupled Weight and Activation Constraints}},
  author = {Songping Peng, Zhiheng Zhang, Daojian Zeng, Lincheng Jiang and Xieping Gao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.874},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.874}
}

References

53 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2024. Safety-tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions. In The Twelfth International Conference on Learning Representations.
  3. Zouying Cao, Yifei Yang, and Hai Zhao. 2025. Scans: Mitigating the exaggerated safety for llms via safety-conscious activation steering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23523–23531. find
  4. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. Pr…
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  6. Aladin Djuhera, Swanand Ravindra Kadhe, Farhan Ahmed, Syed Zawad, and Holger Boche. 2025. Safemerge: Preserving safety alignment in fine-tuned large language models via selective layer-wise model merging. arXiv preprint arXiv:2503.17239. find
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. corr, abs/2407.21783, 2024. doi: 10.48550. arXiv preprint ARXIV.2407.21783, 10.
  8. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. Preprint, arXiv:2101.00027. find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Ruben Härle, Felix Friedrich, Manuel Brack, Björn Deiseroth, Patrick Schramowski, and Kristian Kersting. 2024. Scar: Sparse conditioned autoencoders for concept detection and steering in llms. arXiv preprint arXiv:2411.07122. — [arXiv:2411.07122](https://arxiv.org/abs/2411.07122) find
  11. Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. 2024. Safe lora: The silver lining of reducing safety risks when fine-tuning large language models. Advances in Neural Information Processing Systems, 37:65072–65094. find
  12. Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi, Josh Kimball, and Ling Liu. 2024a. Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning. arXiv preprint arXiv:2408.09600. — [arXiv:2408.09600](https://arxiv.org/abs/2408.09600) find
  13. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Tekin, and Ling Liu. 2024b. Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack. Advances in Neural Information Processing Systems, 37:104521–104555. find
  14. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Harmful fine-tuning attacks and defenses for large language models: A survey, 2024. URL https://arxiv.org/abs/2409.18169. — [arXiv:2409.18169](https://arxiv.org/abs/2409.18169) ↗
  15. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. 2024c. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. arXiv preprint arXiv:2409.01586. — [arXiv:2409.01586](https://arxiv.org/abs/2409.01586) find
  16. Tiansheng Huang, Sihao Hu, and Ling Liu. 2024d. Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack. Advances in Neural Information Processing Systems, 37:74058–74088. find
  17. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and…
  18. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  19. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146.
  20. Jaehan Kim, Minkyoo Song, Seungwon Shin, and Sooel Son. 2025. Defending moe llms against harmful fine-tuning via safety routing alignment. Preprint, arXiv:2509.22745. — [arXiv:2509.22745](https://arxiv.org/abs/2509.22745) find
  21. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: Eliciting truthful answers from a language model. In Thirty-seventh Conference on Neural Information Processing Systems.
  22. Mingjie Li, Wai Man Si, Michael Backes, Yang Zhang, and Yisen Wang. 2025a. Salora: Safety-alignment preserved low-rank adaptation. arXiv preprint arXiv:2501.01765. — [arXiv:2501.01765](https://arxiv.org/abs/2501.01765) find
  23. Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. 2024. Safety layers in aligned large language models: The key to llm security. arXiv preprint arXiv:2408.17003. — [arXiv:2408.17003](https://arxiv.org/abs/2408.17003) find
  24. Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. 2025b. Safety layers in aligned large language models: The key to LLM security. In The Thirteenth International Conference on Learning Representations. find
  25. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  26. Minrui Luo, Fuhang Kuang, Yu Wang, Zirui Liu, and Tianxing He. 2025. Sc-lora: Balancing efficient fine-tuning and knowledge preservation via subspace-constrained lora. arXiv preprint arXiv:2505.23724. — [arXiv:2505.23724](https://arxiv.org/abs/2505.23724) find
  27. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, and 1 others. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249.
  28. Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania. 2023. Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint arXiv:2308.13320. — [arXiv:2308.13320](https://arxiv.org/abs/2308.13320) find
  29. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  30. Gabriel J Perin, Runjin Chen, Xuxi Chen, Nina ST Hirata, Zhangyang Wang, and Junyuan Hong. 2025. Lox: Low-rank extrapolation robustifies llm safety against fine-tuning. arXiv preprint arXiv:2506.15606. — [arXiv:2506.15606](https://arxiv.org/abs/2506.15606) find
  31. Kaustubh Ponkshe, Shaan Shah, Raghav Singhal, and Praneeth Vepakomma. 2025a. Safety subspaces are not distinct: A fine-tuning case study. arXiv preprint arXiv:2505.14185. — [arXiv:2505.14185](https://arxiv.org/abs/2505.14185) find
  32. Kaustubh Ponkshe, Shaan Shah, Raghav Singhal, and Praneeth Vepakomma. 2025b. Safety subspaces are not linearly distinct: A fine-tuning case study. Preprint, arXiv:2505.14185. — [arXiv:2505.14185](https://arxiv.org/abs/2505.14185) find
  33. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations.
  34. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm.stanford. edu/2023/03/13/alpaca. html, 3(6):… ↗
  35. Gemma Team, T Mesnard, C Hardin, R Dadashi, S Bhupatiraju, S Pathak, L Sifre, M Rivière, MS Kale, J Love, and 1 others. Gemma: Open models based on gemini research and technology. arxiv 2024. arXiv preprint arXiv:2403.08295. find
  36. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  37. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476.
  38. Jiong Wang, Jiazhao Li, Yiquan Li, Xiangyu Qi, Junjie Hu, Sharon Li, Patrick Drew McDaniel, Muhao Chen, Bo Li, and Chaowei Xiao. 2024. Backdooralign: Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment. Advances in Neural Information Processing Systems 37. find
  39. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741.
  40. Yibo Wang, Tiansheng Huang, Li Shen, Huanjin Yao, Haotian Luo, Rui Liu, Naiqiang Tan, Jiaxing Huang, and Dacheng Tao. 2025. Panacea: Mitigating harmful fine-tuning for large language models via post-fine-tuning perturbation. CoRR, abs/2501.18100. — [arXiv:2501.18100](https://arxiv.org/abs/2501.18100) find
  41. Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Robie Gonzales, Subhabrata Majumdar, Hassan Sajjad, Frank Rudzicz, and 1 others. 2024. Representation noising: A defence mechanism against harmful finetuning. Advances in Neural Information Processing Systems, 37:12636–12676. find
  42. Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. 2024. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint arXiv:2402.05162. find
  43. William F. Shen, Xinchi Qiu, Nicola Cancedda, and Nicholas D. Lane. 2025. Don’t make it up: Preserving ignorance awareness in llm fine-tuning. Preprint, arXiv:2506.14387. — [arXiv:2506.14387](https://arxiv.org/abs/2506.14387) find
  44. Shuo Yang, Qihui Zhang, Yuyang Liu, Yue Huang, Xiaojun Jia, Kunpeng Ning, Jiayu Yao, Jigang Wang, Hailiang Dai, Yibing Song, and 1 others. 2025a. Asft: Anchoring safety during llm fine-tuning within narrow safety basin. arXiv preprint arXiv:2506.08473. — [arXiv:2506.08473](https://arxiv.org/abs/2506.08473) find
  45. Leheng Sheng, Changshuo Shen, Weixiang Zhao, Junfeng Fang, Xiaohao Liu, Zhenkai Liang, Xiang Wang, An Zhang, and Tat-Seng Chua. 2025. Alphasteer: Learning refusal steering with principled null-space constraint. Preprint, arXiv:2506.07022. — [arXiv:2506.07022](https://arxiv.org/abs/2506.07022) find
  46. Wei Shi, Sihang Li, Tao Liang, Mingyang Wan, Guojun Ma, Xiang Wang, and Xiangnan He. 2025. Route sparse autoencoder to interpret large language models. Preprint, arXiv:2503.08200.
  47. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–…
  48. Shuo Yang, Qihui Zhang, Yuyang Liu, Yue Huang, Xiaojun Jia, Kunpeng Ning, Jiayu Yao, Jigang Wang, Hailiang Dai, Yibing Song, and 1 others. 2025b. Asft: Anchoring safety during llm fine-tuning within narrow safety basin. arXiv preprint arXiv:2506.08473. — [arXiv:2506.08473](https://arxiv.org/abs/2506.08473) find
  49. Xin Yi, Shunfan Zheng, Linlin Wang, Gerard de Melo, Xiaoling Wang, and Liang He. 2025. Nlsr: Neuron-level safety realignment of large language models against harmful fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25706–25714. find
  50. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2016. Character-level convolutional networks for text classification. Preprint, arXiv:1509.01626.
  51. Yi Zhou, Wenpeng Xing, Dezhang Kong, Changting Lin, and Meng Han. 2025. Neurel-attack: Neuron relearning for safety disalignment in large language models. arXiv preprint arXiv:2504.21053. — [arXiv:2504.21053](https://arxiv.org/abs/2504.21053) find
  52. Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, J Zico Kolter, Matt Fredrikson, and Dan Hendrycks. 2024. Improving alignment and robustness with circuit breakers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  53. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2026-preventing-safety-drift-large-language/