Elamur Research⌕
☰
home / papers / 2026 / Towards Identification and Intervention of Safety-Critical Parameters in Large Language Models

Towards Identification and Intervention of Safety-Critical Parameters in Large Language Models

Paper · 2026 · Weiwei Qi, Zefeng Wu, Tianhang Zheng et al.

Summary

LLM safety alignment is hard to apply precisely and tends to erode during later fine-tuning, partly because it is unclear which weights carry safety behaviour. The authors propose the Expected Safety Impact (ESI) framework. It ranks each parameter by the gradient of an expected safety value over harmful prompts, scaled by the parameter's standard deviation. Earlier metrics instead assume uniform or magnitude-proportional changes and rely on cross-entropy loss. Because sampled text is non-differentiable, they score responses with a differentiable judge model. Tokens are relaxed via gumbel-softmax, and a projection matrix links the two vocabularies.

ESI reveals different patterns by architecture. In dense models, safety-critical weights cluster in middle-layer value matrices and MLPs. In MoE models, they sit mostly in late-layer MLP experts. Building on this, the authors propose two methods:

  • Safety Enhancement Tuning (SET) aligns unsafe models by training only the top-ranked weights on safe data.
  • Safety Preserving Adaptation (SPA) freezes those weights during task fine-tuning.

Both offer cheap, targeted ways to add or keep safety without hurting utility.

Key points

  • The metric is ESI(θᵢ) = |σ(θᵢ)∇θᵢS(θ)|. Perturbing just the top 1% of ESI-ranked parameters noticeably degrades model safety.
  • SET cuts attack success rate (ASR) of unaligned models by more than 50% after 100 iterations on 1% of weights. With a 5% update ratio, ASR drops to about 6% on Qwen2.5-14B-base and Llama3-8B-base, while random selection stays above 20%. SET training used the R1-Safety and CB-Safety datasets.
  • SPA tunes only the 10% of parameters with the lowest ESI scores. It was tested on GSM8K, AGNews and MedicalQA with Qwen2.5-7B/14B-it and Llama3-8B-it, against Random and RSN-Tune baselines.
  • On Llama3-8B-it, SPA raised HarmBench ASR by only 0.8% and WildJailbreak ASR by 1.9%. Random selection raised them by 10.4% and 16.3%. Utility stayed comparable, e.g., 90.5% accuracy on AGNews and 78.0% on GSM8K.
  • Code is available at github.com/ZJU-LLM-Safety/SafeWeights-ACL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanyang Technological UniversitySingapore · Singapore156 works · 25,644 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Weiwei Qi, Zefeng Wu, Tianhang Zheng, Zikang Zhang, Xiaojun Jia, Zhan Qin and Kui Ren. 2026. Towards Identification and Intervention of Safety-Critical Parameters in Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1616

@inproceedings{qi2026identification,
  title = {{Towards Identification and Intervention of Safety-Critical Parameters in Large Language Models}},
  author = {Weiwei Qi, Zefeng Wu, Tianhang Zheng, Zikang Zhang, Xiaojun Jia, Zhan Qin and Kui Ren},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1616},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1616}
}

References

3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. 2019. Overview of the mediqa 2019 shared task on textual inference, question entailment and question answering. In Proceedings of the 18th bioNLP workshop and shared task, pages 370–379.
  2. Jianhui Chen, Xiaozhi Wang, Zijun Yao, Yushi Bai, Lei Hou, and Juanzi Li. 2024. Finding safety neurons in large language models. arXiv preprint arXiv:2406.14144. — [arXiv:2406.14144](https://arxiv.org/abs/2406.14144) find
  3. Mark Chen. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. *Figure 4: Impact of parameter selection ratio k% on safety preservation. We compare the ASR of SPA with baselines on Llama3-8B-it and Qwen2.5-14B-it across HarmBench and WildJailbreak.*

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qi-2026-identification-intervention-safety-critical-parameters/