Elamur Research⌕
☰
home / papers / 2026 / Pruning Unsafe Tickets: A Resource-Efficient Framework for Safer and More Robust LLMs

Pruning Unsafe Tickets: A Resource-Efficient Framework for Safer and More Robust LLMs

Paper · 2026 · Wai Man Si, Mingjie Li, Michael Backes et al.

Summary

Open aligned models such as Mistral and LLaVA can still produce harmful content. SFT and RLHF teach a model which answers to prefer, but they do not remove the internal pathways that generate unsafe text, and they need large datasets and heavy compute. The authors use the Lottery Ticket Hypothesis to describe the problem. They argue that sparse "unsafe tickets" survive from pre-training, and that removing them leaves a "safety ticket" subnetwork.

Their post-hoc framework has four stages:

  1. The target model is queried to build its own sets of safe and unsafe responses. These are labeled by an external classifier, clustered and generated at a fixed length.
  2. Parameters are scored with a modified Wanda metric that counts only response-token activations.
  3. Each component gets a contrastive unsafe-versus-safe score, z-normalized within that component.
  4. Components are pruned iteratively, using either greedy or beam search.

The method needs no backpropagation or fine-tuning. It also works with VLMs and quantized models. Unsafe outputs drop to 1–2%, utility loss is small and jailbreak robustness improves. This makes it a practical safety step for deployments on limited hardware.

Key points

  • On Mistral-7B-Instruct-v0.2, unsafe responses fall to 1%. On LLaVA-v1.6-Mistral-7B they fall to 2%. Greedy pruning reaches this in about 455 seconds.
  • After Beam pruning, the share of refusals on the HB dataset climbs from 55.7% to 93.2%. Pruned models still recognize unsafe prompts but shift their output toward refusals.
  • Attribution reinterprets Wanda (|W| times activation norms) and applies a binary mask that drops prompt tokens. The contrastive score is the unsafe attribution divided by the safe attribution, with a small ε added for stability.
  • Across Mistral-7B, Mistral-Nemo and Mistral-Small, pruning touches only two component types: Self-Attn.O and the second MLP layer. In Mistral-Small, 43 of 64 pruned units (67%) came from Self-Attn.O and 21 from MLP2.
  • The stated limitations are that beam search costs more than greedy or one-pass pruning, and that token-level attribution may miss broader contextual cues.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Helmholtz Center for Information SecuritySaarbrücken · Germany14 works · 1,172 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 worksEfficient models & compression · 242 works

Cite

Cite as Wai Man Si, Mingjie Li, Michael Backes and Yang Zhang. 2026. Pruning Unsafe Tickets: A Resource-Efficient Framework for Safer and More Robust LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1209

@inproceedings{si2026pruning,
  title = {{Pruning Unsafe Tickets: A Resource-Efficient Framework for Safer and More Robust LLMs}},
  author = {Wai Man Si, Mingjie Li, Michael Backes and Yang Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1209},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1209}
}

References

54 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Atilla Akkus, Masoud Poorghaffar Aghdam, Mingjie Li, Junjie Chu, Michael Backes, Yang Zhang, and Sinem Sav. 2025. Generated Data with Fake Privacy: Hidden Dangers of Fine-Tuning Large Language Models on Generated Data. In USENIX Security Symposium (USENIX Security), pages 8075– 8093. USENIX. find
  2. Rishabh Bhardwaj, Do Duc Anh, and Soujanya Poria. 2024. Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 414138–14149. ACL.
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  4. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. 2023. Quantifying Memorization Across Neural Language Models. CoRR abs/2202.07646. find
  5. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. 2024. JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models. Co…
  6. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2023. Jailbreaking Black Box Large Language Models in Twenty Queries. CoRR abs/2310.08419.
  7. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. In Annual Conference on Neural Information Processing Systems (NIPS), pages 4299–4307. NIPS. find
  8. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing Chat Language Models by Scaling High-quality Instructional Conversations. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3029–3051. ACL.
  9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 82 others. 2024… — The Llama 3 Herd of Models
  10. Jonathan Frankle and Michael Carbin. 2019. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In International Conference on Learning Representations (ICLR). find
  11. Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. In International Conference on Machine Learning (ICML), pages 10323–10337. PMLR. find
  12. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 30–45. ACL.
  13. Junjie Chu, Yiting Qu, Ye Leng, Michael Backes, Yun Shen, Savvas Zannettou, and Yang Zhang. 2026a. Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Tasks. CoRR abs/2603.11914. — [arXiv:2603.11914](https://arxiv.org/abs/2603.11914) find
  14. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer Feed-Forward Layers Are Key-Value Memories. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5484–5495. ACL. find
  15. Junjie Chu, Xinyue Shen, Ye Leng, Michael Backes, Yun Shen, and Yang Zhang. 2026b. Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks. CoRR abs/2603.04459. — [arXiv:2603.04459](https://arxiv.org/abs/2603.04459) find
  16. Song Han, Huizi Mao, and William J. Dally. 2016. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. In International Conference on Learning Representations (ICLR). find
  17. Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023. Jailbreaker: Automated Jailbreak Across Multiple Large Language Model Chatbots. CoRR abs/2307.08715. — [arXiv:2307.08715](https://arxiv.org/abs/2307.08715) find
  18. Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both Weights and Connections for Efficient Neural Networks. In Annual Conference on Neural Information Processing Systems (NIPS), pages 1135–1143. NIPS. find
  19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171–…
  20. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. 2023. Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation. CoRR abs/2310.06987.
  21. James Diffenderfer and Bhavya Kailkhura. 2021. Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network. In International Conference on Learning Representations (ICLR). find
  22. Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, Sirui Han, Yike Guo, and Yaodong Yang. 2025. PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference. In Annual Meeting of the Association for Com…
  23. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, élio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  24. Yukun Jiang, Mingjie Li, Michael Backes, and Yang Zhang. 2025. Adjacent Words, Divergent Intents: Jailbreaking Large Language Models via Task Concurrency. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS. find
  25. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. 2024. A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity. In International Conference on Machine Learning (ICML), pages 26361–26378. PMLR. find
  26. Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. 2019. Snip: single-Shot Network Pruning based on Connection sensitivity. In International Conference on Learning Representations (ICLR). find
  27. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Annual Meeting of the Association for Computation…
  28. Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2017. Pruning Filters for Efficient ConvNets. In International Conference on Learning Representations (ICLR). find
  29. Kenneth Li, Oam Patel, Fernanda B. Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS.
  30. Mingjie Li, Wai Man Si, Michael Backes, Yang Zhang, and Yisen Wang. 2025. SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation. In International Conference on Learning Representations (ICLR). find
  31. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023b. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. CoRR abs/2310.04451.
  32. Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. 2024. MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models. In European Conference on Computer Vision (ECCV), pages 386–403. Springer. find
  33. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, athaniel Li, Steven Basart, Bo Li, David A. Forsyth, and Dan Hendrycks. 2024. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. CoRR abs/abs/2402.04249.
  34. Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacche… find
  35. Ari S. Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. 2019. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 4933–4943. NeurIPS. find
  36. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.… — Training language models to follow instructions with human feedback
  37. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2025. Safety Alignment Should be Made More Than Just a Few Tokens Deep. In International Conference on Learning Representations (ICLR). find
  38. Mingjie Li, Wai Man Si, Michael Backes, Yang Zhang, and Yisen Wang. 2026. Finding and Reactivating Post-Trained LLMs’ Hidden Safety Mechanisms. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS. find
  39. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 53728–53741. NeurIPS.
  40. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved Baselines with Visual Instruction Tuning. CoRR abs/2310.03744.
  41. Wai Man Si, Mingjie Li, Michael Backes, and Yang Zhang. 2025. Excessive Reasoning Attack on Reasoning LLMs. CoRR abs/2506.14374. — [arXiv:2506.14374](https://arxiv.org/abs/2506.14374) find
  42. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2024. A Simple and Effective Pruning Approach for Large Language Models. In International Conference on Learning Representations (ICLR). find
  43. Aaquib Syed, Can Rager, and Arthur Conmy. 2023. Attribution Patching Outperforms Automated Circuit Discovery. CoRR abs/2310.10348. find
  44. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. LLaMA: Open and Efficient Foundation Language Models. CoRR ab…
  45. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  46. Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. 2023. Activation Addition: Steering Language Models Without Optimization. CoRR abs/2308.10248.
  47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Annual Conference on Neural Information Processing Systems (NIPS), pages 5998–6008. NIPS.
  48. Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. 2024. Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications. In International Conference on Machine Learning (ICML), pages 52588– 52610. PM… find
  49. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs. CoRR abs/2401.06373.
  50. Zhexin Zhang, Junxiao Yang, Pei Ke, Fei Mi, Hongning Wang, and Minlie Huang. 2024. Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 8865–8887. ACL.
  51. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Annual Conference on Neural Information Processing Syst…
  52. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. 2019. Fine-Tuning Language Models from Human Preferences. CoRR abs/1909.08593. find
  53. Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, J. Zico Kolter, Matt Fredrikson, and Dan Hendrycks. 2024. Improving Alignment and Robustness with Circuit Breakers. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS. find
  54. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. CoRR abs/2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/si-2026-pruning-unsafe-tickets-resource-efficient/