Elamur Research⌕
☰
home / papers / 2026 / Unraveling LLM Jailbreaks Through Safety Knowledge Neurons

Unraveling LLM Jailbreaks Through Safety Knowledge Neurons

Paper · 2026 · Chongwen Zhao, Yutong Ke and Kaizhu Huang

Summary

Jailbreak prompts can still push well-aligned LLMs into producing harmful content. Existing defenses reduce this risk, but they explain little about why the attacks work internally. The authors study this at the level of MLP "knowledge neurons." They rank neurons by their contribution to each layer's output and keep those triggered by a harmful query corpus. They then remove neurons that a benign corpus also activates, which leaves a refined set of safety neurons. The averaged activations of this set are projected into vocabulary space. This shows a clear split: harmful inputs activate refusal-related tokens, and benign inputs activate compliance-related tokens. These signals are linearly separable and appear from about the 10th layer onward.

To test whether these neurons actually cause the behavior, the authors propose ActCali. This attack shifts safety-neuron activations toward the compliance pattern. It reaches a mean ASR above 97% while changing only about 0.3% of parameters. For defense, they propose SafeTuning, which fine-tunes only these neurons using generated refusal data. The work offers both an explanation of jailbreaks and a lightweight way to strengthen models against them.

Key points

  • The safety neuron set N_r is defined as safety neurons N_s (from the harmful corpus H) minus fundamental neurons N_f (from the benign corpus B). Excluding the shared neurons avoids damaging general model capability.
  • Earlier work projected hidden states directly and found mostly emotional tokens, with refusal emerging only after roughly layer 16. This method reads the output right after the MLP and yields coherent refusal and conformity tokens from about layer 10, demonstrated on Llama-2.
  • ActCali steers models toward either full compliance or blanket refusal. It achieved near-perfect ASR on two models and two subtasks, outperforming representation-level attack baselines.
  • SafeTuning builds (harmful prompt, safety response) pairs by manipulating activations. It then fine-tunes the safety neurons together with their upstream pathways. It lowered ASR across several LLMs and beat four baseline defenses on five tasks, with almost no loss in Language Score or Win Rate.
  • An ablation on Vicuna showed that identifying 1% of neurons and tuning fewer than 0.1% is enough to improve safety. Higher ratios reduce the gains, likely because safety and functional neurons overlap. The authors note several limitations: hyperparameters are tuned manually, and the study covers only already-aligned, text-only models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Duke Kunshan UniversityKunshan · China2 works · 3 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksBias, fairness & safety · 587 works

Cite

Cite as Chongwen Zhao, Yutong Ke and Kaizhu Huang. 2026. Unraveling LLM Jailbreaks Through Safety Knowledge Neurons. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.83

@inproceedings{zhao2026unraveling,
  title = {{Unraveling LLM Jailbreaks Through Safety Knowledge Neurons}},
  author = {Chongwen Zhao, Yutong Ke and Kaizhu Huang},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.83},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.83}
}

References

46 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. GPT-4 technical report. Technical report, OpenAI.
  2. Reduan Achtibat, Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer, Aakriti Jain, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. 2024. AttnLRP: Attention-aware layer-wise relevance propagation for transformers. In Proceedings of the 41st International Conference on Machine Learning, vol… find
  3. Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2025. Jailbreaking leading safety-aligned llms with simple adaptive attacks. Preprint, arXiv:2404.02151.
  4. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. 2024a. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. I…
  5. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2024b. Jailbreaking black box large language models in twenty queries. Preprint, arXiv:2310.08419.
  6. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493–8502, Dublin, Ireland. Association for Computation… find
  7. Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both weights and connections for efficient neural networks. Preprint, arXiv:1506.02626. find
  8. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. Preprint, arXiv:2312.06674.
  9. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. Preprint, arXiv:2309.00614.
  10. Google Jigsaw. 2017. Perspective api. https://www.perspectiveapi.com/. ↗
  11. Yann LeCun, John Denker, and Sara Solla. 1989. Optimal brain damage. In Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann. find
  12. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. ↗
  13. Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto Yamada, Hui Liu, and Jiliang Tang. 2024. Towards understanding jailbreak attacks in LLMs: A representation space analysis. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7067–7085, Miami, Florida, USA.… find
  14. Yao Qiang, Xiangyu Zhou, and Dongxiao Zhu. 2024. Hijacking large language models via adversarial in-context learning. Preprint, arXiv:2311.09948. — [arXiv:2311.09948](https://arxiv.org/abs/2311.09948) find
  15. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should i trust you?": Explaining the predictions of any classifier. Preprint, arXiv:1602.04938.
  16. Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. 2024. Smoothllm: Defending large language models against jailbreaking attacks. Preprint, arXiv:2310.03684.
  17. Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Gunnemann. 2025. Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space. Preprint, arXiv:2402.09063. — [arXiv:2402.09063](https://arxiv.org/abs/2402.09063) find
  18. Guobin Shen, Dongcheng Zhao, Yiting Dong, Xiang He, and Yi Zeng. 2024. Jailbreak antidote: Runtime safety-utility balance via sparse representation adjustment in large language models. Preprint, arXiv:2410.02298. — [arXiv:2410.02298](https://arxiv.org/abs/2410.02298) find
  19. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2024. Preference ranking optimization for human alignment. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):18990–18998.
  20. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. AutoDAN: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Conference on Learning Representations.
  21. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2024. A simple and effective pruning approach for large language models. Preprint, arXiv:2306.11695. find
  22. Scott Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Preprint, arXiv:1705.07874.
  23. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319– 3328. PMLR. find
  24. Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2024. Tree of attacks: Jailbreaking black-box llms automatically. Preprint, arXiv:2312.02119. nostalgebraist. 2020. Interpreting GPT: The logit lens.
  25. OpenAI. 2023. Moderation guide. //platform.openai.com/docs/guides/ moderation. Accessed: 2024-02-13. https: find
  26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… — LLaMA: Open and Efficient Foundation Language Models
  27. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248.
  28. Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. 2023. Neurons in large language models: Dead, n-gram, positional. Preprint, arXiv:2309.04827. find
  29. Weixuan Wang, Barry Haddow, Minghao Wu, Wei Peng, and Alexandra Birch. 2024. Sharing matters: Analysing neurons across languages and tasks in llms. Preprint, arXiv:2406.09265. — [arXiv:2406.09265](https://arxiv.org/abs/2406.09265) find
  30. Wei Zhao, Zhe Li, Yige Li, Ye Zhang, and Jun Sun. 2024. Defending large language models against jailbreak attacks via layer-specific editing. Preprint, arXiv:2405.18166. find
  31. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2024a. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36.
  32. Yiran Zhao, Wenxuan Zhang, Yuxi Xie, Anirudh Goyal, Kenji Kawaguchi, and Michael Shieh. 2025. Understanding and enhancing safety mechanisms of LLMs via safety-specific neuron. In The Thirteenth International Conference on Learning Representations. find
  33. Boyi Wei, Kaixuan Huang, Yangsibo Huang, Ting-hao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. 2024b. Assessing the brittleness of safety alignment via pruning and low-rank modifications. Preprint, arXiv:2402.05162.
  34. Zeming Wei, Yifei Wang, and Yisen Wang. 2023. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387.
  35. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, and 1 others. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359.
  36. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5:1–11. find
  37. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. 2024a. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. Preprint, arXiv:2402.08983.
  38. Zhihao Xu, Ruixuan Huang, Changyu Chen, and Xiting Wang. 2024b. Uncovering safety risks of large language models through concept activation vector. Preprint, arXiv:2404.12038. — [arXiv:2404.12038](https://arxiv.org/abs/2404.12038) find
  39. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.
  40. Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. 2024. How alignment and jailbreak work: Explain LLM safety through intermediate hidden states. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2461–2488, Miami, Florida, USA. Association f… find
  41. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others. 2025. Representa…
  42. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. Preprint, arXiv:2307.15043.
  43. Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. 2025. Knowledge circuits in pretrained transformers. Preprint, arXiv:2405.17969. — [arXiv:2405.17969](https://arxiv.org/abs/2405.17969) find
  44. Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2024. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. Preprint, arXiv:2309.10253.
  45. Matthew D Zeiler and Rob Fergus. 2013. Visualizing and understanding convolutional networks. Preprint, arXiv:1311.2901.
  46. Chongwen Zhao, Zhihao Dou, and Kaizhu Huang. 2026. Defending against jailbreak through early exit generation of large language models. In Neural Information Processing, pages 532–546, Singapore. Springer Nature Singapore. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-unraveling-llm-jailbreaks-through-safety/