Elamur Research⌕
☰
home / papers / 2025 / Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models

Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models

Paper · 2025 · Lang Gao, Jiahui Geng, Xiangliang Zhang et al.

Summary

The paper asks how jailbreak prompts get past an LLM's safety alignment. Earlier interpretability studies disagreed on which layers matter and on how malicious activations evolve. The authors argue those conflicts come from small samples, often around 100 prompts. They analyze last-token activations for more than 32,000 benign, harmful and jailbreak prompts. These were produced with seven attack methods on models such as Vicuna-7B-v1.3 and Llama-2-7B-chat.

Activations of failed harmful prompts form a compact cluster, which the authors treat as a "safety boundary." Successful jailbreaks move activations outside this region, mostly in the low and middle layers. Based on this, they propose Activation Boundary Defense (ABD). ABD penalizes activations lightly inside the boundary and steeply outside it. Bayesian optimization chooses which layers to apply it to, aiming for high defense success with few layers.

On AdvBench, ABD reaches an average defense success rate (DSR) above 98%. It lowers general ability by under 2%, making it a cheap, training-free defense that preserves utility.

Key points

  • Analysis dataset: 32,507 prompts, roughly 300 times larger than previous analyses.
    • Benign prompts come from Alpaca.
    • Harmful prompts come from five RedEval datasets and AdvBench.
    • Jailbreak prompts come from attacks including GCG-Individual, GCG-Universal, AdvPrompter, AutoDAN, PAIR and DeepInception.
  • Inclusion ratio: this measures the share of jailbreak activations falling inside the harmful activation space. It is below 0.4 without the defense and reaches 1 with ABD.
  • Layer selection: Bayesian optimization independently picks low and middle layers as the most effective places for ABD, matching the analysis.
  • Defense results:
    • Close to 100% DSR on Llama-2-7B-chat for most attacks, and near 100% on Qwen and Vicuna-13B.
    • On the weakly aligned Vicuna-7B, ABD reaches 58% DSR against PAIR and DeepInception. Paraphrase gets 2% and Retokenization 0%.
  • Cost: under 1% extra latency per query and under 2% change in Just-Eval-based general ability. Competing defenses lose up to 37%.
    • Limitations: ABD is weaker on under-aligned models, and only single-turn jailbreaks are studied.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Notre DameNotre Dame · United States51 works · 36,717 citations🎓Huazhong University of Science and TechnologyWuhan · China45 works · 6,345 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Lang Gao, Jiahui Geng, Xiangliang Zhang, Preslav Nakov and Xiuying Chen. 2025. Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1233

@inproceedings{gao2025shaping,
  title = {{Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models}},
  author = {Lang Gao, Jiahui Geng, Xiangliang Zhang, Preslav Nakov and Xiuying Chen},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1233},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1233}
}

References

65 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-Generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2623–2631, Anchorage, AK, USA. ACM.
  2. Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132.
  3. Farzana Anowar, Samira Sadaoui, and Bassant Selim. 2021. Conceptual and empirical comparison of dimensionality reduction algorithms (PCA, KPCA, LDA, MDS, SVD, LLE, Isomap, LE, ICA, t-SNE). Computer Science Review, 40:100378. find
  4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — Qwen Technical Report
  5. Sarah Ball, Frauke Kreuter, and Nina Panickssery. 2024a. Understanding jailbreak success: A study of latent space dynamics in large language models. arXiv preprint arXiv:2406.09289.
  6. Sarah Ball, Frauke Kreuter, and Nina Panickssery. 2024b. Understanding jailbreak success: A study of latent space dynamics in large language models. Preprint, arXiv:2406.09289.
  7. James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc. find
  8. Rishabh Bhardwaj and Soujanya Poria. 2023. Red-teaming large language models using chain of utterances for safety-alignment. Preprint, arXiv:2308.09662. find
  9. J Douglas Carroll and Phipps Arabie. 1998. Multi-dimensional scaling. Measurement, judgment and decision making, pages 179–250. find
  10. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. Preprint, arXiv:2310.08419.
  11. Xiuying Chen, Tairan Wang, Taicheng Guo, Kehan Guo, Juexiao Zhou, Haoyang Li, Zirui Song, Xin Gao, and Xiangliang Zhang. 2025. Unveiling the power of language models in chemical research question answering. Communications Chemistry, 8(1):4.
  12. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  13. Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang. 2024. Comprehensive assessment of jailbreak attacks against llms. arXiv preprint arXiv:2402.05668. — [arXiv:2402.05668](https://arxiv.org/abs/2402.05668) find
  14. Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. 2024. A wolf in sheep‘s clothing: Generalized nested jailbreak prompts can fool large language models easily. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computationa… find
  15. Lang Gao, Kaiyang Wan, Wei Liu, Chenxi Wang, Zirui Song, Zixiang Xu, Yanbo Wang, Veselin Stoyanov, and Xiuying Chen. 2025. Evaluate bias without manual test sets: A concept representation perspective for llms. Preprint, arXiv:2505.15524. — [arXiv:2505.15524](https://arxiv.org/abs/2505.15524) find
  16. Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. 2024. Cold-attack: Jailbreaking llms with stealthiness and controllability. arXiv preprint arXiv:2402.08679. — [arXiv:2402.08679](https://arxiv.org/abs/2402.08679) find
  17. Zeqing He, Zhibo Wang, Zhixuan Chu, Huiyu Xu, Rui Zheng, Kui Ren, and Chun Chen. 2024. Jail-breaklens: Interpreting jailbreak mechanism in the lens of representation and circuit. arXiv:2411.11114. — [arXiv:2411.11114](https://arxiv.org/abs/2411.11114) find
  18. Preprint, Yue Huang, Chujie Gao, Siyuan Wu, Haoran Wang, Xiangqi Wang, Yujun Zhou, Yanbo Wang, Jiayi Ye, Jiawen Shi, Qihui Zhang, et al. 2025. On the trustworthiness of generative foundation models: Guideline, assessment, and perspective. arXiv preprint arXiv:2502.14296. find
  19. Yue Huang, Jingyu Tang, Dongping Chen, Bingda Tang, Yao Wan, Lichao Sun, and Xiangliang Zhang. 2024. Obscureprompt: Jailbreaking large language models via obscure input. arXiv preprint arXiv:2406.13662. — [arXiv:2406.13662](https://arxiv.org/abs/2406.13662) find
  20. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614.
  21. Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. 2024. ArtPrompt: ASCII art-based jailbreak attacks against aligned LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page…
  22. Donald R. Jones. 2001. A taxonomy of global optimization methods based on response surfaces. Journal of Global Optimization, 21(4):345–383. find
  23. Donald R Jones, Matthias Schonlau, and William J Welch. 1998. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13:455–492. find
  24. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169.
  25. Tianlong Li, Xiaoqing Zheng, and Xuanjing Huang. 2024. Rethinking jailbreaking through the lens of representation engineering. ArXiv preprint, abs/2401.06824. — [arXiv:2401.06824](https://arxiv.org/abs/2401.06824) find
  26. Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. 2023. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191.
  27. Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. 2023. The unlocking spell on base llms: Rethinking alignment via in-context learning. ArXiv preprint.
  28. Jianhua Lin. 1991. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145–151. find
  29. Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto Yamada, Hui Liu, and Jiliang Tang. 2024. Towards understanding jailbreak attacks in LLMs: A representation space analysis. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7067–7085, Miami, Florida, USA.… find
  30. Quan Liu, Zhenhong Zhou, Longzhu He, Yi Liu, Wei Zhang, and Sen Su. 2024a. Alignment-enhanced decoding: Defending jailbreaks via token-level adaptive refining of probability distributions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2802–2816, Mia… find
  31. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024b. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Conference on Learning Representations.
  32. Yu Liu, Lang Gao, Mingxin Yang, Yu Xie, Ping Chen, Xiaojin Zhang, and Wei Chen. 2024c. Vuldetect-bench: Evaluating the deep capability of vulnerability detection with large language models. Preprint, arXiv:2406.07595. — [arXiv:2406.07595](https://arxiv.org/abs/2406.07595) find
  33. Yue Liu, Xiaoxin He, Miao Xiong, Jinlan Fu, Shumin Deng, and Bryan Hooi. 2024d. Flipattack: Jailbreak llms via flipping. Preprint, arXiv:2410.02832. — [arXiv:2410.02832](https://arxiv.org/abs/2410.02832) find
  34. MD Abdullah Al Nasim, A. S. M Anas Ferdous, Abdur Rashid, Fatema Tuj Johura Soshi, Parag Biswas, Angona Biswas, and Kishor Datta Gupta. 2025. Trustworthy xai and application. Preprint, arXiv:2410.17139. — [arXiv:2410.17139](https://arxiv.org/abs/2410.17139) find
  35. OpenAI. 2024. Gpt-4o mini: Advancing cost-efficient intelligence. find
  36. Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. 2024. Advprompter: Fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404.16873. — [arXiv:2404.16873](https://arxiv.org/abs/2404.16873) find
  37. Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. 2023. Llm self defense: By self examination, llms know they are being tricked. arXiv preprint arXiv:2308.07308. find
  38. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  39. Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684.
  40. Guobin Shen, Dongcheng Zhao, Yiting Dong, Xiang He, and Yi Zeng. 2024. Jailbreak antidote: Runtime safety-utility balance via sparse representation adjustment in large language models. arXiv preprint arXiv:2410.02298. — [arXiv:2410.02298](https://arxiv.org/abs/2410.02298) find
  41. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2023. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825.
  42. Zirui Song, Bin Yan, Yuhan Liu, Miao Fang, Mingzhe Li, Rui Yan, and Xiuying Chen. 2025. Injecting domain-specific knowledge into large language models: a comprehensive survey. arXiv preprint arXiv:2502.10708. find
  43. Xiongtao Sun, Deyue Zhang, Dongdong Yang, Quanchen Zou, and Hui Li. 2024. Multi-turn context jailbreak attack on large language models from first principles. arXiv preprint arXiv:2408.04686. — [arXiv:2408.04686](https://arxiv.org/abs/2408.04686) find
  44. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  45. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  46. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11). find
  47. Chenxi Wang, Tianle Gu, Zhongyu Wei, Lang Gao, Zirui Song, and Xiuying Chen. 2025a. Word form matters: Llms’ semantic reconstruction under typoglycemia. Preprint, arXiv:2503.01714. find
  48. Chenxi Wang, Zongfang Liu, Dequan Yang, and Xiuying Chen. 2025b. Decoding echo chambers: LLM-powered simulations revealing polarization in social networks. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3913–3923, Abu Dhabi, UAE. Association for Computational…
  49. Yanbo Wang, Jiayi Ye, Siyuan Wu, Chujie Gao, Yue Huang, Xiuying Chen, Yue Zhao, and Xiangliang Zhang. 2025c. Trusteval: A dynamic evaluation toolkit on trustworthiness of generative foundation models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for… find
  50. Yihan Wang, Zhouxing Shi, Andrew Bai, and Cho-Jui Hsieh. 2024. Defending llms against jailbreaking attacks via backtranslation. Preprint, arXiv:2402.16459. find
  51. Yueqi Xie, Minghong Fang, Renjie Pi, and Neil Gong. 2024. GradSafe: Detecting jailbreak prompts for LLMs via safety-critical gradient analysis. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 507–518, Bangkok, Thailand. Associ… find
  52. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5(12):1486–1496. find
  53. Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, and Yuyin Zhou. 2025. Medtrinity-25m: A large-scale multimodal dataset with multi-granular annotations for medicine. In The Thirteenth International Conference on Learning Representatio… find
  54. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. 2024. SafeDecoding: Defending against jailbreak attacks via safety-aware decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 55…
  55. Zixiang Xu, Yanbo Wang, Yue Huang, Xiuying Chen, Jieyu Zhao, Meng Jiang, and Xiangliang Zhang. 2025a. Cross-lingual pitfalls: Automatic probing cross-lingual weakness of multilingual large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (AC… find
  56. Yuqi Zhang, Liang Ding, Lefei Zhang, and Dacheng Tao. 2025. Intention analysis makes llms a good jailbreak defender. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2947–2968.
  57. Chongwen Zhao, Zhihao Dou, and Kaizhu Huang. 2024a. Eeg-defender: Defending against jailbreak through early exit generation of large language models. arXiv preprint arXiv:2408.11308. — [arXiv:2408.11308](https://arxiv.org/abs/2408.11308) find
  58. Wei Zhao, Zhe Li, Yige Li, Ye Zhang, and Jun Sun. 2024b. Defending large language models against jailbreak attacks via layer-specific editing. arXiv preprint arXiv:2405.18166. find
  59. Yujun Zhou, Yufei Han, Haomin Zhuang, Kehan Guo, Zhenwen Liang, Hongyan Bao, and Xiangliang Zhang. 2024a. Defending jailbreak prompts via in-context adversarial game. arXiv preprint arXiv:2402.13148. find
  60. Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. 2024b. How alignment and jailbreak work: Explain llm safety through intermediate hidden states. arXiv preprint arXiv:2406.05644. find
  61. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. Preprint, arXiv:2307.15043.
  62. Zixiang Xu, Yanbo Wang, Yue Huang, Jiayi Ye, Haomin Zhuang, Zirui Song, Lang Gao, Chenxi Wang, Zhaorun Chen, Yujun Zhou, Sixian Li, Wang Pan, Yue Zhao, Jieyu Zhao, Xiangliang Zhang, and Xiuying Chen. 2025b. Socialmaze: A benchmark for evaluating social reasoning in large language models. find
  63. Zixiang Xu, Yanbo Wang, Chenxi Wang, Lang Gao, Zirui Song, Yue Huang, Zhaorun Chen, Xiangliang Zhang, and Xiuying Chen. 2025c. Gta: Graph theory agent and benchmark for algorithmic graph reasoning with llms. find
  64. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736.
  65. Zhiyuan Yu, Xiaogeng Liu, Shunning Liang, Zach Cameron, Chaowei Xiao, and Ning Zhang. 2024. Don’t listen to me: Understanding and exploring jailbreak prompts of large language models. In 33rd USENIX Security Symposium (USENIX Security 24), Philadelphia, PA. USENIX Association. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gao-2025-shaping-safety-boundaries-understanding-defending/