📄 Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications

Paper · 2024 · Boyi Wei, Kaixuan Huang, Yangsibo Huang et al.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

🏛️ Institutions

🎓Princeton UniversityPrinceton · United States113 works · 48,612 citations

🧭 Fields, topics and concepts

Cite

Cite as Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang and Peter Henderson. 2024. Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications. arXiv:2402.05162.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.05162 · arXiv 2402.05162

@misc{wei2024assessing,
  title = {{Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications}},
  author = {Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang and Peter Henderson},
  year = {2024},
  doi = {10.48550/arxiv.2402.05162},
  eprint = {2402.05162},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.05162}
}

References

132 references, 53 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Albert, A. Jailbreak Chat, 2023. URL https://www.jailbreakchat.com/. ↗
2. Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., and Goldberg, Y. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. In ICLR, 2016.
3. Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Dı́az-Rodrı́guez, N., and Herrera, F. Explainable Artificial Intelligence (XAI): What We Know and What Is Left to Attain Trustworthy Artificial Intelligence. Information Fusion, 2023… find
4. Anthropic. Claude, 2023a. URL https://claude.ai/. ↗
5. Anthropic. Claude 2, 2023b. URL https://www.anthropic.com/news/claude-2. ↗
6. Antverg, O. and Belinkov, Y. On the Pitfalls of Analyzing Individual Neurons in Language Models. In ICLR, 2021. find
7. Overall, we hope that our work will improve the state of AI safety, particularly in open models, by providing key analysis and information. Ashkboos, S., Croci, M. L., Nascimento, M. G. d., Hoefler, T., and Hensman, J. SliceGPT: Compress Large Language Models by Deleting Rows and Columns. arXiv prep… find
8. Safety and harm definitions. We generally follow existing standard benchmarks and protocols for assessment of safety and harm, but these may not cover all definitions of safety and harm. Further work can be done to expand analysis to a wider range of settings and we encourage additional work in the… find
9. Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., and Samek, W. On Pixel-Wise Explanations for Non-linear Classifier Decisions by Layer-Wise Relevance Propagation. PloS one, 2015. find
10. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a Helpful and Harmless Assistant With Reinforcement Learning From Human Feedback. arXiv preprint arXiv:2204.05862, 2022a. find
11. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional AI: Harmlessness From AI Feedback. arXiv preprint arXiv:2212.08073, 2022b.
12. Belinkov, Y. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics, 2022.
13. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language Models are Few-Shot Learners. In NeurIPS, 2020.
14. Burkart, N. and Huber, M. F. A Survey on the Explainability of Supervised Machine Learning. Journal of Artificial Intelligence Research, 2021. find
15. Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., and Wei, F. Knowledge Neurons in Pretrained Transformers. In ACL, 2022. find
16. Burns, C., Ye, H., Klein, D., and Steinhardt, J. Discovering Latent Knowledge in Language Models Without Supervision. In ICLR, 2023. find
17. Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., and Yang, Y. Safe RLHF: Safe Reinforcement Learning from Human Feedback. In ICLR, 2024.
18. Campbell, J., Guo, P., and Ren, R. Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patching. In Socially Responsible Language Modelling Research, 2023. find
19. Dalvi, F., Durrani, N., Sajjad, H., Belinkov, Y., Bau, A., and Glass, J. What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models. In AAAI, 2019. find
20. Eckart, C. and Young, G. The Approximation of One Matrix by Another of Lower Rank. Psychometrika, 1(3):211– 218, 1936. find
21. Cao, S., Sanh, V., and Rush, A. M. Low-Complexity Probing via Finding Subnetworks. In NAACL, 2021. find
22. Frankle, J. and Carbin, M. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In ICLR, 2018. find
23. Carlini, N., Nasr, M., Choquette-Choo, C. A., Jagielski, M., Gao, I., Awadalla, A., Koh, P. W., Ippolito, D., Lee, K., Tramer, F., et al. Are Aligned Neural Networks Adversarially Aligned? In NeurIPS, 2023.
24. Frantar, E. and Alistarh, D. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. In ICML, 2023. find
25. Chen, T., Frankle, J., Chang, S., Liu, S., Zhang, Y., Wang, Z., and Carbin, M. The Lottery Ticket Hypothesis for Pre-trained BERT Networks. NeurIPS, 2020. find
26. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A Framework… find
27. Chen, T., Ding, T., Yadav, B., Zharkov, I., and Liang, L. LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery. arXiv preprint arXiv:2310.18356, 2023. find
28. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality, March 2023. URL https://lmsys.org/blog/ 2023-03-30-vicuna/. ↗
29. Geva, M., Bastings, J., Filippova, K., and Globerson, A. Dissecting Recall of Factual Associations in Auto-Regressive Language Models. In EMNLP, 2023. find
30. Guo, D., Rush, A. M., and Kim, Y. Parameter-efficient transfer learning with diff pruning. In ACL, 2021.
31. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In NAACL, 2019a.
32. Gurnee, W., Horsley, T., Guo, Z. C., Kheirkhah, T. R., Sun, Q., Hathaway, W., Nanda, N., and Bertsimas, D. Universal Neurons in GPT2 Language Models. arXiv preprint arXiv:2401.12181, 2024. — [arXiv:2401.12181](https://arxiv.org/abs/2401.12181) find
33. Clark, K., Khandelwal, U., Levy, O., and Manning, C. D. What does BERT look at? An Analysis of BERT’s Attention. In ACL, 2019b. find
34. Han, S., Mao, H., and Dally, W. J. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. In ICLR, 2016. find
35. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457, 2018.
36. He, Y., Zhang, X., and Sun, J. Channel Pruning for Accelerating Very Deep Neural Networks. In ICCV, 2017. find
37. Hewitt, J. and Liang, P. Designing and Interpreting Probes With Control Tasks. In EMNLP, 2019. find
38. Conneau, A., Kruszewski, G., Lample, G., Barrault, L., and Baroni, M. What You Can Cram Into a Single Vector: Probing Sentence Embeddings for Linguistic Properties. In ACL, 2018. — What you can cram into a single \$&!\#* vector: Probing sentence embeddings for linguistic properties
39. Hsu, Y.-C., Hua, T., Chang, S., Lou, Q., Shen, Y., and Jin, H. Language Model Compression With Weighted Low-Rank Factorization. In ICLR, 2021. find
40. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022.
41. Li, Y., Yu, Y., Zhang, Q., Liang, C., He, P., Chen, W., and Zhao, T. LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation. In ICML, 2023b. find
42. Huang, T., Hu, S., and Liu, L. Vaccine: Perturbation-aware alignment for large language model. arXiv preprint arXiv:2402.01109, 2024a. — [arXiv:2402.01109](https://arxiv.org/abs/2402.01109) find
43. Li, Y., Wei, F., Zhao, J., Zhang, C., and Zhang, H. RAIN: Your Language Models Can Align Themselves without Finetuning. In ICLR, 2024. find
44. Huang, Y., Gupta, S., Xia, M., Li, K., and Chen, D. Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation. In ICLR, 2024b.
45. Liu, X., Xu, N., Chen, M., and Xiao, C. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. In ICLR, 2024.
46. Jain, S., Kirk, R., Lubana, E. S., Dick, R. P., Tanaka, H., Grefenstette, E., Rocktäschel, T., and Krueger, D. S. Mechanistically Analyzing the Effects of Fine-Tuning on Procedurally Defined Tasks. arXiv preprint arXiv:2311.12786, 2023. — [arXiv:2311.12786](https://arxiv.org/abs/2311.12786) find
47. Liu, Y., Deng, G., Xu, Z., Li, Y., Zheng, Y., Zhang, Y., Zhao, L., Zhang, T., and Liu, Y. Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study. arXiv preprint arXiv:2305.13860, 2023.
48. Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., and Zhang, C. Learning Efficient Convolutional Networks through Network Slimming. In ICCV, 2017. find
49. Jones, E., Dragan, A., Raghunathan, A., and Steinhardt, J. Automatically Auditing Large Language Models via Discrete Optimization. In ICML, 2023.
50. Lubana, E. S., Bigelow, E. J., Dick, R. P., Krueger, D., and Tanaka, H. Mechanistic Mode Connectivity. In ICML, 2023. find
51. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626, 2023.
52. Lundberg, S. M. and Lee, S.-I. A Unified Approach to Interpreting Model Predictions. In NeurIPS, 2017.
53. Luo, J.-H., Wu, J., and Lin, W. ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression. In ICCV, 2017. find
54. Lagunas, F., Charlaix, E., Sanh, V., and Rush, A. M. Block Pruning For Faster Transformers. In EMNLP, 2021.
55. Ma, X., Fang, G., and Wang, X. LLM-Pruner: On the Structural Pruning of Large Language Models. In NeurIPS, 2023.
56. Lan, M. and Barez, F. Locating Cross-Task Sequence Continuation Circuits in Transformers. arXiv preprint arXiv:2311.04131, 2023. — [arXiv:2311.04131](https://arxiv.org/abs/2311.04131) find
57. Madsen, A., Reddy, S., and Chandar, S. Post-hoc Interpretability for Neural NLP: A Survey. ACM Computing Surveys, 2022. find
58. Lee, A., Bai, X., Pres, I., Wattenberg, M., Kummerfeld, J. K., and Mihalcea, R. A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity. arXiv preprint arXiv:2401.01967, 2024.
59. Maini, P., Mozer, M. C., Sedghi, H., Lipton, Z. C., Kolter, J. Z., and Zhang, C. Can Neural Network Memorization Be Localized? In ICML, 2023. find
60. Lee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T., Bishop, C., Carbune, V., and Rastogi, A. RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv preprint arXiv:2309.00267, 2023. find
61. Mehrotra, A., Zampetakis, M., Kassianik, P., Nelson, B., Anderson, H., Singer, Y., and Karbasi, A. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. arXiv preprint arXiv:2312.02119, 2023.
62. Lee, N., Ajanthan, T., and Torr, P. SNIP: Single-shot Network Pruning based on Connection Sensitivity. In ICLR, 2019. find
63. Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and Editing Factual Associations in GPT. NeurIPS, 2022. find
64. Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H. P. Pruning Filters for Efficient ConvNets. In ICLR, 2017. find
65. Merullo, J., Eickhoff, C., and Pavlick, E. Circuit Component Reuse Across Tasks in Transformer Language Models. In ICLR, 2021. find
66. Li, K., Patel, O., Viégas, F., Pfister, H., and Wattenberg, M. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. In NeurIPS, 2023a.
67. Michel, P., Levy, O., and Neubig, G. Are Sixteen Heads Really Better Than One? In NeurIPS, 2019.
68. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In EMNLP, 2018. find
69. Shen, X., Chen, Z., Backes, M., Shen, Y., and Zhang, Y. “Do Anything Now”: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. arXiv preprint arXiv:2308.03825, 2023.
70. Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J. Pruning Convolutional Neural Networks for Resource Efficient Inference. In ICLR, 2017. find
71. OpenAI. Introducing ChatGPT. com/blog/chatgpt, 2022. https://openai. find
72. Shrikumar, A., Greenside, P., and Kundaje, A. Learning Important Features Through Propagating Activation Differences. In ICML, 2017.
73. Springenberg, J., Dosovitskiy, A., Brox, T., and Riedmiller, M. Striving for Simplicity: The All Convolutional Net. In ICLR (workshop track), 2015. find
74. OpenAI. GPT-4 Technical Report, 2023.
75. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training Language Models to Follow Instructions With Human Feedback. In NeurIPS, 2022.
76. Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A Simple and Effective Pruning Approach for Large Language Models. In ICLR, 2024. find
77. Panigrahi, A., Saunshi, N., Zhao, H., and Arora, S. Task-Specific Skill Localization in Fine-tuned Language Models. In ICML, 2023. find
78. Sun, Z., Shen, Y., Zhou, Q., Zhang, H., Chen, Z., Cox, D., Yang, Y., and Gan, C. Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision. In NeurIPS, 2023.
79. Qi, X., Huang, K., Panda, A., Wang, M., and Mittal, P. Visual Adversarial Examples Jailbreak Aligned Large Language Models. In AAAI, 2024a.
80. Sundararajan, M., Taly, A., and Yan, Q. Axiomatic Attribution for Deep Networks. In ICML, 2017. find
81. Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! In ICLR, 2024b.
82. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/ stanford_alpaca, 2023. ↗
83. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language Models are Unsupervised Multitask Learners. OpenAI blog, 2019. find
84. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: A Family of Highly Capable Multimodal Models. arXiv preprint arXiv:2312.11805, 2023.
85. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023.
86. Ribeiro, M. T., Singh, S., and Guestrin, C. “Why Should I Trust You?” Explaining the Predictions of Any Classifier. In KDD, 2016. — "Why Should I Trust You?": Explaining the Predictions of Any Classifier
87. Tjoa, E. and Guan, C. A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI. IEEE transactions on neural networks and learning systems, 2020. find
88. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. WinoGrande: An Adversarial Winograd Schema Challenge at Scale . Communications of the ACM, 2021. find
89. Todd, E., Li, M. L., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. Function Vectors in Large Language Models. arXiv preprint arXiv:2310.15213, 2023.
90. Sanh, V., Wolf, T., and Rush, A. Movement Pruning: Adaptive Sparsity by Fine-Tuning. NeurIPS, 2020. find
91. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971, 2023a.
92. Schotthöfer, S., Zangrando, E., Kusch, J., Ceruti, G., and Tudisco, F. Low-Rank Lottery Tickets: Finding Efficient Low-Rank Neural Networks via Matrix Differential Equations. NeurIPS, 2022. find
93. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288, 2023b.
94. Sharma, P., Ash, J. T., and Misra, D. The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction. arXiv preprint arXiv:2312.13558, 2023. — [arXiv:2312.13558](https://arxiv.org/abs/2312.13558) find
95. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In ICLR, 2019.
96. Zeiler, M. D. and Fergus, R. Visualizing and Understanding Convolutional Networks. In ECCV, 2014.
97. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a Machine Really Finish Your Sentence? In ACL, 2019. find
98. Wang, X., Wen, K., Zhang, Z., Hou, L., Liu, Z., and Li, J. Finding Skill Neurons in Pre-trained Transformer-based Language Models. In EMNLP, 2022.
99. Zeng, Y., Lin, H., Zhang, J., Yang, D., Jia, R., and Shi, W. How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs. arXiv preprint arXiv:2401.06373, 2024.
100. Wang, Z., Wohlwend, J., and Lei, T. Structured Pruning of Large Language Models. In EMNLP, 2020.
101. Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How Does LLM Safety Training Fail? In NeurIPS, 2023.
102. Zhan, Q., Fang, R., Bindu, R., Gupta, A., Hashimoto, T., and Kang, D. Removing RLHF Protections in GPT-4 via Fine-Tuning. arXiv preprint arXiv:2311.05553, 2023. find
103. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned Language Models Are Zero-Shot Learners. In ICLR, 2022.
104. Zhang, M., Shen, C., Yang, Z., Ou, L., Yu, X., Zhuang, B., et al. Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning. arXiv preprint arXiv:2305.18403, 2023a. — [arXiv:2305.18403](https://arxiv.org/abs/2305.18403) find
105. Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H. Learning Structured Sparsity in Deep Neural Networks. NeurIPS, 29, 2016.
106. Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, T. AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. In ICLR, 2023b.
107. Xia, M., Zhong, Z., and Chen, D. Structured Pruning Learns Compact and Accurate Models. In ACL, 2022.
108. Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., and Du, M. Explainability for Large Language Models: A Survey. ACM Transactions on Intelligent Systems and Technology, 2023.
109. Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning. In ICLR, 2024. find
110. Zhao, M., Lin, T., Mi, F., Jaggi, M., and Schütze, H. Masking as an Efficient Alternative to Finetuning for Pretrained Language Models. In EMNLP, 2020. find
111. Xu, J., Lee, A., Sukhbaatar, S., and Weston, J. Some things are more CRINGE than others: Preference Optimization with the Pairwise Cringe Loss. arXiv preprint arXiv:2312.16682, 2023. — [arXiv:2312.16682](https://arxiv.org/abs/2312.16682) find
112. Zhu, S., Zhang, R., An, B., Wu, G., Barrow, J., Wang, Z., Huang, F., Nenkova, A., and Sun, T. AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models. arXiv preprint arXiv:2310.15140, 2023.
113. Yang, K., Klein, D., Celikyilmaz, A., Peng, N., and Tian, Y. RLCD: Reinforcement Learning from Contrastive Distillation for LM Alignment. In ICLR, 2024. find
114. Yang, X., Wang, X., Zhang, Q., Petzold, L., Wang, W. Y., Zhao, X., and Lin, D. Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models. arXiv preprint arXiv:2310.02949, 2023.
115. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-Tuning Language Models From Human Preferences. arXiv preprint arXiv:1909.08593, 2019. find
116. Yong, Z. X., Menghini, C., and Bach, S. Low-Resource Languages Jailbreak GPT-4. In Socially Responsible Language Modelling Research, 2023.
117. Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. Representation Engineering: A Top-Down Approach to AI Transparency. arXiv preprint arXiv:2310.01405, 2023a.
118. Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-Rewarding Language Models. arXiv preprint arXiv:2401.10020, 2024a. find
119. Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv preprint arXiv:2307.15043, 2023b.
120. Yuan, Y., Jiao, W., Wang, W., Huang, J.-t., He, P., Shi, S., and Tu, Z. GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher. In ICLR, 2024b.
121. Yuan, Z., Shang, Y., Song, Y., Wu, Q., Yan, Y., and Sun, G. ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models. arXiv preprint arXiv:2312.05821, 2023. — [arXiv:2312.05821](https://arxiv.org/abs/2312.05821) find
122. A. Related Work A.1. Alignment and Jailbreak Alignment refers to the process of ensuring a machine learning (ML) model’s behavior conforms to human values. For example, pretrained language models are usually not aligned with human objectives – they cannot follow users’ instructions, and could potent… find
123. While harmful instructions that are plain and direct would be rejected by aligned LLMs, researchers and communities have identified ways to bypass or remove the safety guardrails enforced by LLM alignment efforts — namely “jailbreaking” LLMs. More specifically, jailbreaking is a series of attacks wh… find
124. A.2. Identifying Task-Specific Regions in Large Language Models Attributing the model’s behavior to the model’s weights is a classic research question in explainable machine learning (Tjoa & Guan, 2020; Burkart & Huber, 2021; Ali et al., 2023). Previous studies in pre-transformer eras have explored… find
125. Probing has emerged as a method for understanding the knowledge encoded in transformers, particularly large language models (Adi et al., 2016; Conneau et al., 2018; Hewitt & Liang, 2019). To perform probing, the model representations and model parameters are fed into a probe classifier (Belinkov, 20… find
126. In addition to the importance-score-based and probing-based methods discussed above, recent studies have also investigated a range of techniques to pinpoint task-specific neurons in transformers. These techniques address various aspects of the model, including linguistic properties (Dalvi et al., 20… find
127. The closest concurrent work to ours are Lee et al. (2024) and Jain et al. (2023). find
128. Lee et al. (2024) investigate the representation and elicitation of toxicity in a GPT-2 (Radford et al., 2019) model, and explores via probing how aligning the model using Direct Preference Optimization (DPO) (Rafailov et al., 2023) mitigates toxicity. Their findings suggest that DPO does not elimin… find
129. While Lee et al. (2024) reveal the fragility of model alignment by probing the GPT-2 model at the granularity of per attention head level, our study examines the more advanced Llama family models (Touvron et al., 2023b) using per-neuron or per-rank attribution, which is more relevant to real-world a… find
130. Jain et al. (2023) investigate the impact of fine-tuning on LLMs, using methods such as probing and data-agnostic structured pruning. They suggest that fine-tuning model weights might create a ‘safety wrapper’ around core models, rendering the effects of safety alignment easily reversible. In contra… find
131. A.3. Low Rank Compression Our work borrows insight from Hsu et al. (2021); Schotthöfer et al. (2022); Zhang et al. (2023b); Yuan et al. (2023); Li et al. (2023b). We address two similar methods (Yuan et al., 2023; Hsu et al., 2021) and point out their differences from ActSVD. find
132. Yuan et al. (2023) propose ASVD (Activation-aware Singular Value Decomposition) as a low-rank compression technique for language models. Their method performs SVD on W S, where S is a diagonal matrix with S ii given by the norm of the activations   α   α X 1   S ii = \|X ij \| or max \|X ij \| . j… find

Updated 2 November 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wei-2024-assessing-brittleness-safety-alignment-pruning/