Elamur Research⌕
☰
home / papers / 2024 / Improving Alignment and Robustness with Circuit Breakers

Improving Alignment and Robustness with Circuit Breakers

Paper · 2024 · Andy Zou, Long Phan, Justin Wang et al.

Summary

Language models trained to refuse harmful requests can still be jailbroken. Adversarial training usually protects only against attacks seen during training, and it reduces capability. Zou et al. propose "circuit breakers," which change the model's internal representations instead of patching individual attack routes. When the model starts moving toward harmful content, its generation is derailed. The core technique, Representation Rerouting (RR), trains LoRA adapters with two objectives. On a harmful "circuit breaker" dataset, it pushes hidden states away from those of the frozen original model, toward orthogonality. On a retain dataset, it keeps hidden states close to the original. Because RR targets the harmful representation itself rather than a specific attack, it works regardless of the attack and adds no inference cost. On HarmBench prompts, Llama-3-8B-Instruct and Mistral-7B show much lower attack success rates against many unseen attacks, including embedding-space attacks. Their MT Bench and MMLU scores barely change. The method also helps multimodal models resist image hijacks and makes tool-using agents less likely to take harmful actions. This suggests the trade-off between robustness and utility is less fixed than commonly assumed.

Key points

  • Training setup: RR is implemented within the LoRRA (RepE) framework. It combines a ReLU-cosine rerouting loss with an L2 retain loss, and a schedule shifts the weighting between the two over training. For Llama-3-8B-Instruct, the circuit breakers are applied at layers 10 and 20.
  • Cygnet: The authors built Cygnet, a Llama-3-8B-Instruct finetune that combines circuit breakers with other control methods. Against unseen attacks, it produces about two orders of magnitude less harmful content while maintaining or even improving capability.
  • Multimodal and agent results: Multimodal models withstand image-based attacks, including PGD, with little loss on capability benchmarks. A new function-calling safety benchmark shows that agents under attack take far fewer harmful actions.
  • Harmfulness probes: The authors also tested Harmfulness Probes (HP), which detect harmful representations instead of altering them. These are linear and MLP classifiers (MLP hidden sizes 64 and 32) trained on layer-16 activations for Mistral and final-layer activations for Llama-3. Their false positive rate is calibrated on WildChat. The probes beat refusal training but trail RR, and they were tested only against attackers unaware of the probe.
  • Ablations and limitations: Training on broad harm categories, such as Harmful and Illegal Activities, generalizes better than training on narrow ones like Cybercrime. The scope is limited to single-turn conversations and to attacks that seek harmful content, not traditional attacks that flip class labels.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Networks and Communications · 100 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 587 works

Cite

Cite as Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson and Dan Hendrycks. 2024. Improving Alignment and Robustness with Circuit Breakers. arXiv:2406.04313.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.04313 · arXiv 2406.04313

@misc{zou2024improving,
  title = {{Improving Alignment and Robustness with Circuit Breakers}},
  author = {Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson and Dan Hendrycks},
  year = {2024},
  doi = {10.48550/arxiv.2406.04313},
  eprint = {2406.04313},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.04313}
}

References

78 references, 44 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G. Alon and M. Kamfonas. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132, 2023.
  2. M. Andriushchenko, F. Croce, and N. Flammarion. Jailbreaking leading safety-aligned LLMs with simple adaptive attacks. arXiv preprint arXiv:2404.02151, 2024.
  3. Anthropic. Tool use (function calling), 2024. URL https://docs.anthropic.com/en/ docs/tool-use\#forcing-tool-use. Anthropic documentation. ↗
  4. Anthropic. Prefill claude’s response, 2024. URL https://docs.anthropic.com/en/docs/ prefill-claudes-response. Anthropic documentation. ↗
  5. Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. find
  6. L. Bailey, E. Ong, S. Russell, and S. Emmons. Image hijacks: Adversarial images can control generative models at runtime. arXiv preprint arXiv:2309.00236, 2023. — [arXiv:2309.00236](https://arxiv.org/abs/2309.00236) find
  7. D. Bau, H. Strobelt, W. Peebles, J. Wulff, B. Zhou, J.-Y. Zhu, and A. Torralba. Semantic photo manipulation with a generative image prior. arXiv preprint arXiv:2005.07727, 2020. — [arXiv:2005.07727](https://arxiv.org/abs/2005.07727) find
  8. E. Beeching, C. Fourrier, N. Habib, S. Han, N. Lambert, N. Rajani, O. Sanseviero, L. Tunstall, and T. Wolf. Open llm leaderboard. https://huggingface.co/spaces/HuggingFaceH4/ open_llm_leaderboard, 2023. ↗
  9. N. Carlini, M. Nasr, C. A. Choquette-Choo, M. Jagielski, I. Gao, P. W. W. Koh, D. Ippolito, F. Tramer, and L. Schmidt. Are aligned neural networks adversarially aligned? Advances in Neural Information Processing Systems, 36, 2023.
  10. M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
  11. P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong. Jailbreaking black box large language models in twenty queries, 2023.
  12. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. find
  13. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
  14. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021.
  15. N. Ding, Y. Chen, B. Xu, Y. Qin, Z. Zheng, S. Hu, Z. Liu, M. Sun, and B. Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023.
  16. M. Feffer, A. Sinha, Z. C. Lipton, and H. Heidari. Red-teaming for generative ai: Silver bullet or security theater? arXiv preprint arXiv:2401.15897, 2024. — [arXiv:2401.15897](https://arxiv.org/abs/2401.15897) find
  17. GlaiveAI. Glaive function calling v2 dataset, 2024. URL https://huggingface.co/ datasets/glaiveai/glaive-function-calling-v2. Accessed: 2024-05-21. ↗
  18. G. Goh, N. Cammarata, C. Voss, S. Carter, M. Petrov, L. Schubert, A. Radford, and C. Olah. Multimodal neurons in artificial neural networks. Distill, 6(3):e30, 2021. find
  19. Groq. Groqcloud models documentation, 2024. URL https://console.groq.com/docs/ models. GroqCloud documentation. ↗
  20. A. Helbling, M. Phute, M. Hull, and D. H. Chau. Llm self defense: By self examination, llms know they are being tricked. arXiv preprint arXiv:2308.07308, 2023.
  21. D. Hendrycks. Introduction to AI safety, ethics, and society. Taylor and Francis, 2024. find
  22. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  23. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  24. G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022.
  25. H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, and M. Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023.
  26. N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P.-y. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614, 2023.
  27. T. Kim, S. Kotha, and A. Raghunathan. Jailbreaking is best solved by definition. arXiv preprint arXiv:2403.14725, 2024. — [arXiv:2403.14725](https://arxiv.org/abs/2403.14725) find
  28. N. Leveson. Engineering a safer world: Systems thinking applied to safety. 2012. find
  29. N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A.-K. Dombrowski, S. Goel, L. Phan, G. Mukobi, N. Helm-Burger, R. Lababidi, L. Justen, A. B. Liu, M. Chen, I. Barrass, O. Zhang, X. Zhu, R. Tamirisa, B. Bharathi, A. Khoja, Z. Zhao, A. Herbert-Voss, C. B. Breuer, S. Marks, O. Patel, A.… — The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
  30. N. M. Li Maximilian, Davies Xander. Circuit breaking: Removing model behaviors with targeted ablation. arXiv preprint arXiv:2309.05973, 2023. — [arXiv:2309.05973](https://arxiv.org/abs/2309.05973) find
  31. S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022.
  32. T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár. Microsoft coco: Common objects in context, 2015.
  33. H. Ling, K. Kreis, D. Li, S. W. Kim, A. Torralba, and S. Fidler. Editgan: High-precision semantic image editing. Advances in Neural Information Processing Systems, 34:16331–16345, 2021. find
  34. H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://llava-vl.github.io/blog/ 2024-01-30-llava-next/. ↗
  35. X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023.
  36. X. Liu, N. Xu, M. Chen, and C. Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. ICLR, 2024.
  37. X. Liu, Y. Zhu, J. Gu, Y. Lan, C. Yang, and Y. Qiao. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models, 2024.
  38. A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
  39. N. Mangaokar, A. Hooda, J. Choi, S. Chandrashekaran, K. Fawaz, S. Jha, and A. Prakash. Prp: Propagating universal perturbations to attack large language model guard-rails. arXiv preprint arXiv:2402.15911, 2024. find
  40. M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. Forsyth, and D. Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. 2024.
  41. A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119, 2023.
  42. K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35, 2022. find
  43. K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022. find
  44. Meta AI. Llama-3 8b instruct. https://huggingface.co/meta-llama/ Meta-Llama-3-8B-Instruct, 2024. Instruction-tuned version of the Llama 3 model. ↗
  45. G. Mialon, R. Dessì, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Rozière, T. Schick, J. Dwivedi-Yu, A. Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842, 2023.
  46. T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.
  47. Mistral. Mistral 7b v0.2. https://huggingface.co/mistralai/ Mistral-7B-Instruct-v0.2, 2024. Fine-tuned model for instruction-following tasks. ↗
  48. E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning. Fast model editing at scale. arXiv preprint arXiv:2110.11309, 2021. find
  49. Z. Mowshowitz. Jailbreaking chatgpt on release day. https://www.lesswrong.com/posts/ RYcoJdvmoBbi5Nax7/jailbreaking-chatgpt-on-release-day, 2022. Accessed: 2024-05-19. ↗
  50. OpenAI. Gpt-4 technical report, 2023.
  51. OpenAI. Chat completions (tool_choice), 2024. URL https://platform.openai.com/ docs/api-reference/chat/create\#chat-create-tool_choice. OpenAI documentation. ↗
  52. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  53. E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022.
  54. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.
  55. M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
  56. A. Robey, E. Wong, H. Hassani, and G. J. Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684, 2023.
  57. P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263, 2023.
  58. K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. WINOGRANDE: an adversarial winograd schema challenge at scale, 2019. find
  59. C. Schlarmann and M. Hein. On the adversarial robustness of multi-modal foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3677–3685, 2023. find
  60. L. Schwinn, D. Dobre, S. Xhonneux, G. Gidel, and S. Gunnemann. Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space. arXiv preprint arXiv:2402.09063, 2024. — [arXiv:2402.09063](https://arxiv.org/abs/2402.09063) find
  61. L. Shen, W. Tan, S. Chen, Y. Chen, J. Zhang, H. Xu, B. Zheng, P. Koehn, and D. Khashabi. The language barrier: Dissecting safety challenges of llms in multilingual contexts. arXiv preprint arXiv:2401.13136, 2024.
  62. X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models, 2024.
  63. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  64. D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy, 2019. find
  65. A. Turner, L. Thiergart, D. Udell, G. Leech, U. Mini, and M. MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248, 2023.
  66. P. Upchurch, J. Gardner, G. Pleiss, R. Pless, N. Snavely, K. Bala, and K. Weinberger. Deep feature interpolation for image content changes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. find
  67. J. Vega, I. Chaudhary, C. Xu, and G. Singh. Bypassing the safety training of open-source llms with priming attacks. arXiv preprint arXiv:2312.12321, 2023. — [arXiv:2312.12321](https://arxiv.org/abs/2312.12321) find
  68. A. Wei, N. Haghtalab, and J. Steinhardt. Jailbroken: How does llm safety training fail? arXiv preprint arXiv:2307.02483, 2023.
  69. F. Yan, H. Mao, C. C.-J. Ji, T. Zhang, S. G. Patil, I. Stoica, and J. E. Gonzalez. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_ function_calling_leaderboard.html, 2024. ↗
  70. Z.-X. Yong, C. Menghini, and S. H. Bach. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446, 2023.
  71. X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Pro…
  72. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence?, 2019. find
  73. Y. Zeng, H. Lin, J. Zhang, D. Yang, R. Jia, and W. Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373, 2024.
  74. W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng. Wildchat: 1m chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Bl8u7ZRlbM. ↗
  75. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023.
  76. A. Zhou, B. Li, and H. Wang. Robust prompt optimization for defending language models against jailbreaking attacks. arXiv preprint arXiv:2401.17263, 2024. — [arXiv:2401.17263](https://arxiv.org/abs/2401.17263) find
  77. A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023.
  78. A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zou-2024-improving-alignment-robustness-circuit-breakers/