Elamur Research⌕
☰
home / papers / 2023 / Fundamental Limitations of Alignment in Large Language Models

Fundamental Limitations of Alignment in Large Language Models

Paper · 2023 · Yotam Wolf, Noam Wies, Oshri Avnery et al.

Summary

Wolf et al. ask whether aligning a large language model, via RLHF finetuning or aligning prompts, can make it robust to adversarial users. To analyze this, they develop Behavior Expectation Bounds (BEB), a framework grounded in probability theory. Each behavior vertical (e.g., helpfulness, politeness) assigns sentences a score from −1 to +1, and a model's alignment is the expected score of its outputs. The model is viewed as a mixture of good and bad components, described by α, β, γ-distinguishability and σ-similarity. Under these assumptions the authors prove a general result. If an undesired behavior keeps any nonzero probability, some prompt can elicit it, and success becomes more likely as the prompt grows longer. Aligning prefixes, multi-turn conversations and best-of-n sampling do not remove this weakness. Experiments on Llama 2 13B and its RLHF chat version estimate the framework's parameters. They show the model's output distribution shifting toward a negative persona as adversarial context accumulates. The work gives a formal account of persona-style jailbreaks. It argues that merely suppressing harmful behavior, rather than eliminating it, cannot guarantee safety.

Key points

  • The paper proves four theorems:
    • Theorem 1: attenuated but nonzero bad behavior is exploitable.
    • Theorem 2: an aligning prefix offers only a bounded guardrail.
    • Theorem 3: a user can misalign the model over a conversation, even with limited prompt length per turn.
    • Theorem 4: best-of-n selection does not ensure alignment.
  • Experiments compare Llama 2 13B with Llama 2 13B chat on two behaviors, "agreeableness" and "anti-immigration." They track the decay of KL divergence between a negative component P− and the model as misaligning prompts lengthen.
  • The estimated β for the pretrained model is about 1–2, roughly 5× smaller than for the RLHF model. Since the theory links misalignment speed to 1/β, this gives a preliminary hint that RLHF makes undesired behaviors more distinguishable.
  • For agreeableness, distinguishability is about twice that of anti-immigration, and σ-similarity about half. In the prompt-distinguishability variant, β ≈ 20.
  • Method details:
    • KL divergence was estimated from 64 sampled 8-token continuations, averaged over 10 sequences.
    • Behavior expectation was measured by manually labeling model responses (32 per prompt step for the chat model), judged by their first sentence.
    • The base model produced negative replies after a single adversarial sentence, while the chat model initially stayed positive.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hebrew University of JerusalemJerusalem · Israel50 works · 9,147 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksSafety Research · 221 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,343 works

Cite

Cite as Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine and Amnon Shashua. 2023. Fundamental Limitations of Alignment in Large Language Models. arXiv:2304.11082.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.11082 · arXiv 2304.11082

@misc{wolf2023fundamental,
  title = {{Fundamental Limitations of Alignment in Large Language Models}},
  author = {Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine and Amnon Shashua},
  year = {2023},
  doi = {10.48550/arxiv.2304.11082},
  eprint = {2304.11082},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.11082}
}

References

47 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.
  2. Jacob Andreas. Language models as agent models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5769–5779, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022. findings-emnlp.423. ↗
  3. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. find
  4. Imane El Atillah. Man ends his life after an ai chatbot ’encouraged’ him to sacrifice himself to stop climate change. Euronews, 2023. find
  5. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. find
  6. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610–623, 2021. find
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  8. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
  9. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335, 2023.
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
  11. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3356–3369, Online, November 2020. Association for Computational Li…
  12. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916, 2021. find
  13. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum? id=nZeVKeeFYf9. ↗
  14. Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. Social biases in NLP models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5491–5501, Online, July 2020. Ass…
  15. Ole Jorgensen, Dylan Cope, Nandi Schoots, and Murray Shanahan. Improving activation steering in language models with mean-centring. arXiv preprint arXiv:2312.03813, 2023. — [arXiv:2312.03813](https://arxiv.org/abs/2312.03813) find
  16. Chak Tou Leong, Yi Cheng, Jiashuo Wang, Jian Wang, and Wenjie Li. Self-detoxifying language models via toxification reversal. arXiv preprint arXiv:2310.09573, 2023. find
  17. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214–3252, Dublin, Ireland, May 2022. Association for Computational Linguisti…
  18. Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Aligning large language models with human preferences through representation engineering. arXiv preprint arXiv:2312.15997, 2023. find
  19. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/ peft, 2022. ↗
  20. AI Meta. Introducing llama: A foundational, 65-billion-parameter large language model. Meta AI. https://ai.facebook. com/blog/large-language-model-llama-meta-ai, 2023. ↗
  21. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1953–1967, Online, November 2020. Associat…
  22. Cleo Nardo. The waluigi effect (mega-post). Less Wrong, 2023. find
  23. Richard Ngo. The alignment problem from a deep learning perspective. arXiv:2209.00626, 2022. arXiv preprint Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375, 2023.
  24. Matt O’Brien. Musk, scientists call for halt to ai race sparked by chatgpt. AP News, 2023. find
  25. OpenAI. Gpt-4 technical report, 2023.
  26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training l…
  27. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=JYtwGwIL7ye. ↗
  28. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. find
  29. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022.
  30. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. find
  31. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. find
  32. Kevin Roose. A conversation with bing’s chatbot left me deeply unsettled. New York Times, 2023. find
  33. John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe, Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, Rapha Gontijo Lopes, Shengjia Zhao, Arun Vijayvergiya, Eric Sigler, Adam Perelman, Chelsea Voss, Mike Heaton, Joel Parish, Dave Cummings, Rajeev Nayak,… find
  34. Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. On the ethics of building ai in a responsible manner. arXiv preprint arXiv:2004.04644, 2020. — [arXiv:2004.04644](https://arxiv.org/abs/2004.04644) find
  35. Varshini Subhash. Can large language models change user preference adversarially? arXiv preprint arXiv:2302.10291, 2023. — [arXiv:2302.10291](https://arxiv.org/abs/2302.10291) find
  36. Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. Alignment for advanced machine learning systems. Ethics of Artificial Intelligence, pp. 342–382, 2016. find
  37. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. find
  38. Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248, 2023.
  39. Pranav Narayanan Venkit, Mukund Srinath, and Shomir Wilson. A study of implicit bias in pretrained language models against people with disabilities. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 1324–1332, Gyeongju, Republic of Korea, October 2022. Internation… find
  40. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find
  41. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sea… — [doi:10.1145/3531146.3533088](https://doi.org/10.1145/3531146.3533088) find
  42. Colin G West. Advances in apparent conceptual physics reasoning in gpt-4. arXiv e-prints, pp. arXiv–2303, 2023. find
  43. Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=f3JNQd7CHM. ↗
  44. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2950–2968, Online…
  45. Dian Yu and Kenji Sagae. Automatically exposing problems with neural dialog models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 456–470, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.186… find
  46. Eliezer Yudkowsky. Creating friendly ai 1.0: The analysis and design of benevolent goal architectures. The Singularity Institute, San Francisco, USA, 2001. find
  47. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wolf-2023-fundamental-limitations-alignment-large-language/