Elamur Research⌕
☰
home / papers / 2024 / Deliberative Alignment: Reasoning Enables Safer Language Models

Deliberative Alignment: Reasoning Enables Safer Language Models

Paper · 2024 · Melody Y. Guan, Manas Joglekar, Eric Wallace et al.

Summary

Conventional safety training (SFT, RLHF) has two weaknesses. LLMs answer instantly without deliberating, and they infer safety rules only indirectly from labeled examples. As a result, they can be jailbroken yet still refuse harmless requests. OpenAI researchers propose Deliberative Alignment, which teaches the model the actual text of the safety specifications and trains it to recall and reason over them in its chain-of-thought before answering.

The training data is entirely synthetic. A reasoning model trained only for helpfulness receives category-specific policy excerpts in its prompt. A policy-aware judge model filters the resulting (prompt, CoT, output) completions. The policies are then removed from the prompt (context distillation), and the remaining examples are used for SFT. A subsequent RL stage uses the same judge to provide reward. No human-written CoTs or answers are required.

Applied to the o-series models, o1 reduces both harmful compliance and overrefusal compared with GPT-4o and other strong LLMs. It also generalizes better to out-of-distribution cases. The approach moves alignment away from expensive human labeling toward scalable, interpretable reasoning over explicit rules.

Key points

  • Two-stage pipeline:
    • SFT of the base reasoning model (G_base) on judge-filtered (prompt, CoT, output) data.
    • High-compute RL in which the judge (G_RM), given the spec, supplies the reward.
    • The only label used is the broad safety category of each prompt.
  • Specification contents:
    • Content policies covering areas that include self-harm, violence, harassment, extremist material, sexually explicit content, unlawful activities, and advice in regulated domains.
    • Each request type is classed as allowed, disallowed, or requiring a safe completion.
    • Style guidelines describe how to comply, refuse, or safe-complete.
    • The spec is partly based on OpenAI's published model spec.
  • Compact prompts: To keep context short, spec(category) gives a high-level view of all categories but full detail only for the relevant one.
  • Evaluation:
    • On StrongREJECT (jailbreak prompts) versus XSTest (benign prompts), o1 pushes the Pareto frontier beyond GPT-4o.
    • Error bars come from 1,000 bootstrap trials.
    • o1 saturates many of the hardest internal safety benchmarks.
  • Ablations:
    • The SFT stage gives a strong reasoning prior, and RL refines it.
    • Building the policies into the model is more reliable than supplying them in context at deployment time.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States310 works · 284,480 citations🎓Harvard UniversityCambridge · United States73 works · 37,309 citations🏢OpenAISan Francisco · United States13 works · 18,282 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksSafety Research · 221 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 1,344 worksReasoning · 510 worksBias, fairness & safety · 209 works

Cite

Cite as Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel and Amelia Glaese. 2024. Deliberative Alignment: Reasoning Enables Safer Language Models. arXiv:2412.16339.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.16339 · arXiv 2412.16339

@misc{guan2024deliberative,
  title = {{Deliberative Alignment: Reasoning Enables Safer Language Models}},
  author = {Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel and Amelia Glaese},
  year = {2024},
  doi = {10.48550/arxiv.2412.16339},
  eprint = {2412.16339},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.16339}
}

References

38 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. OpenAI, Learning to reason with LLMs, 2024. [Online]. Available: https://openai.com/index/learning-to-reason-with-llms/. ↗
  2. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., “Training language models to follow instructions with human feedback,” in NeurIPS, 2022.
  3. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al., “The LLaMA 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. find
  4. M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024.
  5. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
  6. A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043, 2023.
  7. A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?” NeurIPS, 2024.
  8. M. Andriushchenko, F. Croce, and N. Flammarion, “Jailbreaking leading safety-aligned llms with simple adaptive attacks,” arXiv preprint arXiv:2404.02151, 2024. find
  9. J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins, “Solving math word problems with process-and outcome-based feedback,” arXiv preprint arXiv:2211.14275, 2022.
  10. C. Snell, D. Klein, and R. Zhong, “Learning by distilling context,” arXiv preprint arXiv:2209.15189, 2022. find
  11. A. Askell, Y. Bai, A. Chen, et al., “A general language assistant as a laboratory for alignment,” arXiv preprint arXiv:2112.00861, 2021. find
  12. A. Souly, Q. Lu, D. Bowen, et al., “A strongreject for empty jailbreaks,” arXiv preprint arXiv:2402.10260, 2024. find
  13. P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy, “Xstest: A test suite for identifying exaggerated safety behaviours in large language models,” arXiv preprint arXiv:2308.01263, 2024.
  14. OpenAI, Introducing the model spec, 2024. [Online]. Available: https://cdn.openai.com/spec/model-spec-2024-05-08.html. ↗
  15. W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng, “Wildchat: 1m chatgpt interaction logs in the wild,” arXiv preprint arXiv:2405.01470, 2024.
  16. X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang, “”do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models,” arXiv preprint arXiv:2308.03825, 2024.
  17. P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,” arXiv preprint arXiv:2310.08419, 2024.
  18. P. Chao, E. Debenedetti, A. Robey, et al., “Jailbreakbench: An open robustness benchmark for jailbreaking large language models,” arXiv preprint arXiv:2404.01318, 2024.
  19. P. Kumar, E. Lau, S. Vijayakumar, et al., “Refusal-trained llms are easily jailbroken as browser agents,” arXiv preprint arXiv:2410.13886, 2024. — [arXiv:2410.13886](https://arxiv.org/abs/2410.13886) find
  20. OpenAI, O1 system card, 2024. [Online]. Available: https://cdn.openai.com/o1-system-card.pdf. ↗
  21. OpenAI, Gpt-4o system card, 2024. [Online]. Available: https://cdn.openai.com/gpt-4o-system-card.pdf. ↗
  22. Anthropic, Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet, 2024. [Online]. Available: https://assets.anthropic.com/m/1cd9d098ac3e6467/original/Claude-3-Model-Card-October-Addendum.pdf. ↗
  23. G. Gemini Team, “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024.
  24. J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus, “Measuring short-form factuality in large language models,” arXiv preprint arXiv:2411.04368, 2024.
  25. A. Parrish, A. Chen, N. Nangia, V. Padmakumar, J. Phang, J. Thompson, P. M. Htut, and S. R. Bowman, “BBQ: A hand-built bias benchmark for question answering,” arXiv preprint arXiv:2110.08193, 2021.
  26. Y. Bai, S. Kadavath, S. Kundu, et al., “Constitutional AI: Harmlessness from AI feedback,” arXiv preprint arXiv:2212.08073, 2022.
  27. A. Madaan, N. Tandon, P. Gupta, et al., “Self-refine: Iterative refinement with self-feedback,” arXiv preprint arXiv:2303.17651, 2023. find
  28. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” in Advances in Neural Information Processing Systems, vol. 30, 2017. find
  29. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” arXiv preprint arXiv:2305.18290, 2024.
  30. L. Pan, M. Saxon, W. Xu, D. Nathani, X. Wang, and W. Y. Wang, “Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies,” arXiv preprint arXiv:2308.03188, 2023. find
  31. S. Welleck, X. Lu, P. West, F. Brahman, T. Shen, D. Khashabi, and Y. Choi, “Generating sequences by learning to self-correct,” in The Eleventh International Conference on Learning Representations, vol. 2, 2023.
  32. I. Schlag, S. Sukhbaatar, A. Celikyilmaz, W.-t. Yih, J. Weston, J. Schmidhuber, and X. Li, “Large language model programs,” arXiv preprint arXiv:2305.05364, 2023. find
  33. Y. Zhang, J. Chi, H. Nguyen, K. Upasani, D. M. Bikel, J. Weston, and E. M. Smith, “Backtracking improves generation safety,” arXiv preprint arXiv:2409.14586, 2024. — [arXiv:2409.14586](https://arxiv.org/abs/2409.14586) find
  34. S. Russell, Human compatible: Artificial intelligence and the problem of control, 1st. USA: Penguin Books, 2019, isbn: 9780525558637. find
  35. N. Bostrom, Superintelligence: Paths, Dangers, Strategies, 1st. USA: Oxford University Press, Inc., 2014, isbn: 0199678111. find
  36. S. M. Omohundro, “The basic ai drives,” in Proceedings of the 2008 Conference on Artificial General Intelligence 2008: Proceedings of the First AGI Conference, NLD: IOS Press, 2008, pp. 483–492, isbn: 9781586038335. find
  37. O. Järviniemi and E. Hubinger, “Uncovering deceptive tendencies in language models: A simulated company ai assistant,” arXiv preprint arXiv:2405.01576, 2024. — [arXiv:2405.01576](https://arxiv.org/abs/2405.01576) find
  38. T. Hagendorff, “Deception abilities emerged in large language models,” Proceedings of the National Academy of Sciences, vol. 121, no. 24, Jun. 2024, issn: 1091-6490. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guan-2024-deliberative-alignment-reasoning-enables-safer/