Constitutional AI: Harmlessness from AI Feedback
Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2212.08073
Summary
The paper looks for a way to train a language assistant to avoid harmful responses without relying on human labels that mark harm. The authors, from Anthropic, propose Constitutional AI (CAI). In this approach, humans provide only a short list of principles in natural language (a "constitution") and a few few-shot examples. Training proceeds in two stages. In the SL stage, a model previously trained only to be helpful responds to provocative red-teaming prompts. It then critiques its own responses in light of the principles and revises them, and it is fine-tuned on the revised texts. In the RL stage, the model compares pairs of responses. A preference model is trained on these comparisons and then serves as the reward. The authors call this approach RLAIF. Both stages can draw on chain-of-thought. The final RL-CAI model does not evade difficult questions; instead it explains why it refuses. Crowdworkers rate it as less harmful than RLHF models trained on human labels, and no less helpful. This shows that AI oversight can be scaled when the models themselves help carry it out.
Key points
- Instead of tens of thousands of human labels on harmfulness, about ten simple principles are used. The authors acknowledge that they chose them rather arbitrarily, by trial and error.
- The SL-CAI, RL-CAI and RL-CAI w/ CoT models are compared with Helpful RLHF and HH RLHF. The comparison used Elo scores for helpfulness and harmlessness, derived from crowdworkers' pairwise preferences.
- For models with 52B parameters, RL-CAI becomes less harmful at the same level of helpfulness. The authors describe this as a Pareto improvement over standard RLHF.
- The SL stage gives an "on-distribution" starting point for RL and mitigates the problem of exploring the response space. Most of the gain in quality and robustness comes from the RL stage.
- The authors warn about dual use: the method also makes it easier to train malicious systems. In addition, models could be released with less human review.
Cited by
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- The Falcon Series of Open Language Models
- Qwen Technical Report
- Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
- Jailbreaking Black Box Large Language Models in Twenty Queries
- AlpaGasus: Training A Better Alpaca with Fewer Data
- TheoremQA: A Theorem-driven Question Answering Dataset
- Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Safe RLHF: Safe Reinforcement Learning from Human Feedback
- RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
- Towards Measuring the Representation of Subjective Global Opinions in Language Models
- Bias and Fairness in Large Language Models: A Survey
- Gemini: A Family of Highly Capable Multimodal Models
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
- The False Promise of Imitating Proprietary LLMs
- Evaluating Large Language Models: A Comprehensive Survey
- V-STaR: Training Verifiers for Self-Taught Reasoners
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
- Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation
- Large Language Models Cannot Self-Correct Reasoning Yet
- TrustLLM: Trustworthiness in Large Language Models
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models
- BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
- A Survey on Large Language Models for Code Generation
- Challenges and Applications of Large Language Models
- Language Models can Solve Computer Tasks
- A Watermark for Large Language Models
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
- DeepInception: Hypnotize Large Language Model to Be Jailbreaker
- Generative Judge for Evaluating Alignment
- Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
- Self-Alignment with Instruction Backtranslation
- Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
- On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
- WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct
- Augmented Language Models: a Survey
- Orca: Progressive Learning from Complex Explanation Traces of GPT-4
- A Comprehensive Overview of Large Language Models
- GPT-4 Technical Report
- REFINER: Reasoning Feedback on Intermediate Representations
- Instruction Tuning with GPT-4
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
- XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
- Whose Opinions Do Language Models Reflect?
- Role-Play with Large Language Models
- Towards Understanding Sycophancy in Language Models
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- Preference Ranking Optimization for Human Alignment
- A Roadmap to Pluralistic Alignment
- Aligning Large Multimodal Models with Factually Augmented RLHF
- Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
- MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
- Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
- OpenChat: Advancing Open-source Language Models with Mixed-Quality Data
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
- Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
- Jailbroken: How Does LLM Safety Training Fail?
- Long-form factuality in large language models
- Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
- The Rise and Potential of Large Language Model Based Agents: A Survey
- Baichuan 2: Open Large-scale Language Models
- Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
- Large Language Models as Optimizers
- Large Language Model Unlearning
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- Low-Resource Languages Jailbreak GPT-4
- GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher
- Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning
- Instruction Tuning for Large Language Models: A Survey
- Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
- Generative Verifiers: Reward Modeling as Next-Token Prediction
- ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search
- SLiC-HF: Sequence Likelihood Calibration with Human Feedback
- A Survey of Large Language Models
- WildChat: 1M ChatGPT Interaction Logs in the Wild
- LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
- Secrets of RLHF in Large Language Models Part I: PPO
- Universal and Transferable Adversarial Attacks on Aligned Language Models
Cite
@misc{bai2022constitutional,
title = {{Constitutional AI: Harmlessness from AI Feedback}},
author = {Bai, Yuntao and Kadavath, Saurav and Kundu, Sandipan and Askell, Amanda and Kernion, Jackson and Jones, Andy and Chen, Anna and Goldie, Anna and Mirhoseini, Azalia and McKinnon, Cameron and Chen, Carol and Olsson, Catherine and Olah, Christopher and Hernandez, Danny and Drain, Dawn and Ganguli, Deep and Li, Dustin and Tran-Johnson, Eli and Perez, Ethan and Kerr, Jamie and Mueller, Jared and Ladish, Jeffrey and Landau, Joshua and Ndousse, Kamal and Lukosuite, Kamile and Lovitt, Liane and Sellitto, Michael and Elhage, Nelson and Schiefer, Nicholas and Mercado, Noemi and DasSarma, Nova and Lasenby, Robert and Larson, Robin and Ringer, Sam and Johnston, Scott and Kravec, Shauna and Showk, Sheer El and Fort, Stanislav and Lanham, Tamera and Telleen-Lawton, Timothy and Conerly, Tom and Henighan, Tom and Hume, Tristan and Bowman, Samuel R. and Hatfield-Dodds, Zac and Mann, Ben and Amodei, Dario and Joseph, Nicholas and McCandlish, Sam and Brown, Tom and Kaplan, Jared},
year = {2022},
doi = {10.48550/arxiv.2212.08073},
eprint = {2212.08073},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2212.08073}
}References
- Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. (2021). A general langua… find our card
- Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C.,… find
- Bowman, S. R., Hyun, J., Perez, E., Chen, E., Pettit, C., Heiner, S., Lukosuite, K., Askell, A., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Olah, C., Amodei, D., Amodei, D., Drain, D., Li, D., Tran-Johnson, E., Kernion, J., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse,… find
- Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. find
- Christiano, P., Shlegeris, B., and Amodei, D. (2018). Supervising strong learners by amplifying weak experts. find
- Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., Jones, A., Bowman, S., Chen, A., Conerly, T., DasSarma, N., Drain, D., Elhage, N., El-Showk, S., Fort, S., Dodds, Z. H., Henighan, T., Hernandez, D., Hume, T., Jacobson, J., Johns… find our card
- Gao, L., Schulman, J., and Hilton, J. (2022). Scaling laws for reward model overoptimization. find
- Glaese, A., McAleese, N., Tr˛ebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., Campbell-Gillingham, L., Uesato, J., Huang, P.-S., Comanescu, R., Yang, F., See, A., Dathathri, S., Greig, R., Chen, C., Fritz, D., Elias, J. S., Green, R., Mokrá, S.,… find our card
- Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. (2022). Large language models can self-improve. find our card
- Irving, G., Christiano, P., and Amodei, D. (2018). Ai safety via debate. find
- Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Dodds, Z. H., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., Kravec,… find
- Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022). Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916. find our card
- Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., Sutton, C., and Odena, A. (2021). Show your work: Scratchpads for intermediate computation with language models. find our card
- Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. find
- Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. (2022). Red teaming language models with language models. find our card
- Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J. (2022). Self-critiquing models for assisting human evaluators. find our card
- Scheurer, J., Campos, J. A., Chan, J. S., Chen, A., Cho, K., and Perez, E. Training language models with language feedback. find
- Shi, W., Dinan, E., Shuster, K., Weston, J., and Xu, J. (2022). When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels. find
- Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., and Hassabis, D. (2017). Mastering chess and shogi by self-play with a general reinforcement learning algorithm. find our card
- Solaiman, I. and Dennison, C. (2021). Process for adapting language models to society (PALMS) with values-targeted datasets. CoRR, abs/2106.10328. find
- Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. find our card
- Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. (2020). Learning to summarize from human feedback. find
- Thoppilan, R., Freitas, D. D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H. S., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhou, Y., Chang, C., Kr… find our card
- Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (2022). Chain of thought prompting elicits reasoning in large language models. find our card
- Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., and Dinan, E. (2020). Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079. find
- Zhao, J., Khashabi, D., Khot, T., Sabharwal, A., and Chang, K.-W. (2021). Ethical-advice taker: Do language models understand natural language interventions? find