The Capacity for Moral Self-Correction in Large Language Models
Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2302.07459
Summary
The paper asks whether large dialogue models trained with RLHF can reduce their own biased or discriminatory outputs when simply asked to in plain language. The authors evaluate Anthropic decoder-only models ranging from 810M to 175B parameters, at 50 to 1000 RLHF steps. They use three test sets: BBQ for stereotypes across nine social dimensions, Winogender for occupational gender bias, and a new admissions-style benchmark for racial discrimination. The new benchmark is adapted from a dataset previously used in counterfactual fairness research. Each test runs under three prompting conditions that build on one another: a bare question (Q), the question plus an instruction to be unbiased (Q+IF), and the instruction plus chain-of-thought reasoning (Q+IF+CoT). The ability to self-correct appears at around 22B parameters and generally grows with both model size and RLHF training. Importantly, the prompts never define bias or reveal the evaluation metrics, so the models rely on concepts they learned during training. This suggests that natural-language guidance could be a cheap tool for fairness, but the authors stress that the results justify only cautious optimism.
Key points
- BBQ results: With Q+IF+CoT, the 175B model's bias score fell by 84% compared with the plain Q condition. Bias in the baseline grows with scale, but the prompted conditions reverse that trend. More RLHF training lowered bias in every condition.
- Winogender results: Pronoun choices could be pushed either way relative to U.S. Bureau of Labor Statistics occupational data. With instructions, the correlation fell to roughly 0, and with a "Match Stats" prompt it rose to nearly 1. Without instructions it stayed around 0.5. RLHF had little effect on this metric.
- Discrimination results: At 175B, the model favored white over Black applicants by about 3% under Q. Under Q+IF+CoT it favored Black applicants by about 7%, an over-correction. Demographic parity occurred at roughly 600 RLHF steps with Q+IF and roughly 200 steps with Q+IF+CoT.
- Limitations:
- The benchmarks are tied to American English and U.S. cultural norms.
- The discrimination test compares only two races.
- Prompt sensitivity was not tested systematically.
- The same techniques could be reversed to elicit harmful outputs.
- Related work: The findings echo Constitutional AI. They also contrast with earlier negative results for RoBERTa-large on the UnQover benchmark.
Cited by
- ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models
- Towards Measuring the Representation of Subjective Global Opinions in Language Models
- Large Language Models Cannot Self-Correct Reasoning Yet
- Language Models can Solve Computer Tasks
- Measuring Faithfulness in Chain-of-Thought Reasoning
- Self-Alignment with Instruction Backtranslation
- A Comprehensive Overview of Large Language Models
- Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models
- On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
- Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
- DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
- Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
- Large Language Models as Optimizers
- Low-Resource Languages Jailbreak GPT-4
Cite
@misc{ganguli2023capacity,
title = {{The Capacity for Moral Self-Correction in Large Language Models}},
author = {Ganguli, Deep and Askell, Amanda and Schiefer, Nicholas and Liao, Thomas I. and Lukovsiute, Kamile and Chen, Anna and Goldie, Anna and Mirhoseini, Azalia and Olsson, Catherine and Hernandez, Danny and Drain, Dawn and Li, Dustin and Tran-Johnson, Eli and Perez, Ethan and Kernion, Jackson and Kerr, Jamie and Mueller, Jared and Landau, Joshua and Ndousse, Kamal and Nguyen, Karina and Lovitt, Liane and Sellitto, Michael and Elhage, Nelson and Mercado, Noemi and DasSarma, Nova and Rausch, Oliver and Lasenby, Robert and Larson, Robin and Ringer, Sam and Kundu, Sandipan and Kadavath, Saurav and Johnston, Scott and Kravec, Shauna and Showk, Sheer El and Lanham, Tamera and Telleen-Lawton, Timothy and Henighan, Tom and Hume, Tristan and Bai, Yuntao and Hatfield-Dodds, Zac and Mann, Ben and Amodei, Dario and Joseph, Nicholas and McCandlish, Sam and Brown, Tom and Olah, Christopher and Clark, Jack and Bowman, Samuel R. and Kaplan, Jared},
year = {2023},
doi = {10.48550/arxiv.2302.07459},
eprint = {2302.07459},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2302.07459}
}References
- [1] A. Abid, M. Farooqi, and J. Zou. Large language models associate Muslims with violence. Nature Machine Intelligence, 3(6):461–463, June 2021. Number: 6 Publisher: Nature Publishing Group. find