A General Language Assistant as a Laboratory for Alignment
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2112.00861
Summary
The authors from Anthropic seek a path toward a general-purpose text-based assistant aligned with human values. They define alignment through three HHH criteria: the assistant should be helpful, honest, and harmless. First, they test the simplest baseline technique: adding to the model's context a prompt with examples of the desired assistant behavior. Then, using context distillation, they transfer this effect into the model itself. Next, they compare how three training objectives scale with model size: imitation learning, binary discrimination, and ranked preference modeling. Finally, they propose an intermediate stage of preference model pre-training (PMP) on large public datasets that reflect human preferences. The benefit of the prompt grows with model size, generalizes across different alignment evaluations, and barely degrades the performance of large models. Ranked preference modeling markedly outperforms imitation. Binary discrimination, by contrast, yields almost the same results as imitation. PMP makes it possible to fine-tune a preference model on fewer examples. The work lays the methodological groundwork for training assistants with reinforcement learning from human feedback.
Key points
- The baseline HHH prompt imposes almost no "alignment tax" on large models. This was verified even on code synthesis: on HumanEval, the fraction of solutions passing the tests (pass@10) is nearly the same with and without the prompt (T = 0.6, top P = 0.95).
- Ranked preference modeling wins when responses can be ordered from worse to better, for example by helpfulness. If the choice is binary, however (say, whether a function passes the tests), it is no better than imitation.
- For PMP, public data from Stack Exchange, Reddit, and Wikipedia edits were used. After that, the model is fine-tuned on smaller specialized human preference datasets.
- Models of various sizes were studied: from roughly 10⁷ to more than 10¹⁰ parameters. The advantages of most methods grow with scale.
- In the appendix, the authors analyze the HHH criteria and possible conflicts between them in detail. There they also discuss a possible fourth "H" — handleable, i.e. controllability close to the notion of corrigibility.
Cited by
- Constitutional AI: Harmlessness from AI Feedback
- Qwen Technical Report
- GPT-NeoX-20B: An Open-Source Autoregressive Language Model
- A Survey on Evaluation of Large Language Models
- AlpaGasus: Training A Better Alpaca with Fewer Data
- Adapting Language Models to Compress Contexts
- Safe RLHF: Safe Reinforcement Learning from Human Feedback
- RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
- Towards Measuring the Representation of Subjective Global Opinions in Language Models
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
- Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
- Improving alignment of dialogue agents via targeted human judgements
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
- How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection
- Evaluating Large Language Models: A Comprehensive Survey
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
- Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation
- Editing Models with Task Arithmetic
- BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
- Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
- Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
- OpenAssistant Conversations -- Democratizing Large Language Model Alignment
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
- LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- Let's Verify Step by Step
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
- Visual Instruction Tuning
- Teaching language models to support answers with verified quotes
- Learning to Compress Prompts with Gist Tokens
- Orca: Progressive Learning from Complex Explanation Traces of GPT-4
- A Comprehensive Overview of Large Language Models
- Instruction Tuning with GPT-4
- Discovering Language Model Behaviors with Model-Written Evaluations
- Red Teaming Language Models with Language Models
- InFoBench: Evaluating Instruction Following Ability in Large Language Models
- XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
- Whose Opinions Do Language Models Reflect?
- A Roadmap to Pluralistic Alignment
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
- Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
- Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
- Emergent Abilities of Large Language Models
- The Rise and Potential of Large Language Model Based Agents: A Survey
- RRHF: Rank Responses to Align Language Models with Human Feedback without tears
- SLiC-HF: Sequence Likelihood Calibration with Human Feedback
- A Survey of Large Language Models
- Secrets of RLHF in Large Language Models Part I: PPO
- Large Language Models Are Human-Level Prompt Engineers
Cite
@misc{askell2021general,
title = {{A General Language Assistant as a Laboratory for Alignment}},
author = {Askell, Amanda and Bai, Yuntao and Chen, Anna and Drain, Dawn and Ganguli, Deep and Henighan, Tom and Jones, Andy and Joseph, Nicholas and Mann, Ben and DasSarma, Nova and Elhage, Nelson and Hatfield-Dodds, Zac and Hernandez, Danny and Kernion, Jackson and Ndousse, Kamal and Olsson, Catherine and Amodei, Dario and Brown, Tom and Clark, Jack and McCandlish, Sam and Olah, Chris and Kaplan, Jared},
year = {2021},
doi = {10.48550/arxiv.2112.00861},
eprint = {2112.00861},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2112.00861}
}References
- Josh Abramson, Arun Ahuja, Iain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, Petko Georgiev, Aurelia Guy, Tim Harley, Felix Hill, Alden Hung, Zachary Kenton, Jessica Landon, Timothy Lillicrap, Kory Mathewson, Sona Mokra, Alista… find
- Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021, 2108.07732. find our card
- Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, 2016, 1606.06565. + find our card
- Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning, 2021, 2111.10952. find
- Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations, 2019, 1904.06387. find
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
- Samuel R. Bowman. When combating hype, proceed with caution, 2021, 2110.08300. + find
- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021, 2110.14168. find our card
- Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2017, 1706.03741. + find
- Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, 2021, 2106.01345. find
- Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts, 2018, 1810.08575. find
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… find our card
- The Common Crawl Foundation. Common crawl. URL http://commoncrawl.org. ↗
- Iason Gabriel. Artificial intelligence, values, and alignment. Minds and Machines, 30(3):411– 437, Sep 2020. doi:10.1007/s11023-020-09539-2. — [doi:10.1007/s11023-020-09539-2](https://doi.org/10.1007/s11023-020-09539-2) find
- Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020, 2101.00027. find
- Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models, 2020, 2009.11462. find our card
- Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values, 2021, 2008.02275. find our card
- Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety, 2021, 2109.13916. find
- Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning, 2016, 1606.03476. find
- Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017, 1712.00409. find
- Laura Hanu and Unitary team. Detoxify. Github. https://github.com/unitaryai/detoxify, 2020. ↗
- Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate, 2018, 1805.00899. + find
- Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari, 2018, 1811.06521. find
- Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. Delphi: Towards machine ethics and norms, 2021, 2110.07574. find
- Andy L. Jones. Scaling scaling laws with board games, 2021, 2104.03113. + find
- Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020, 2001.08361. find
- Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation, 2021, 2107.07566. find our card
- Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021, 2104.08691. find
- Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2021, 2109.07958. find our card
- Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. Quixbugs: A multilingual program repair benchmark set based on the quixey challenge. In Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity… — [doi:10.1145/3135932.3135941](https://doi.org/10.1145/3135932.3135941) find
- Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv:1801.10198 [cs], 2018, 1801.10198. URL http://arxiv.org/abs/1801.10198. ↗ our card
- James Edwin Mahon. The definition of lying and deception. In Ed Zalta, editor, Stanford Encyclopedia of Philosophy. 2015. find
- Helen Ngo, Cooper Raterink, João G. M. Araújo, Ivan Zhang, Carol Chen, Adrien Morisot, and Nicholas Frosst. Mitigating harm in language models with conditional-likelihood filtration, 2021, 2108.07790. find
- Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context, 2016, 1606.06031. find our card
- Vinay V. Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics, 2020, 2007.07400. find
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. find
- Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales, 2019, arXiv:1909.12673. find
- Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. openai.com, 2019. find
- Irene Solaiman and Christy Dennison. Process for adapting language models to society (PALMS) with values-targeted datasets. CoRR, abs/2106.10328, 2021, 2106.10328. URL https://arxiv.org/abs/2106.10328. ↗
- Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015. find
- Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. CoRR, 2015, 1508.07909. find
- Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2020, 2009.01325. find
- Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Da… find our card
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing S… find
- Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2021, 2109.01652. + find
- Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models, 2021, arXiv:2109.07445. + find
- Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback, 2021, 2109.10862. + find our card
- Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019, 1905.07830. find