Elamur Research⌕
☰
home / papers / 2023 / Training Socially Aligned Language Models on Simulated Social Interactions

Training Socially Aligned Language Models on Simulated Social Interactions

Paper · 2023 · Ruibo Liu, Ruixin Yang, Chenyan Jia et al.

Summary

Language models are usually aligned with social values in one of two ways. Supervised fine-tuning (SFT) trains them to imitate curated "good" data. RLHF optimizes them against a learned reward model. The authors argue that SFT generalizes poorly and is easily jailbroken, while reward models can be exploited through reward gaming. Their alternative draws on how people absorb norms from peer feedback. They built SANDBOX, an open-source simulated society of about one hundred LM-based agents. Agents answer controversial or risky questions, gather ratings and written critiques from neighbouring agents through a scheme called Back-Scatter, and then revise their answers. Separate memoryless observer agents score each answer for alignment and engagement. The logged interactions become 169k training samples, which feed Stable Alignment, a three-stage training method that needs no separate reward model. Stable Alignment outperforms existing methods on six alignment benchmarks and resists jailbreaking prompts roughly as well as ChatGPT. This makes alignment cheaper and more stable, especially where compute is limited.

Key points

  • SANDBOX and Back-Scatter: In each round, a central agent drafts a response, nearby agents rate and critique it, and the central agent revises.
    • Agents keep a memory of past answers and retrieve relevant ones via embedding search, which keeps their opinions consistent.
    • Observer agents rate alignment and engagement on a 7-point Likert scale.
    • A simulation ends when the product of the two ratings stops improving (Pareto optimality).
  • Training data: The 169k samples come in three types: Imitation, Self-Critic, and Realignment. Beyond good and bad examples, they include collective ratings, detailed feedback, and step-by-step revisions.
  • Model scale: In the simulations, moving from a 6.8B to a 175B GPT-3 model (about 20× larger) gave no significant alignment gain. Already-aligned models reached good outcomes in fewer interaction rounds.
  • Benchmarks and resources: Stable Alignment beats existing methods on six alignment benchmarks. Because it needs no reward model, unlike OpenAI's RLHF, it is easier to deploy in resource-constrained settings.
  • Jailbreak robustness: On adversarial prompts such as bank-robbery requests, the model refuses about as reliably as ChatGPT, while Alpaca produces actionable plans. Its answers also rarely open with a stock "As an AI language model" disclaimer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,053 citations🎓University of British ColumbiaVancouver · Canada104 works · 37,402 citations🎓University of MichiganAnn Arbor · United States86 works · 15,493 citations🎓Dartmouth CollegeHanover · United States41 works · 42,719 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M. Dai, Diyi Yang and Soroush Vosoughi. 2023. Training Socially Aligned Language Models on Simulated Social Interactions. arXiv:2305.16960.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.16960 · arXiv 2305.16960

@misc{liu2023training,
  title = {{Training Socially Aligned Language Models on Simulated Social Interactions}},
  author = {Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M. Dai, Diyi Yang and Soroush Vosoughi},
  year = {2023},
  doi = {10.48550/arxiv.2305.16960},
  eprint = {2305.16960},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.16960}
}

References

68 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gati Aher, Rosa I. Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies, 2023. find
  2. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. ArXiv preprint, abs/1606.06565, 2016. URL https://arxiv.org/abs/1606.06565. ↗
  3. Jacob Andreas. Language models as agent models. ArXiv preprint, abs/2212.01681, 2022. URL https://arxiv.org/abs/2212.01681. ↗
  4. Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. ArXiv preprint, abs/2209.06899, 2022. URL https://arxiv.org/abs/2209.06899. ↗
  5. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. ArXiv preprint, abs/2112.00861, 2021. URL https://arxiv.org/abs/2112.00861. ↗
  6. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. ArXiv preprint, abs/2212.08073, 2022. URL https://arxiv.org/abs/2212.08073. ↗
  7. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. ArXiv preprint, abs/2108.07258, 2021. URL https://arxiv.org/abs/2108.07258. ↗
  8. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  9. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys. org (accessed 14 April 2023), 2023. ↗
  10. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Informat… find
  11. Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 698–718, Online and Punta Cana, Dominica… find
  12. Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese, 198 (Suppl 27):6435–6467, 2021. find
  13. Iason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411–437, 2020. find
  14. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. ArXiv preprint, abs/2210.10760, 2022. URL https://arxiv.org/abs/2210.10760. ↗
  15. Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894–6910, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.…
  16. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3356–3369, Online, 2020. Association for Computational Linguistics…
  17. Amelia Glaese, Nat McAleese, Maja Tr˛ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. ArXiv preprint, abs/2209.14375, 2022. URL https://arxiv.org/abs/2209.143… ↗
  18. Charles AE Goodhart. Problems of monetary management: the UK experience. Springer, 1984. find
  19. Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms. arXiv preprint arXiv:2305.15717, 2023.
  20. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning AI with shared human values. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
  21. OpenReview.net, 2021. URL https://openreview.net/forum?id=dNy_RKzJacY. ↗
  22. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7038–7051, Online and Punta Cana, Dominican Repub…
  23. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022.
  24. Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. abs/1805.00899, 2018. URL https://arxiv.org/abs/1805.00899. ↗
  25. Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp… find
  26. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. Alignment of language agents. ArXiv preprint, abs/2103.14659, 2021. URL https://arxiv.org/abs/2103.14659. ↗
  27. Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille, 2015. find
  28. Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, and Shane Legg. Avoiding side effects by considering future tasks. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conf… find
  29. Ranjay Krishna, Donsuk Lee, Li Fei-Fei, and Michael S Bernstein. Socially situated artificial intelligence enables learning from human interaction. Proceedings of the National Academy of Sciences, 119(39):e2115730119, 2022. find
  30. Kai-Fu Lee. A human blueprint for ai coexistence., 2021. find
  31. Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research c… — [arXiv:1803.03453](https://arxiv.org/abs/1803.03453) find
  32. Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. Ai safety gridworlds. arXiv preprint arXiv:1711.09883, 2017. — [arXiv:1711.09883](https://arxiv.org/abs/1711.09883) find
  33. ArXiv preprint, Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. ArXiv preprint, abs/1811.07871, 2018. URL https://arxiv.org/abs/1811.07871. ↗
  34. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214–3252, Dublin, Ireland, 2022. Association for Computational Linguistics.…
  35. H Liu, C Sferrazza, and P Abbeel. Chain of hindsight aligns language models with feedback. ArXiv preprint, abs/2302.02676, 2023. URL https://arxiv.org/abs/2302.02676. ↗
  36. Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. Mitigating political bias in language models through reinforced calibration. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intel… find
  37. AAAI Press, 2021. URL https://ojs.aaai.org/index.php/AAAI/article/view/17744. ↗
  38. Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, and Soroush Vosoughi. Quantifying and alleviating political bias in language models. Artificial Intelligence, 304:103654, 2022. find
  39. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744, 2022.
  40. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. find
  41. OpenReview.net, 2022. URL https://openreview.net/forum?id=JYtwGwIL7ye. ↗
  42. Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pp. 1–18, 2022.
  43. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. ArXiv preprint, abs/2304.03442, 2023. URL https://arxiv.org/abs/2304.03442. ↗
  44. Laurie Ann Paul. Transformative experience. OUP Oxford, 2014. find
  45. Richard Pettigrew. Choosing for changing selves. Oxford University Press, 2019. find
  46. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. ArXiv preprint, abs/2112.11446, 2021. URL https://arxiv.org/abs/2… ↗
  47. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023.
  48. Jason DM Rennie and Nathan Srebro. Loss functions for preference levels: Regression with discrete ordered labels. In Proceedings of the IJCAI multidisciplinary workshop on advances in preference handling, volume 1. AAAI Press, Menlo Park, CA, 2005. find
  49. Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 815–823.
  50. IEEE Computer Society, 2015. doi: 10.1109/CVPR.2015.7298682. URL https://doi.org/10.1109/CVPR.2015.7298682. ↗
  51. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget.(may 2023), 2023.
  52. Jacob Steinhardt. Ml systems will have weird failure modes. https://bounded-regret.ghost.io/ml-systems-will-have-weird-failure-modes-2/, 2022. ↗
  53. Varshini Subhash. Can large language models change user preference adversarially? ArXiv preprint, abs/2302.10291, 2023. URL https://arxiv.org/abs/2302.10291. — [arXiv:2302.10291](https://arxiv.org/abs/2302.10291) ↗
  54. Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. Understanding the capabilities, limitations, and societal impact of large language models. ArXiv preprint, abs/2102.02503, 2021. URL https://arxiv.org/abs/2102.02503. ↗
  55. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. ↗
  56. Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. Alignment for advanced machine learning systems. Ethics of Artificial Intelligence, pp. 342–382, 2016. find
  57. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971, 2023a. URL https://arxiv.org/abs/2302.13971. ↗
  58. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  59. Pranav Narayanan Venkit, Mukund Srinath, and Shomir Wilson. A study of implicit bias in pretrained language models against people with disabilities. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 1324–1332, Gyeongju, Republic of Korea, 2022. International Commi… find
  60. Leandro von Werra et al. Transformer https://github.com/CarperAI/trlx, 2023. ↗
  61. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. ArXiv preprint, abs/2212.10560, 2022. URL https://arxiv.org/abs/2212.10560. ↗
  62. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. Taxonomy of risks posed by language models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 214–229, 2022. find
  63. Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models. ArXiv preprint, abs/2304.11082, 2023. URL https://arxiv.org/abs/2304.11082. ↗
  64. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2950–2968, Online…
  65. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. ArXiv preprint, abs/2304.05302, 2023. URL https://arxiv.org/abs/2304.05302. ↗
  66. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023. reinforcement learning x. find
  67. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. find
  68. Caleb Ziems, Jane Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. The moral integrity corpus: A benchmark for ethical dialogue systems. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3755–3773, Dublin, Ireland, May 2022. Associat…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2023-training-socially-aligned-language-models/