Elamur Research⌕
☰
home / papers / 2022 / Scaling Laws for Reward Model Overoptimization

Scaling Laws for Reward Model Overoptimization

Paper · 2022 · Leo Gao, John Schulman and Jacob Hilton

Summary

In reinforcement learning from human feedback (RLHF), responses are scored by a reward model (RM), but the RM only approximates the true preferences of humans. If the model is optimized too strongly against this score, the actual quality of responses eventually declines. This is a manifestation of Goodhart's law. Measuring this effect precisely with live annotators is prohibitively expensive. The authors from OpenAI therefore built a synthetic setup. The role of humans is played by a fixed "gold" RM with 6B parameters. Its labels are used to train proxy RMs ranging in size from 3M to 3B. Policies based on GPT-3 in the InstructGPT environment were optimized against the proxy in two ways: best-of-n sampling (BoN) and PPO. Throughout, the authors tracked how the gold score changes as a function of the KL distance from the initial policy. For each method, the curve has its own simple analytic form. Its coefficients vary smoothly, approximately logarithmically, with proxy RM size. This makes it possible to estimate in advance how much more optimization is still safe. This is useful both for RLHF practice and for AI alignment research.

Key points

  • Formulas: for BoN the gold score is R = d(α − βd), and for RL R = d(α − β·log d). Here d is the square root of the KL between the optimized and the initial policy. The BoN form was validated by a genuine forward prediction: it was fitted on data up to n = 1,000 (≈6 nats) and then confirmed for n up to 60,000 (≈10 nats).
  • Data: 100,000 synthetic comparisons, of which 10% were held out for testing. The label is deterministic: the response with the higher gold score is considered better. With fewer than about 2,000 comparisons, the proxy RM performs nearly at chance level.
  • At equal KL, RL both improves the result and overoptimizes more slowly than BoN. KL is therefore poorly suited for comparing different methods. For RL, the coefficient α is nearly independent of RM size.
  • A larger policy yields higher scores, but the gap between the proxy score and the gold score remains nearly the same. Only two policy sizes were tested.
  • The KL penalty in PPO raises the proxy score at the same KL. However, the relationship between the gold score and KL does not noticeably improve. The authors caution that this conclusion may depend on hyperparameters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Scaling laws · 30 works

Cite

Cite as Leo Gao, John Schulman and Jacob Hilton. 2022. Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2210.10760 · arXiv 2210.10760

@misc{gao2022scaling,
  title = {{Scaling Laws for Reward Model Overoptimization}},
  author = {Leo Gao, John Schulman and Jacob Hilton},
  year = {2022},
  doi = {10.48550/arxiv.2210.10760},
  eprint = {2210.10760},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2210.10760}
}

References

56 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stuart Armstrong and Sören Mindermann. Occam's razor is insufficient to infer the preferences of irrational agents. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 20… find
  2. Stuart Armstrong et al. General purpose intelligence: arguing the orthogonality thesis. Analysis and Metaphysics, 12(68):1–20, 2013. find
  3. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701, 2021. — [arXiv:2102.06701](https://arxiv.org/abs/2102.06701) find
  4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. find
  5. Note that Bai et al. [2022] scaled the policy size with the RM size, while we hold the policy size constant. find
  6. Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Inc., USA, 1st edition, 2014. ISBN 0199678111. find
  7. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  8. Donald T Campbell. Reforms as experiments. American psychologist, 24(4):409, 1969. find
  9. Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness, 2019. URL https://arxiv.org/abs/1902.06705. — [arXiv:1902.06705](https://arxiv.org/abs/1902.06705) ↗
  10. Joseph Carlsmith. Is power-seeking AI an existential risk? arXiv preprint arXiv:2206.13353, 2022. — [arXiv:2206.13353](https://arxiv.org/abs/2206.13353) find
  11. Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. Adversarial attacks and defences: A survey. arXiv preprint arXiv:1810.00069, 2018. — [arXiv:1810.00069](https://arxiv.org/abs/1810.00069) find
  12. Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge: How to tell if your eyes deceive you, 12 2021. URL https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8. ↗
  13. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. find
  14. Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Rese… find
  15. Ajeya Cotra. Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover, 2022. URL https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to. ↗
  16. Hanjun Dai, Hui Li, Tian Tian, Xin Huang, Lin Wang, Jun Zhu, and Le Song. Adversarial attack on graph structured data. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1115–1… find
  17. Javid Ebrahimi, Daniel Lowd, and Dejing Dou. On adversarial examples for character-level neural machine translation. arXiv preprint arXiv:1806.09030, 2018.
  18. Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. Reinforcement learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417, 2017. — [arXiv:1705.08417](https://arxiv.org/abs/1705.08417) find
  19. Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:1810.00123, 2018. — [arXiv:1810.00123](https://arxiv.org/abs/1810.00123) find
  20. Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fr… — Improving alignment of dialogue agents via targeted human judgements
  21. Adam Gleave and Geoffrey Irving. Uncertainty estimation for language reward models. arXiv preprint arXiv:2203.07472, 2022. — [arXiv:2203.07472](https://arxiv.org/abs/2203.07472) find
  22. Charles Goodhart. Problems of monetary management: the uk experience in papers in monetary economics. Monetary Economics, 1, 1975. find
  23. Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint arXiv:1412.5068, 2014. — [arXiv:1412.5068](https://arxiv.org/abs/1412.5068) find
  24. Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers, 2021. URL https://arxiv.org/abs/2104.13733. ↗
  25. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020.
  26. Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021.
  27. Keith Hoskin. The "awful idea of accountability" : inscribing people into the measurement of objects. Accountability : power, ethos and the technologies of managing / edited by Rolland Munro and Jan Mouritsen, 1996. find
  28. Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820, 2019. — [arXiv:1906.01820](https://arxiv.org/abs/1906.01820) find
  29. Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018. find
  30. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  31. Tomasz Korbak, Ethan Perez, and Christopher L Buckley. Rl with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205.11275, 2022. find
  32. Victoria Krakovna and Ramana Kumar. Classifying specification problems as variants of goodhart’s law, 8 2019. URL https://vkrakovna.wordpress.com/2019/08/19/classifying-specification-problems-as-variants-of-goodharts-law/. ↗
  33. Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of AI ingenuity, 4 2020. URL https://www.deepmind.com/blog/specification-gaming-the-flip-side-of-ai-ingenuity. ↗
  34. Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J. Bentley, Samuel Bernard, Guillaume Beslon, David M. Bryson, Patryk Chrabaszcz, Nick Cheney, Antoine Cully, Stephane Doncieux, Fred C. Dyer, Kai Olav Ellefsen, Robert Feldt, Stephan Fischer, Stephanie For… find
  35. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018. find
  36. Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents, 2017. URL https://arxiv.org/abs/1703.06748. — [arXiv:1703.06748](https://arxiv.org/abs/1703.06748) ↗
  37. David Manheim and Scott Garrabrant. Categorizing variants of goodhart’s law. arXiv preprint arXiv:1803.04585, 2018. — [arXiv:1803.04585](https://arxiv.org/abs/1803.04585) find
  38. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022.
  39. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.
  40. Richard Ngo. The alignment problem from a deep learning perspective. arXiv:2209.00626, 2022. find
  41. Stephen M. Omohundro. The basic ai drives. In Proceedings of the First Conference on Artificial General Intelligence, pages 483–492. IOS Press, 2008. URL http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf. ↗
  42. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Jan Leike, and Ryan Lowe. Training language models to… — Training language models to follow instructions with human feedback
  43. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544, 2022. — [arXiv:2201.03544](https://arxiv.org/abs/2201.03544) find
  44. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. find
  45. Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004.10802, 2020. — [arXiv:2004.10802](https://arxiv.org/abs/2004.10802) find
  46. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking, 2022. URL https://arxiv.org/abs/2209.13085. — [arXiv:2209.13085](https://arxiv.org/abs/2209.13085) ↗
  47. Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015. find
  48. Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. arXiv preprint arXiv:1912.02975, 2019. — [arXiv:1912.02975](https://arxiv.org/abs/1912.02975) find
  49. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. Computing Research Repository, 2020. version 3. find
  50. Cass R Sunstein, Daniel Kahneman, David Schkade, and Ilana Ritov. Predictably incoherent judgments. Stan. L. Rev., 54:1153, 2001. find
  51. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
  52. Jessica Taylor. Quantilizers: A safer alternative to maximizers for limited optimization. In Workshops at the Thirtieth AAAI Conference on Artificial Intelligence, 2016. find
  53. Alex Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal policies tend to seek power. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 23063–23074. Curran Associates,… — [arXiv:1806.07937](https://arxiv.org/abs/1806.07937) find
  54. Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.06893, 2018b. — [arXiv:1804.06893](https://arxiv.org/abs/1804.06893) find
  55. Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 4480–4488, 2016. find
  56. Simon Zhuang and Dylan Hadfield-Menell. Consequences of misaligned AI. Advances in Neural Information Processing Systems, 33:15763–15773, 2020. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/