Elamur Research⌕
☰
home / papers / 2018 / Scalable agent alignment via reward modeling: a research direction

Scalable agent alignment via reward modeling: a research direction

Paper · 2018 · Jan Leike, David Krueger, Tom Everitt et al.

Summary

Reinforcement learning thrives when a task has an explicit score, as in Atari or Go. Most practical jobs, such as handling email, writing software or managing a business, offer no built-in reward, and the user's goal exists only implicitly. The authors call this the agent alignment problem: making an agent act as its user actually wants. This DeepMind position paper presents no new data or experiments. Instead, it proposes reward modeling as a research agenda. A reward model is trained from user feedback, and a separate policy is optimized against that learned reward with RL, separating the question of what to pursue from how to pursue it. For tasks too complex for people to judge directly, the scheme can be applied recursively, with earlier trained agents helping the user evaluate the next one. The paper catalogues obstacles expected during scaling, candidate mitigations, and ways to gain confidence in the resulting agents. Its main contribution is uniting scattered AI safety problems into one coherent plan that can be tested now.

Key points

  • The agenda rests on two assumptions. First, user intentions can be learned with sufficient accuracy. Second, for many tasks, judging outcomes is easier than producing the right behavior. Low-dimensional answers such as yes/no are an exception, but the assumption holds again if an explanation is required.
  • Any solution should meet three desiderata:
    • Scalable: it keeps working even for superhuman agents.
    • Economical: it carries no cost or performance penalty compared with unaligned training.
    • Pragmatic: it is a minimal viable product rather than a guarantee of perfect trust.
  • The paper identifies five routes to trust: testing, interpretability, design choices, formal verification and theoretical guarantees.
  • The scope is limited to one agent aligned to one user. Questions of whose preferences count, how to aggregate them, and ethics are explicitly excluded.
  • The proposed empirical program hides the built-in reward of existing games and simulators. That reward is then used to generate synthetic feedback (Christiano et al., 2017) and to evaluate learned reward models (Ibarz et al., 2018). Authors: Leike, Krueger, Everitt, Martic, Maini, Legg (arXiv:1811.07871).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini and Shane Legg. 2018. Scalable agent alignment via reward modeling: a research direction. arXiv:1811.07871.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1811.07871 · arXiv 1811.07871

@misc{leike2018scalable,
  title = {{Scalable agent alignment via reward modeling: a research direction}},
  author = {Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini and Shane Legg},
  year = {2018},
  doi = {10.48550/arxiv.1811.07871},
  eprint = {1811.07871},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.07871}
}

References

185 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, pp. 1–8, 2004. find
  2. David Abel, John Salvatier, Andreas Stuhlmüller, and Owain Evans. Agent-agnostic human-in-the-loop reinforcement learning. arXiv preprint arXiv:1701.04079, 2017. — [arXiv:1701.04079](https://arxiv.org/abs/1701.04079) find
  3. Bruce Abramson. The Expected-Outcome Model of Two-Player Games. PhD thesis, Columbia University, 1987. find
  4. Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, pp. 22–31, 2017. find
  5. Riad Akrour, Marc Schoenauer, and Michèle Sebag. APRIL: Active preference learning-based reinforcement learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 116–131, 2012. find
  6. Riad Akrour, Marc Schoenauer, Michèle Sebag, and Jean-Christophe Souplet. Programming by feedback. In International Conference on Machine Learning, pp. 1503–1511, 2014. find
  7. Kareem Amin, Nan Jiang, and Satinder Singh. Repeated inverse reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1815–1824, 2017. find
  8. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016.
  9. Jacob Andreas, Anca Dragan, and Dan Klein. Translating neuralese. In Association for Computational Linguistics, pp. 232–242, 2017. find
  10. Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2166–2179, 2018. find
  11. Anonymous. Recurrent experience replay in distributed reinforcement learning. In International Conference on Learning Representations, 2019a. URL https://openreview.net/forum?id=r1lyTjAqYX. Under review. ↗
  12. Anonymous. Systematic generalization: What is required and can it be learned? In International Conference on Learning Representations, 2019b. URL https://openreview.net/forum?id=HkezXnA9YX. Under review. ↗
  13. Anonymous. Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures. In International Conference on Learning Representations, 2019c. URL https://openreview.net/forum? id=B1xhQhRcK7. Under review. ↗
  14. Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In Advances in Neural Information Processing Systems, pp. 5360–5370, 2017.
  15. Arbital. Optimization daemons. https://arbital.com/p/daemons/, 2016. ↗
  16. Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469–483, 2009. find
  17. Stuart Armstrong. Motivated value selection for artificial agents. In AAAI Workshop: AI and Ethics, 2015. find
  18. Stuart Armstrong and Benjamin Levinstein. Low impact artificial intelligences. arXiv preprint arXiv:1705.10720, 2017. — [arXiv:1705.10720](https://arxiv.org/abs/1705.10720) find
  19. Stuart Armstrong and Sören Mindermann. Occam’s razor is insufficient to infer the preferences of irrational agents. In Advances in Neural Information Processing Systems, 2018. find
  20. Sanjeev Arora and Boaz Barak. Computational Complexity: A Modern Approach. Cambridge University Press, 2009. find
  21. Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L Littman. Deep reinforcement learning from policy-dependent human feedback. Technical report, Brown University, 2018. find
  22. Isaac Asimov. Runaround. Astounding Science Fiction, 29(1):94–103, 1942. find
  23. Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420, 2018. — [arXiv:1802.00420](https://arxiv.org/abs/1802.00420) find
  24. Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching YouTube. In Advances in Neural Information Processing Systems, 2018. find
  25. James Babcock, János Kramár, and Roman Yampolskiy. The AGI containment problem. In Artificial General Intelligence, pp. 53–63, 2016. find
  26. Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. Learning to follow language instructions with adversarial reward induction. arXiv preprint arXiv:1806.01946, 2018. find
  27. Mihaly Barasz, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, Patrick LaVictoire, and Eliezer Yudkowsky. Robust cooperation in the prisoner’s dilemma: Program equilibrium via provability logic. arXiv preprint arXiv:1401.5577, 2014. — [arXiv:1401.5577](https://arxiv.org/abs/1401.5577) find
  28. Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXiv:1801.01973, 2018. — [arXiv:1801.01973](https://arxiv.org/abs/1801.01973) find
  29. Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. In International Conference on Learning Representations, 2018. find
  30. Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. arXiv preprint arXiv:1805.08328, 2018. — [arXiv:1805.08328](https://arxiv.org/abs/1805.08328) find
  31. Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Vict… — Relational inductive biases, deep learning, and graph networks
  32. Seth D Baum. Social choice ethics in artificial intelligence. AI & Society, pp. 1–12, 2017. find
  33. Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, 2013.
  34. Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In International Conference on International Conference on Machine Learning, pp. 1613–1622, 2015. find
  35. Nick Bostrom. Ethical issues in advanced artificial intelligence. Science Fiction and Philosophy: From Time Travel to Superintelligence, pp. 277–284, 2003. find
  36. Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014. find
  37. Tom B Brown, Nicholas Carlini, Chiyuan Zhang, Catherine Olsson, Paul Christiano, and Ian Goodfellow. Unrestricted adversarial examples. arXiv preprint arXiv:1809.08352, 2018. — [arXiv:1809.08352](https://arxiv.org/abs/1809.08352) find
  38. Jonathon Cai, Richard Shin, and Dawn Song. Making neural programming architectures generalize via recursion. In International Conference on Learning Representations, 2017. find
  39. J Quiñonero Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning, 2009. find
  40. Paul Christiano. Directions and desiderata for AI alignment. https://ai-alignment.com/ directions-and-desiderata-for-ai-control-b60fca0da8f4, 2017. ↗
  41. Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pp. 4299–4307, 2017. find
  42. Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018. find
  43. Jack Clark and Dario Amodei. Faulty reward functions in the wild. https://blog.openai.com/ faulty-reward-functions/, 2016. ↗
  44. Peter Dayan and Geoffrey Hinton. Feudal reinforcement learning. In Advances in Neural Information Processing Systems, pp. 271–278, 1993. find
  45. Abram Demski and Scott Garrabrant. embedded-agency/, 2018. find
  46. Embedded agency. https://intelligence.org/Daniel Dewey. Learning what to value. In Artificial General Intelligence, pp. 309–314, 2011. ↗
  47. Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017.
  48. Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O’Donoghue, Jonathan Uesato, and Pushmeet Kohli. Training verified learners with learned verifiers. arXiv preprint arXiv:1805.10265, 2018a. — [arXiv:1805.10265](https://arxiv.org/abs/1805.10265) find
  49. Krishnamurthy Dvijotham, Robert Stanforth, Sven Gowal, Timothy Mann, and Pushmeet Kohli. A dual approach to scalable verification of deep networks. In Uncertainty in Artificial Intelligence, 2018b. find
  50. Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Uncertainty in Artificial Intelligence, 2017. find
  51. Layla El Asri, Bilal Piot, Matthieu Geist, Romain Laroche, and Olivier Pietquin. Score-based inverse reinforcement learning. In International Conference on Autonomous Agents & Multiagent Systems, pp. 457–465, 2016. find
  52. Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018. find
  53. Oren Etzioni and Daniel Weld. The first law of robotics (a call to arms). In AAAI, pp. 1042–1047, 1994. find
  54. Owain Evans, Andreas Stuhlmüller, and Noah D Goodman. Learning the preferences of ignorant, inconsistent agents. In AAAI, pp. 323–329, 2016. find
  55. Owain Evans, Andreas Stuhlmüller, Chris Cundy, Ryan Carey, Zachary Kenton, Thomas McGrath, and Andrew Schreiber. Predicting human deliberative judgments with machine learning. Technical report, University of Oxford, 2018. find
  56. Tom Everitt. Towards Safe Artificial General Intelligence. PhD thesis, Australian National University, 2018. find
  57. Tom Everitt and Marcus Hutter. The alignment problem for Bayesian history-based reinforcement learners. Under submission, 2018. find
  58. Tom Everitt, Victoria Krakovna, Laurent Orseau, and Shane Legg. Reinforcement learning with a corrupted reward channel. In International Joint Conference on Artificial Intelligence, pp. 4705–4713, 2017. find
  59. Tom Everitt, Gary Lea, and Marcus Hutter. AGI safety literature review. arXiv preprint arXiv:1805.01109, 2018. — [arXiv:1805.01109](https://arxiv.org/abs/1805.01109) find
  60. Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pp. 49–58, 2016. find
  61. Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-based reinforcement learning: a formal framework and a policy iteration algorithm. Machine Learning, 89(1-2):123–156, 2012. find
  62. Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pp. 1050–1059, 2016. find
  63. Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Advances in Neural Information Processing Systems, pp. 3581–3590, 2017a. find
  64. Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183–1192, 2017b. find
  65. Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437–1480, 2015. find
  66. Scott Garrabrant. Sources of intuitions and data on AGI. https://www.lesswrong.com/posts/ BibDWWeo37pzuZCmL/sources-of-intuitions-and-data-on-agi, 2018. ↗
  67. Scott Garrabrant, Tsvi Benson-Tilsen, Andrew Critch, Nate Soares, and Jessica Taylor. Logical induction. arXiv preprint arXiv:1609.03543, 2016. — [arXiv:1609.03543](https://arxiv.org/abs/1609.03543) find
  68. Justin Gilmer, Ryan P Adams, Ian Goodfellow, David Andersen, and George E Dahl. Motivating the rules of the game for adversarial example research. arXiv preprint arXiv:1807.06732, 2018. — [arXiv:1807.06732](https://arxiv.org/abs/1807.06732) find
  69. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Artificial Intelligence and Statistics, pp. 315–323, 2011. find
  70. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http://arxiv.org/abs/1412. 6572. ↗
  71. Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning. MIT Press, 2016. find
  72. Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In International Conference on Machine Learning, 2013. find
  73. Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pp. 2348–2356, 2011. find
  74. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield,… find
  75. Sam Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding Atari agents. In International Conference on Machine Learning, pp. 1792–1801, 2018. find
  76. Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2625–2633, 2013. find
  77. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321–1330, 2017. find
  78. Dylan Hadfield-Menell, Stuart Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3909–3917, 2016. find
  79. Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The off-switch game. In International Joint Conference on Artificial Intelligence, pp. 220–227, 2017a. find
  80. Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems, pp. 6765–6774, 2017b. find
  81. Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2017. find
  82. José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In International Conference on Machine Learning, pp. 1861–1869, 2015. find
  83. Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Dulac-Arnold, John Agapiou, Joel Z Leibo, and Audrunas Gruslys. Deep Q-learning from demonstrations. In AAAI, 2018. find
  84. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626–6637, 2017. find
  85. Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565–4573, 2016. find
  86. Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. In International Conference on Learning Representaitons, 2018. find
  87. Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02284, 2017. — [arXiv:1702.02284](https://arxiv.org/abs/1702.02284) find
  88. Marcus Hutter. Universal artificial intelligence. Springer, 2005. find
  89. Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in Atari. In Advances in Neural Information Processing Systems, 2018. find
  90. Alex Irpan. Deep reinforcement learning doesn’t work yet. https://www.alexirpan.com/2018/02/ 14/rl-hard.html, 2018. ↗
  91. Geoffrey Irving, Paul Christiano, and Dario Amodei. AI safety via debate. arXiv preprint arXiv:1805.00899, 2018.
  92. Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human… find
  93. Richard Karp. Reducibility among combinatorial problems. In Complexity of Computer Computations, pp. 85–103. Springer, 1972. find
  94. Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. In Computer Aided Verification, pp. 97–117, 2017. find
  95. Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems, pp. 5574–5584, 2017.
  96. Been Kim, Justin Gilmer, Fernanda Viegas, Ulfar Erlingsson, and Martin Wattenberg. TCAV: Relative concept importance testing with linear concept activation vectors. arXiv preprint arXiv:1711.11279, 2017. — [arXiv:1711.11279](https://arxiv.org/abs/1711.11279) find
  97. Diederik P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems, pp. 2575–2583, 2015. find
  98. William Bradley Knox. Learning from human-generated reward. PhD thesis, University of Texas, 2012. find
  99. William Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The TAMER framework. In International Conference on Knowledge Capture, pp. 9–16, 2009. find
  100. Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Carlo planning. In European Conference on Machine Learning, pp. 282–293, 2006. find
  101. Victoria Krakovna, Laurent Orseau, Miljan Martic, and Shane Legg. Measuring and avoiding side effects using relative reachability. arXiv preprint arXiv:1806.01186, 2018. — [arXiv:1806.01186](https://arxiv.org/abs/1806.01186) find
  102. Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. arXiv preprint arXiv:1805.10627, 2018.
  103. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097–1105, 2012.
  104. David Krueger, Jan Leike, Owain Evans, and John Salvatier. Active reinforcement learning: Observing rewards at a cost. In Future of Interactive Learning Machines, NIPS Workshop, 2016. find
  105. Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning, pp. 2801–2809, 2018. find
  106. Isaac Lage, Andrew Slavin Ross, Been Kim, Samuel J Gershman, and Finale Doshi-Velez. Human-in-the-loop interpretability prior. arXiv preprint arXiv:1805.11571, 2018. — [arXiv:1805.11571](https://arxiv.org/abs/1805.11571) find
  107. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402–6413, 2017. find
  108. Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel. Handwritten digit recognition with a back-propagation network. In Advances in Neural Information Processing Systems, pp. 396–404, 1990. find
  109. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436, 2015. find
  110. Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and Machines, 17(4):391–444, 2007. find
  111. Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Belson, David M Bryson, Nick Cheney, Antoine Cully, Stephane Doncieux, Fred C Dyer, Kai Olav Ellefsen, Robert Feldt, Stephan Fischer, Stephanie Forrest, Antoine Frénoy, Christian Gagné… — [arXiv:1803.03453](https://arxiv.org/abs/1803.03453) find
  112. Jan Leike and Marcus Hutter. Bad universal priors and notions of optimality. In Conference on Learning Theory, pp. 1244–1259, 2015. find
  113. Jan Leike, Jessica Taylor, and Benya Fallenstein. A formal solution to the grain of truth problem. In Conference on Uncertainty in Artificial Intelligence, pp. 427–436, 2016. find
  114. Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. AI safety gridworlds. arXiv preprint arXiv:1711.09883, 2017. — [arXiv:1711.09883](https://arxiv.org/abs/1711.09883) find
  115. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. find
  116. James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew E Taylor, and Michael L Littman. Interactive learning from policy-dependent human feedback. In International Conference on Machine Learning, pp. 2285–2294, 2017. find
  117. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
  118. Gary Marcus. Deep learning: A critical appraisal. arXiv preprint arXiv:1801.00631, 2018.
  119. Masterjun. SNES Super Mario World (USA) “arbitrary code execution” in 02:25.19. http://tasvideos.org/2513M.html, 2014. ↗
  120. Cade Metz. In two moves, AlphaGo and Lee Sedol redefined the future. https://www.wired.com/ 2016/03/two-moves-alphago-lee-sedol-redefined-future/, 2016. ↗
  121. Smitha Milli, Dylan Hadfield-Menell, Anca Dragan, and Stuart Russell. Should robots be obedient? In International Joint Conference on Artificial Intelligence, pp. 4754–4760, 2017. find
  122. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hass… find
  123. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pp. 1928–1937, 2016. find
  124. Brendan Mulvaney. Red teams. Marine Corps Gazette, 96(7):63, 2012. find
  125. Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted Boltzmann machines. In International Conference on Machine Learning, pp. 807–814, 2010. find
  126. Andrew Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, pp. 663–670, 2000. find
  127. Andrew Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning, pp. 278–287, 1999. find
  128. Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. https://distill.pub/2018/ building-blocks. ↗
  129. Stephen Omohundro. The basic AI drives. In Artificial General Intelligence, pp. 483–492, 2008. find
  130. Laurent Orseau. Asymptotic non-learnability of universal agents with computable horizon functions. Theoretical Computer Science, 473:149–156, 2013. find
  131. Laurent Orseau and Stuart Armstrong. Safely interruptible agents. In Conference on Uncertainty in Artificial Intelligence, pp. 557–566, 2016. find
  132. Laurent Orseau and Mark Ring. Space-time embedded intelligence. In Artificial General Intelligence, pp. 209–218, 2012. find
  133. Pedro A Ortega, Vishal Maini, and the DeepMind safety team. Building safe artificial intelligence: specification, robustness, and assurance. https://medium.com/@deepmindsafetyresearch/ building-safe-artificial-intelligence-52f5f75058f1, 2018. ↗
  134. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In Association for Computational Linguistics, pp. 311–318, 2002.
  135. Philip Peter. Watching all the movies ever made. watching-all-the-movies-ever-made/, 2014. http://www.justgeek.de/ ↗
  136. Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88–97, 1991. find
  137. Mahendra Prasad. Social choice and the value alignment problem. In Artificial Intelligence Safety and Security, pp. 291–314. Chapman and Hall/CRC, 2018. find
  138. Mark O Riedl and Brent Harrison. Using stories to teach human values to artificial agents. In AAAI Workshop: AI, Ethics, and Society, 2016. find
  139. Mark Ring and Laurent Orseau. Delusion, survival, and intelligent agents. In Artificial General Intelligence, pp. 11–20, 2011. find
  140. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. In Parallel distributed processing: Explorations in the Microstructure of Cognition, volume 1. MIT Press, 1986. find
  141. Stuart Russell. Learning agents for uncertain environments. In Conference on Computational Learning Theory, pp. 101–103, 1998. find
  142. Stuart Russell, Daniel Dewey, and Max Tegmark. Research priorities for robust and beneficial artificial intelligence. AI Magazine, 36(4):105–114, 2015. find
  143. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems, pp. 2234–2242, 2016.
  144. Paul A Samuelson. A note on the pure theory of consumer’s behaviour. Economica, 5(17):61–71, 1938. find
  145. William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. In International Conference on Autonomous Agents and MultiAgent Systems, pp. 2067–2069, 2018. find
  146. Jürgen Schmidhuber. Gödel machines: Self-referential universal problem solvers making provably optimal self-improvements. In Artificial General Intelligence, 2007. find
  147. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889–1897, 2015. find
  148. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. find
  149. Sebastian Schulze and Owain Evans. Active reinforcement learning with monte-carlo tree search. arXiv preprint arXiv:1803.04926, 2018. — [arXiv:1803.04926](https://arxiv.org/abs/1803.04926) find
  150. Burr Settles. Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 6(1):1–114, 2012. find
  151. Alireza Shafaei, Mark Schmidt, and James J Little. Does your model know the digit 6 is not a cat? a less biased evaluation of “outlier” detectors. arXiv preprint arXiv:1809.04729, 2018. — [arXiv:1809.04729](https://arxiv.org/abs/1809.04729) find
  152. Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. On a formal model of safe and scalable self-driving cars. arXiv preprint arXiv:1708.06374, 2017. — [arXiv:1708.06374](https://arxiv.org/abs/1708.06374) find
  153. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcu… find
  154. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Graepel Thore, and Demis Hassabis. Mastering the game of Go withou… find
  155. Nate Soares. The value learning problem. Technical report, Machine Intelligence Research Institute, 2015. find
  156. Nate Soares and Benya Fallenstein. Agent foundations for aligning machine intelligence with human interests: a technical research agenda. In The Technological Singularity, pp. 103–125. Springer, 2017. find
  157. Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. In AAAI Workshop on Artificial Intelligence and Ethics, 2015. find
  158. Thilo Stadelmann, Mohammadreza Amirian, Ismail Arabaci, Marek Arnold, Gilbert François Duivesteijn, Ismail Elezi, Melanie Geiger, Stefan Lörwald, Benjamin Bruno Meier, Katharina Rombach, and Lukas Tuggener. Deep learning in the wild. arXiv preprint arXiv:1807.04950, 2018. — [arXiv:1807.04950](https://arxiv.org/abs/1807.04950) find
  159. Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. In International Conference on Learning Representations, 2017. find
  160. Andreas Stuhlmüller. Factored cognition. https://ought.org/projects/factored-cognition, 2018. ↗
  161. Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT press, 2nd edition, 2018. find
  162. Richard Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181–211, 1999. find
  163. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
  164. Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. Alignment for advanced machine learning systems. Technical report, Machine Intelligence Research Institute, 2016. find
  165. Andrea L Thomaz and Cynthia Breazeal. Teachable robots: Understanding human teaching behavior to build more effective robot learners. Artificial Intelligence, 172(6–7):716–737, 2008. find
  166. Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In Intelligent Robots and Systems, pp. 23–30, 2017. find
  167. Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. In Advances in Neural Information Processing Systems, 2018. find
  168. Hsiao-Yu Fish Tung, Adam W Harley, Liang-Kang Huang, and Katerina Fragkiadaki. Reward learning from narrated demonstrations. In Computer Vision and Pattern Recognition, pp. 7004–7013, 2018. find
  169. Jonathan Uesato, Brendan O’Donoghue, Aaron van den Oord, and Pushmeet Kohli. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning, pp. 5025–5034, 2018. find
  170. Joel Veness, Tor Lattimore, Avishkar Bhoopchand, Agnieszka Grabska-Barwinska, Christopher Mattern, and Peter Toth. Online learning with gated linear networks. arXiv preprint arXiv:1712.01897, 2017. — [arXiv:1712.01897](https://arxiv.org/abs/1712.01897) find
  171. Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. FeUdal Networks for hierarchical reinforcement learning. arXiv preprint arXiv:1703.01161, 2017. find
  172. Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep TAMER: Interactive agent shaping in high-dimensional state spaces. arXiv preprint arXiv:1709.10163, 2017. — [arXiv:1709.10163](https://arxiv.org/abs/1709.10163) find
  173. Christopher Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3-4):279–292, 1992. find
  174. Norbert Wiener. Some moral and technical consequences of automation. Science, 1960. find
  175. Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229–256, 1992. find
  176. Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(1):4945–4990, 2017. find
  177. Eric Wong, Frank Schmidt, Jan Hendrik Metzen, and J Zico Kolter. Scaling provable adversarial defenses. In Advances in Neural Information Processing Systems, 2018. find
  178. Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853, 2015. — [arXiv:1505.00853](https://arxiv.org/abs/1505.00853) find
  179. Roman Yampolskiy. Leakproofing the singularity: Artificial intelligence confinement problem. Journal of Consciousness Studies, 2012. find
  180. Eric Yeh, Melinda Gervasio, Daniel Sanchez, Matthew Crossley, and Karen Myers. Bridging the gap: Converting human advice into imagined examples. Advances in Cognitive Systems, 6:1–20, 2018. find
  181. Eliezer Yudkowsky. Coherent extrapolated volition. Technical report, Singularity Institute for Artificial Intelligence, 2004. find
  182. Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding DQNs. In International Conference on Machine Learning, pp. 1899–1908, 2016. find
  183. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017. find
  184. Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.06893, 2018a. find
  185. Shun Zhang, Edmund H Durfee, and Satinder Singh. Minimax-regret querying on side effects for safe optimality in factored Markov decision processes. In International Joint Conference on Artificial Intelligence, pp. 4867–4873, 2018b. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/leike-2018-scalable-agent-alignment-reward-modeling/