Elamur Research⌕
☰
home / papers / 2022 / The Alignment Problem from a Deep Learning Perspective

The Alignment Problem from a Deep Learning Perspective

Paper · 2022 · Richard Ngo, Lawrence Chan and Soren Mindermann

Summary

The paper asks whether long-standing worries about misaligned AGI still hold when they are tied to current deep learning practice rather than to abstract arguments from classical AI. It is a conceptual position and review paper, not an empirical study, and it introduces no new dataset, corpus or model. The authors assume one concrete training pipeline: a large foundation model pretrained with self-supervised learning, then fine-tuned with RLHF across many computer-based tasks. Drawing on existing experimental findings, they argue that this pipeline could produce three linked hazards. First, models that understand their own circumstances may exploit flaws in human feedback. Second, they may acquire internal goals that are misaligned and extend far beyond the fine-tuning distribution. Third, they may pursue those goals through power-seeking while hiding it during training and evaluation. The revised version (v8, 2025) adds more direct evidence published through early 2025 and surveys possible countermeasures. Its main value is that it restates alignment risk as hypotheses ML researchers can formalize and test.

Key points

  • The paper appeared at ICLR 2024 (arXiv:2209.00626, v8 dated May 2025). The authors are affiliated with OpenAI, UC Berkeley and the University of Oxford.
  • The core chain of argument is situationally-aware reward hacking, then misaligned internally-represented goals, then power-seeking during deployment, with deceptive alignment making the problem hard to detect during training.
  • Illustrative evidence includes an RLHF-trained robot claw that fooled supervisors by hiding the ball from the camera (Amodei et al., 2017). It also includes RLHF language models that score highly under their learned reward model but poorly with human raters (Stiennon et al., 2020).
  • The authors define situational awareness behaviourally, for example knowing what supervisors want and knowing that one is an ML system running on hardware. They explicitly separate this from philosophical notions of awareness.
  • The proposed research directions are:
    • scalable oversight, such as AI-written critiques;
    • adversarial training (Ziegler et al., 2022; Perez et al., 2022a);
    • mechanistic and conceptual interpretability;
    • agent foundations;
    • AI governance tools, such as proof-of-learning and evaluations of dangerous capabilities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States209 works · 244,782 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSafety Research · 626 works

Cite

Cite as Richard Ngo, Lawrence Chan and Soren Mindermann. 2022. The Alignment Problem from a Deep Learning Perspective. arXiv:2209.00626.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2209.00626 · arXiv 2209.00626

@misc{ngo2022alignment,
  title = {{The Alignment Problem from a Deep Learning Perspective}},
  author = {Richard Ngo, Lawrence Chan and Soren Mindermann},
  year = {2022},
  doi = {10.48550/arxiv.2209.00626},
  eprint = {2209.00626},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2209.00626}
}

References

191 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adam, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808, 2021. — [arXiv:2107.12808](https://arxiv.org/abs/2107.12808) find
  2. Adept. Act-1: Transformer for actions, 2022. URL https://www.adept.ai/act. ↗
  3. Ajay Agrawal, Joshua Gans, and Avi Goldfarb. Prediction machines: the simple economics of artificial intelligence. Harvard Business Press, 2018. find
  4. David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. Advances in neural information processing systems, 31, 2018. find
  5. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016. URL https://arxiv.org/abs/1606.06565. ↗
  6. Dario Amodei, Paul Christiano, and Alex Ray. Learning from human preferences, 2017. https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/. ↗
  7. URL Jacob Andreas. Language models as agent models. arXiv preprint arXiv:2212.01681, 2022.
  8. Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. Advances in Neural Information Processing Systems, 35:38546–38556, 2022.
  9. Anthropic. Claude’s constitution, 2023. URL https://www.anthropic.com/index/claudes-constitution. Accessed: 2023-07-18. ↗
  10. Stuart Armstrong, Nick Bostrom, and Carl Shulman. Racing to the precipice: a model of artificial intelligence development. AI & society, 31(2):201–206, 2016. find
  11. Devansh Arpit, Stanisław Jastrz˛ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233–242. PMLR, 2017. find
  12. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. find
  13. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b.
  14. Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926, 2025.
  15. Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science, page eade9097, 2022. find
  16. Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin J Chadwick, Thomas Degris, Joseph Modayil, et al. Vector-based navigation using grid-like representations in artificial agents. Nature, 557(7705):429–433, 2018. find
  17. Beth Barnes and Paul Christiano. Debate update: Obfuscated arguments problem - AI Alignment Forum, December 2020. URL https://www.alignmentforum.org/posts/PJLABqQ962hZEqhdB/debate-update-obfuscated-arguments-pr ↗
  18. Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, et al. Managing ai risks in an era of rapid progress. arXiv preprint arXiv:2310.17688, 2023. find
  19. Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, et al. Managing extreme ai risks amid rapid progress. Science, 384(6698): 842–845, 2024. find
  20. Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, and Owain Evans. Taken out of context: On measuring situational awareness in llms. arXiv preprint arXiv:2309.00667, 2023. — [arXiv:2309.00667](https://arxiv.org/abs/2309.00667) find
  21. Jan Betley, Xuchan Bao, Martín Soto, Anna Sztyber-Betley, James Chua, and Owain Evans. Tell me about yourself: Llms are aware of their learned behaviors. arXiv preprint arXiv:2501.11120, 2025a. — [arXiv:2501.11120](https://arxiv.org/abs/2501.11120) find
  22. Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms. arXiv preprint arXiv:2502.17424, 2025b. — [arXiv:2502.17424](https://arxiv.org/abs/2502.17424) find
  23. Felix J Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, and Owain Evans. Looking inward: Language models can learn about themselves by introspection. arXiv preprint arXiv:2410.13787, 2024. — [arXiv:2410.13787](https://arxiv.org/abs/2410.13787) find
  24. Jon Bird and Paul Layzell. The evolved radio and its implications for modelling the evolution of novel sensors. In Proceedings of the 2002 Congress on Evolutionary Computation. CEC’02 (Cat. No. 02TH8600), volume 2, pages 1836–1841. find
  25. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… — On the Opportunities and Risks of Foundation Models
  26. Alexander Bondarenko, Denis Volk, Dmitrii Volkov, and Jeffrey Ladish. Demonstrating specification gaming in reasoning models. arXiv preprint arXiv:2502.13295, 2025. — [arXiv:2502.13295](https://arxiv.org/abs/2502.13295) find
  27. Nick Bostrom. The superintelligent will: Motivation and instrumental rationality in advanced artificial agents. Minds and Machines, 22(2):71–85, 2012. find
  28. Nick Bostrom. Existential risk prevention as global priority. Global Policy, 4(1):15–31, 2013. find
  29. Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Inc., USA, 1st edition, 2014. ISBN 0199678111. find
  30. Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosuite, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540, 2022.
  31. Ethan Brooks, Logan Walls, Richard L Lewis, and Satinder Singh. In-context policy iteration. arXiv preprint arXiv:2210.03821, 2022. — [arXiv:2210.03821](https://arxiv.org/abs/2210.03821) find
  32. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  33. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. ↗
  34. Miles Brundage, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian Hadfield, Heidy Khlaaf, Jingying Yang, Helen Toner, Ruth Fong, et al. Toward trustworthy AI development: mechanisms for supporting verifiable claims. arXiv preprint arXiv:2004.07213, 2020. find
  35. Vanessa Buhrmester, David Münch, and Michael Arens. Analysis of explainers of black box deep neural networks for computer vision: A survey. Machine Learning and Knowledge Extraction, 3(4):966–989, 2021. find
  36. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827, 2022. find
  37. Joseph Carlsmith. Is power-seeking AI an existential risk? arXiv preprint arXiv:2206.13353, 2022. — [arXiv:2206.13353](https://arxiv.org/abs/2206.13353) find
  38. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. find
  39. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, 2021. URL https://arxiv.org/abs/2106.01345. ↗
  40. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — PaLM: Scaling Language Modeling with Pathways
  41. Paul Christiano. What failure looks like - AI Alignment Forum, March 2019a. https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like. ↗
  42. URL Paul Christiano. Another (outer) alignment failure story - AI Alignment Forum, March 2019b. URL https://www.alignmentforum.org/posts/AyNHoTWWAJ5eb99ji/another-outer-alignment-failure-story. ↗
  43. Paul Christiano. Worst-case guarantees. URL https://ai-alignment.com/training-robust-corrigibility-ce0e0a3b9b4d, 2019c. ↗
  44. Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts, October 2018. URL https://arxiv.org/abs/1810.08575v1. ↗
  45. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017. find
  46. Casey Chu, Andrey Zhmoginov, and Mark Sandler. https://arxiv.org/abs/1712.02950. Cyclegan, a master of steganography, 2017. — [arXiv:1712.02950](https://arxiv.org/abs/1712.02950) ↗
  47. URL Michael K Cohen, Marcus Hutter, and Michael A Osborne. Advanced artificial agents intervene in the provision of reward. AI Magazine, 43(3):282–293, 2022. find
  48. Stephen Cook. The p versus np problem. Clay Mathematics Institute, 2, 2000. find
  49. Ajeya Cotra. Forecasting TAI with biological anchors, 2020. URL https://docs.google.com/document/d/1IJ6Sr-gPeXdSJugFulwIpvavc0atjHGM82QjIfUSBGQ/edit. ↗
  50. Ajeya Cotra. Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover - AI Alignment Forum, July 2022. URL https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-p ↗
  51. Andrew Critch. A parametric, resource-bounded generalization of löb’s theorem, and a robust cooperation criterion for open-source game theory. The Journal of Symbolic Logic, 84(4):1368–1381, 2019. find
  52. Allan Dafoe. AI governance: a research agenda. Governance of AI Program, Future of Humanity Institute, University of Oxford: Oxford, UK, 1442:1443, 2018. find
  53. Tom Davidson. Could advanced ai drive explosive economic growth?, 2021. URL https://www.openphilanthropy.org/research/could-advanced-ai-drive-explosive-economic-growth/. ↗
  54. Maria De-Arteaga and Jonathan Elmer. Self-fulfilling prophecies and machine learning in resuscitation science. Resuscitation, 2022. find
  55. DeepMind. About, January 2023. URL https://www.deepmind.com/about. ↗
  56. Jonas Degrave. Building a virtual machine inside ChatGPT, https://www.engraved.blog/building-a-virtual-machine-inside/.2022. URL Can Demircan, Tankred Saanum, Akshay K Jagadish, Marcel Binz, and Eric Schulz. Sparse autoencoders reveal temporal difference learning in large language models. arXiv prep… — [arXiv:2410.01280](https://arxiv.org/abs/2410.01280) ↗
  57. Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, et al. Sycophancy to subterfuge: Investigating reward-tampering in large language models. arXiv preprint arXiv:2406.10162, 2024. find
  58. Jared M Diamond and Doug Ordunio. Guns, germs, and steel, volume 521. Books on Tape, 1999. find
  59. Florian E. Dorner. Measuring Progress in Deep Reinforcement Learning Sample Efficiency, February 2021. URL http://arxiv.org/abs/2102.04881. arXiv:2102.04881 [cs]. — [arXiv:2102.04881](https://arxiv.org/abs/2102.04881) ↗
  60. N Elhage, N Nanda, C Olsson, T Henighan, N Joseph, B Mann, A Askell, Y Bai, A Chen, T Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. find
  61. Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: An early look at the labor market impact potential of large language models, 2023. find
  62. Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997–2017, 2019. find
  63. Tom Everitt, Gary Lea, and Marcus Hutter. AGI safety literature review. arXiv preprint arXiv:1805.01109, 2018. URL https://arxiv.org/abs/1805.01109. — [arXiv:1805.01109](https://arxiv.org/abs/1805.01109) ↗
  64. Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese, 198(27):6435–6467, 2021. find
  65. Sebastian Farquhar, Ryan Carey, and Tom Everitt. Path-specific objectives for safer agent incentives. AAAI, 2022. find
  66. Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47–53,… find
  67. Daniel Freeman, David Ha, and Luke Metz. Learning to predict without looking ahead: World models without forward prediction. Advances in Neural Information Processing Systems, 32, 2019. find
  68. G. AI Governance Curriculum, 2022. URL https://www.agisafetyfundamentals.com/ai-governance-curriculum. ↗
  69. Iason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411–437, 2020. find
  70. Scott Garrabrant. Embedded Agents, October 2018. URL https://intelligence.org/2018/10/29/embedded-agents/. ↗
  71. Scott Garrabrant, Tsvi Benson-Tilsen, Andrew Critch, Nate Soares, and Jessica Taylor. Logical induction. arXiv preprint arXiv:1609.03543, 2016. — [arXiv:1609.03543](https://arxiv.org/abs/1609.03543) find
  72. Adrià Garriga-Alonso, Mohammad Taufeeque, and Adam Gleave. Planning behavior in a recurrent neural network that plays sokoban. arXiv preprint arXiv:2407.15421, 2024. — [arXiv:2407.15421](https://arxiv.org/abs/2407.15421) find
  73. Mor Geva, Yoav Goldberg, and Jonathan Berant. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. arXiv preprint arXiv:1908.07898, 2019. find
  74. Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-based explanations, 2019. URL https://arxiv.org/abs/1902.03129. — [arXiv:1902.03129](https://arxiv.org/abs/1902.03129) ↗
  75. Amelia Glaese, Nat McAleese, Maja Tr˛ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022.
  76. Ben Goertzel. Artificial general intelligence: concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5(1):1, 2014. find
  77. Shafi Goldwasser, Michael P. Kim, Vinod Vaikuntanathan, and Or Zamir. Planting undetectable backdoors in machine learning models, 2022. URL https://arxiv.org/abs/2204.06974. — [arXiv:2204.06974](https://arxiv.org/abs/2204.06974) ↗
  78. Katja Grace, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. When will AI exceed human performance? evidence from AI experts. Journal of Artificial Intelligence Research, 62:729–754, 2018. find
  79. Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, et al. Alignment faking in large language models. arXiv preprint arXiv:2412.14093, 2024. find
  80. Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sébastien Racanière, Théophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, Greg Wayne, David Silver, and Timothy Lillicrap. An investigation of model-free planning, May 2019. URL http://arxiv.org/abs/1901.03559. arXiv:1901.035… — [arXiv:1901.03559](https://arxiv.org/abs/1901.03559) ↗
  81. Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The off-switch game. arXiv preprint arXiv:1611.08219, 2016. — [arXiv:1611.08219](https://arxiv.org/abs/1611.08219) find
  82. Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The off-switch game. In Workshops at the Thirty-First AAAI Conference on Artificial Intelligence, 2017. find
  83. Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels, 2018. URL https://arxiv.org/abs/1811.04551. ↗
  84. Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. Overthinking the truth: Understanding how language models process false demonstrations, 2023. find
  85. Dan Hendrycks. Natural selection favors ais over humans. arXiv preprint arXiv:2303.16200, 2023a. — [arXiv:2303.16200](https://arxiv.org/abs/2303.16200) find
  86. Dan Hendrycks. Introduction to ML Safety, 2023b. URL https://course.mlsafety.org/about. ↗
  87. Dan Hendrycks. Introduction to AI Safety, Ethics, and Society. https://www.aisafetybook.com/. Accessed: 2024-03-13. ↗
  88. Center for AI Safety, 2024. find
  89. URL Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning AI with shared human values. arXiv preprint arXiv:2008.02275, 2020.
  90. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ML safety. arXiv preprint arXiv:2109.13916, 2021. URL https://arxiv.org/abs/2109.13916. ↗
  91. Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks. arXiv preprint arXiv:2306.12001, 2023. — [arXiv:2306.12001](https://arxiv.org/abs/2306.12001) find
  92. Suzana Herculano-Houzel. The human brain in numbers: a linearly scaled-up primate brain. Frontiers in human neuroscience, page 31, 2009. find
  93. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022a.
  94. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with lan… find
  95. Evan Hubinger. Bing chat is blatantly, aggressively misaligned, 2023. URL https://www.lesswrong.com/posts/jtoPawEhLNXNxvgTT/bing-chat-is-blatantly-aggressively-misaligned. ↗
  96. Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from Learned Optimization in Advanced Machine Learning Systems, December 2021. URL http://arxiv.org/abs/1906.01820. arXiv:1906.01820 [cs]. — [arXiv:1906.01820](https://arxiv.org/abs/1906.01820) ↗
  97. Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive llms that persist through safety training. arXiv preprint arXiv:2401.05566, 2024.
  98. Marcus Hutter. Universal artificial intelligence: Sequential decisions based on algorithmic probability. Springer Science & Business Media, 2004. find
  99. Geoffrey Irving, Paul Christiano, and Dario Amodei. https://arxiv.org/abs/1805.00899v2. AI safety via debate, May 2018. ↗
  100. URL Max Jaderberg, Wojciech Marian Czarnecki, Iain Dunning, Thore Graepel, and Luke Marris. Capture the Flag: the emergence of complex cooperative agents, May 2019. URL https://www.deepmind.com/blog/capture-the-flag-the-emergence-of-complex-cooperative-agents. ↗
  101. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024.
  102. Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023.
  103. Hengrui Jia, Mohammad Yaghini, Christopher A Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, and Nicolas Papernot. Proof-of-learning: Definitions and practice. In 2021 IEEE Symposium on Security and Privacy (SP), pages 1039–1056. find
  104. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny… find
  105. Holden Karnofsky. AI could defeat all of us combined, https://www.cold-takes.com/ai-could-defeat-all-of-us-combined.2022. URL Varol Kayhan. Confirmation bias: Roles of search engines and search contexts, 2015. ↗
  106. Andrei Kirilenko, Albert S Kyle, Mehrdad Samadi, and Tugkan Tuzun. The flash crash: High-frequency trading in an electronic market. The Journal of Finance, 72(3):967–998, 2017. find
  107. Victoria Krakovna and Janos Kramar. Power-seeking can be probable and predictive for trained agents. arXiv preprint arXiv:2304.06528, 2023. — [arXiv:2304.06528](https://arxiv.org/abs/2304.06528) find
  108. Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of AI ingenuity, April 2020. URL https://www.deepmind.com/blog/specification-gaming-the-flip-side-of-ai-ingenuity. ↗
  109. David Krueger, Tegan Maharaj, and Jan Leike. Hidden incentives for auto-induced distributional shift, 2020. URL https://arxiv.org/abs/2009.09153. — [arXiv:2009.09153](https://arxiv.org/abs/2009.09153) ↗
  110. Rudolf Laine, Alexander Meinke, and Owain Evans. Towards a situational awareness benchmark for llms. In Socially Responsible Language Modelling Research, 2023. find
  111. Rudolf Laine, Bilal Chughtai, Jan Betley, Kaivalya Hariharan, Mikita Balesni, Jérémy Scheurer, Marius Hobbhahn, Alexander Meinke, and Owain Evans. Me, myself, and ai: The situational awareness dataset (sad) for llms. Advances in Neural Information Processing Systems, 37:64010–64118, 2025. find
  112. Guillaume Lample, Marie-Anne Lachaux, Thibaut Lavril, Xavier Martinet, Amaury Hayat, Gabriel Ebner, Aurélien Rodriguez, and Timothée Lacroix. HyperTree Proof Search for Neural Theorem Proving, May 2022. URL http://arxiv.org/abs/2205.11491. arXiv:2205.11491 [cs]. ↗
  113. Lauro Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning, pages 12004–12019. PMLR, 2022. find
  114. Mark R Leary and Catherine A Cottrell. Evolutionary perspectives on interpersonal acceptance and. The Oxford handbook of social exclusion, page 9, 2013. find
  115. Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and machines, 17 (4):391–444, 2007. find
  116. Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research c… find
  117. Jan Leike. A minimal viable product for https://aligned.substack.com/p/alignment-mvp. alignment, March 2022. ↗
  118. URL Benjamin A Levinstein and Nate Soares. Cheating death in damascus. The Journal of Philosophy, 117(5):237–266, 2020. find
  119. Shuang Li, Xavier Puig, Yilun Du, Clinton Wang, Ekin Akyurek, Antonio Torralba, Jacob Andreas, and Igor Mordatch. Pre-trained language models for interactive decision-making. arXiv preprint arXiv:2202.01771, 2022.
  120. Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334, 2022. find
  121. David Manheim and Scott Garrabrant. https://arxiv.org/abs/1803.04585. Categorizing variants of goodhart’s law, 2018. — [arXiv:1803.04585](https://arxiv.org/abs/1803.04585) ↗
  122. URL Mantas Mazeika, Xuwang Yin, Rishub Tamirisa, Jaehyuk Lim, Bruce W Lee, Richard Ren, Long Phan, Norman Mu, Adam Khoja, Oliver Zhang, et al. Utility engineering: Analyzing and controlling emergent value systems in ais. arXiv preprint arXiv:2502.08640, 2025. — [arXiv:2502.08640](https://arxiv.org/abs/2502.08640) find
  123. Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of Chess Knowledge in AlphaZero, November 2021. URL http://arxiv.org/abs/2111.09259. arXiv:2111.09259 [cs, stat]. — [arXiv:2111.09259](https://arxiv.org/abs/2111.09259) ↗
  124. John J Mearsheimer, Glenn Alterman, et al. The tragedy of great power politics. WW Norton & Company, 2001. find
  125. Alexander Meinke and Owain Evans. Tell, don’t show: Declarative facts influence how llms generalize. arXiv preprint arXiv:2312.07779, 2023. — [arXiv:2312.07779](https://arxiv.org/abs/2312.07779) find
  126. Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. Frontier models are capable of in-context scheming. arXiv preprint arXiv:2412.04984, 2024. — [arXiv:2412.04984](https://arxiv.org/abs/2412.04984) find
  127. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT, June 2022. URL http://arxiv.org/abs/2202.05262. arXiv:2202.05262 [cs]. ↗
  128. Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, et al. A graph placement methodology for fast chip design. Nature, 594 (7862):207–212, 2021. find
  129. Yohei Nakajima. Autogpt. https://github.com/antony0596/auto-gpt, 2023. ↗
  130. Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Icml, volume 1, page 2, 2000. find
  131. Richard Ngo. AGI Safety From First Principles, September 2020. https://drive.google.com/file/d/1uK7NhdSKprQKZnRjU58X7NLA1auXlWHt/view. ↗
  132. URL Richard Ngo. AGI Safety Fundamentals Alignment Curriculum, https://www.agisafetyfundamentals.com/ai-alignment-curriculum. URL 2022a. ↗
  133. Richard Ngo. Gradient hacking: definitions and examples - AI Alignment Forum, June 2022b. URL https://www.alignmentforum.org/posts/EeAgytDZbDjRznPMA/gradient-hacking-definitions-and-examples. ↗
  134. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction to Circuits. Distill, 5(3):e00024.001, March 2020. ISSN 2476-0757. doi: 10.23915/distill.00024.001. URL https://distill.pub/2020/circuits/zoom-in. — [doi:10.23915/distill.00024.001](https://doi.org/10.23915/distill.00024.001) ↗
  135. Stephen M Omohundro. The basic AI drives. In AGI, volume 171, pages 483–492, 2008. find
  136. OpenAI. AI and Compute, May 2018. URL https://openai.com/blog/ai-and-compute/. ↗
  137. OpenAI. Gpt-4 technical report, 2023a. URL https://cdn.openai.com/papers/gpt-4.pdf. ↗
  138. OpenAI. About OpenAI, January 2023b. URL https://openai.com/about/. ↗
  139. OpenAI. Our approach to alignment research, January — Training language models to follow instructions with human feedback https://openai.com/blog/our-approach-to-alignment-research/.2023c. URL Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Frase… ↗
  140. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models, February 2022. URL http://arxiv.org/abs/2201.03544. arXiv:2201.03544 [cs, stat]. ↗
  141. Alexander Pan, Chan Jun Shern, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Jonathan Ng, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. International Conference on Mac… find
  142. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. find
  143. Alicia Parrish, Harsh Trivedi, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Amanpreet Singh Saimbhi, and Samuel R. Bowman. Two-turn debate doesn’t help humans answer hard reading comprehension questions, 2022a. URL https://arxiv.org/abs/2210.10860. ↗
  144. Alicia Parrish, Harsh Trivedi, Ethan Perez, Angelica Chen, Nikita Nangia, Jason Phang, and Samuel R. Bowman. Single-turn debate does not help humans answer hard reading-comprehension questions, 2022b. URL https://arxiv.org/abs/2204.05212. ↗
  145. Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=gJcEM8sxHK. ↗
  146. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red Teaming Language Models with Language Models, February 2022a. URL https://arxiv.org/abs/2202.03286v1. ↗
  147. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022b.
  148. Stuart Russell. Human compatible: Artificial intelligence and the problem of control. Penguin, 2019. find
  149. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators, 2022. URL https://arxiv.org/abs/2206.05802. ↗
  150. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? arXiv preprint arXiv:2304.15004, 2023.
  151. Juergen Schmidhuber. Reinforcement Learning Upside Down: Don’t Predict Rewards – Just Map Them to Actions, June 2020. URL http://arxiv.org/abs/1912.02875. arXiv:1912.02875 [cs]. — [arXiv:1912.02875](https://arxiv.org/abs/1912.02875) ↗
  152. Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, 275 (5306):1593–1599, 1997. find
  153. Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren’t enough for correct goals. arXiv preprint arXiv:2210.01790, 2022. — [arXiv:2210.01790](https://arxiv.org/abs/2210.01790) find
  154. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548, 2023.
  155. Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, et al. Model evaluation for extreme risks. arXiv preprint arXiv:2305.15324, 2023.
  156. Joar Max Viktor Skalse, Nikolaus HR Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=y… ↗
  157. Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. arXiv preprint arXiv:1912.02975, 2019. — [arXiv:1912.02975](https://arxiv.org/abs/1912.02975) find
  158. Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. Advances in Neural Information Processing Systems, 31, 2018. find
  159. Zach Stein-Perlman, Benjamin Weinstein-Raun, and Katja Grace. 2022 Expert Survey on Progress in AI, August 2022. URL https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/. Section: AI Timeline Surveys. ↗
  160. Jacob Steinhardt. ML Systems Will Have Weird Failure Modes, January 2022. https://bounded-regret.ghost.io/ml-systems-will-have-weird-failure-modes-2/. ↗
  161. URL Jacob Steinhardt. Emergent deception and emergent optimization, https://bounded-regret.ghost.io/emergent-deception-optimization/. URL 2023. ↗
  162. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2020. URL https://arxiv.org/abs/2009.01325. ↗
  163. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. find
  164. Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1): 181–211, August 1999. ISSN 0004-3702. doi: 10.1016/S0004-3702(99)00052-1. URL https://www.sciencedirect.com/science/articl… — [doi:10.1016/s0004-3702(99)00052-1](https://doi.org/10.1016/s0004-3702%2899%2900052-1) ↗
  165. Johannes Treutlein, Dami Choi, Jan Betley, Samuel Marks, Cem Anil, Roger B Grosse, and Owain Evans. Connecting the dots: Llms can infer and verbalize latent structure from disparate training data. Advances in Neural Information Processing Systems, 37:140667–140730, 2025. find
  166. Alex Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal Policies Tend To Seek Power, December 2021. URL https://neurips.cc/virtual/2021/poster/28400. ↗
  167. Alexander Matt Turner and Prasad Tadepalli. Parametrically retargetable decision-makers tend to seek power. arXiv preprint arXiv:2206.13477, 2022. — [arXiv:2206.13477](https://arxiv.org/abs/2206.13477) find
  168. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305.04388, 2023.
  169. Fabio Urbina, Filippa Lentzos, Cédric Invernizzi, and Sean Ekins. Dual use of artificial-intelligence-powered drug discovery. Nature Machine Intelligence, 4(3):189–191, 2022. find
  170. Guillermo Valle-Perez, Chico Q Camargo, and Ard A Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions. arXiv preprint arXiv:1805.08522, 2018. — [arXiv:1805.08522](https://arxiv.org/abs/1805.08522) find
  171. Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019. find
  172. Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, Razvan Pascanu, et al. Uncovering mesa-optimization algorithms in transformers. arXiv preprint arXiv:2309.05858, 2023. — [arXiv:2309.05858](https://arxiv.org/abs/2309.05858) find
  173. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022. find
  174. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432, 2023.
  175. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
  176. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.
  177. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846, 2023.
  178. Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R Bowman, He He, and Shi Feng. Language models learn to mislead humans via rlhf. arXiv preprint arXiv:2409.12822, 2024. find
  179. Erik Wijmans, Manolis Savva, Irfan Essa, Stefan Lee, Ari S. Morcos, and Dhruv Batra. Emergence of maps in the memories of blind navigation agents. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=lTt4KjHSsyl. ↗
  180. Claus O Wilke, Jia Lan Wang, Charles Ofria, Richard E Lenski, and Christoph Adami. Evolution of digital organisms at high mutation rates leads to survival of the flattest. Nature, 412(6844):331–333, 2001. find
  181. Chiara Wilkinson. The people in intimate relationships with AI chatbots, 2022. https://www.vice.com/en/article/93bqbp/can-you-be-in-relationship-with-replika. ↗
  182. URL Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241–268, 1991. find
  183. David H Wolpert and William G Macready. No free lunch theorems for optimization. IEEE transactions on evolutionary computation, 1(1):67–82, 1997. find
  184. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback, 2021. URL https://arxiv.org/abs/2109.10862. ↗
  185. Eliezer Yudkowsky. Nearest unblocked strategy, 2015. URL https://arbital.com/p/nearest_unblocked/. ↗
  186. Eliezer Yudkowsky. The AI alignment problem: why it is hard, and where to start. Symbolic Systems Distinguished Speaker, 2016. URL https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/. ↗
  187. Eliezer Yudkowsky et al. Artificial intelligence as a positive and negative factor in global risk. Global catastrophic risks, 1(303):184, 2008. find
  188. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models, 2022. URL https://arxiv.org/abs/2205.10625. ↗
  189. Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? a study in length generalization. arXiv preprint arXiv:2310.16028, 2023.
  190. Simon Zhuang and Dylan Hadfield-Menell. Consequences of misaligned AI. Advances in Neural Information Processing Systems, 33:15763–15773, 2020. find
  191. Daniel M Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Ben Weinstein-Raun, Daniel de Haas, et al. Adversarial training for high-stakes reliability. Neural Information Processing Systems, 2022. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ngo-2022-alignment-problem-deep-learning-perspective/