📄 AI Alignment: A Comprehensive Survey

Paper · 2023 · Jiaming Ji, Tianyi Qiu, Boyuan Chen et al.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

🏛️ Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations

🧭 Fields, topics and concepts

Cite

Cite as Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Lukas Vierling, Donghai Hong, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Juntao Dai, Xuehai Pan, Kwan Yee Ng, Aidan O'Gara, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo and Wen Gao. 2023. AI Alignment: A Comprehensive Survey. arXiv:2310.19852.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.19852 · arXiv 2310.19852

@misc{ji2023ai,
  title = {{AI Alignment: A Comprehensive Survey}},
  author = {Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Lukas Vierling, Donghai Hong, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Juntao Dai, Xuehai Pan, Kwan Yee Ng, Aidan O'Gara, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo and Wen Gao},
  year = {2023},
  doi = {10.48550/arxiv.2310.19852},
  eprint = {2310.19852},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.19852}
}

References

518 references, 79 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [321] Lewis Ho, Joslyn Barnhart, Robert Trager, Yoshua Bengio, Miles Brundage, Allison Carnegie, Rumman Chowdhury, Allan Dafoe, Gillian Hadfield, Margaret Levi, et al. 2023. International institutions for advanced ai. arXiv preprint arXiv:2307.04699. — [arXiv:2307.04699](https://arxiv.org/abs/2307.04699) find
2. [322] Marius Hobbhahn. 2022. Eliciting latent knowledge (elk) - distillation/summary. https://www.alignm entforum.org/posts/rxoBY9CMkqDsHt25t/eliciting-latent-knowledge-elk-disti llation-summary. ↗
3. [323] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Process… find
4. [324] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations.
5. [325] Andreas Holzinger. 2016. Interactive machine learning for health informatics: when do we need the human-in-the-loop? Brain Informatics, 3(2):119–131. find
6. [326] Andreas Holzinger, Chris Biemann, Constantinos S Pattichis, and Douglas B Kell. 2017. What do we need to build explainable ai systems for the medical domain? arXiv preprint arXiv:1712.09923. — [arXiv:1712.09923](https://arxiv.org/abs/1712.09923) find
7. [327] Joey Hong, Kush Bhatia, and Anca Dragan. 2022. On the sensitivity of reward inference to misspecified human models. In The Eleventh International Conference on Learning Representations. find
8. [328] Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. Q2:: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Proces… find
9. [329] Jeremy Howard. 2023. Ai safety and the age of dislightenment. https://www.fast.ai/posts/20 23-11-07-dislightenment.html. ↗
10. [330] Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, Noam Brown, and Jakob Foerster. 2021. Off-belief learning. In International Conference on Machine Learning, pages 4369–4379. PMLR. find
11. [331] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. 2020. “other-play” for zero-shot coordination. In International Conference on Machine Learning, pages 4399–4410. PMLR. find
12. [332] Xuming Hu, Junzhe Chen, Xiaochuan Li, Yufei Guo, Lijie Wen, Philip S. Yu, and Zhijiang Guo. 2024. Do large language models know about facts? In The Twelfth International Conference on Learning Representations. find
13. [333] Sandy H. Huang, Nicolas Papernot, Ian J. Goodfellow, Yan Duan, and Pieter Abbeel. 2017. Adversarial attacks on neural network policies. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net. find
14. [334] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2023. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, pages 1769–1782. PMLR. find
15. [335] Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. 2024. Catastrophic jailbreak of open-source LLMs via exploiting generation. In The Twelfth International Conference on Learning Representations.
16. [336] Evan Hubinger. 2020. arXiv:2012.07532. — [arXiv:2012.07532](https://arxiv.org/abs/2012.07532) find
17. An overview of 11 proposals for building safe advanced ai. [337] Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. 2024. Sleeper agents: Training deceptive llms that persist through safety training. a… find
18. [338] Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. 2019a. Deceptive alignment. https://www.alignmentforum.org/posts/zthDPAjh9w6Ytbeks/deceptive-a lignment. ↗
19. [339] Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. 2019b. The inner alignment problem. https://www.alignmentforum.org/posts/pL56xPoniLvtMDQ4J/the-i nner-alignment-problem. arXiv preprint [340] Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse,… — [arXiv:1906.01820](https://arxiv.org/abs/1906.01820) ↗
20. [341] Drew Arad Hudson and Christopher D. Manning. 2018. Compositional attention networks for machine reasoning. In International Conference on Learning Representations. find
21. [342] Eyke Hüllermeier, Johannes Fürnkranz, Weiwei Cheng, and Klaus Brinker. 2008. Label ranking by learning pairwise preferences. Artificial Intelligence, 172(16-17):1897–1916. find
22. [343] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. 2017. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1–35. find
23. [344] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. 2018. Reward learning from human preferences and demonstrations in atari. Advances in Neural Information Processing Systems, 31. find
24. [345] Geoffrey Irving, Paul Christiano, and Dario Amodei. 2018. arXiv:1805.00899. find
25. [346] Charles Isbell, Christian R Shelton, Michael Kearns, Satinder Singh, and Peter Stone. 2001. A social reinforcement learning agent. In Proceedings of the fifth international conference on Autonomous agents, pages 377–384. find
26. [347] Jacob Steinhardt. 2023. Emergent deception and emergent optimization. https://bounded-regret.ghost.io/emergent-deception-optimization. ↗
27. [348] Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. 2019. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science, 364(6443)… find
28. [349] Ashesh Jain, Brian Wojcik, Thorsten Joachims, and Ashutosh Saxena. 2013. Learning trajectory preferences for manipulators via iterative improvement. Advances in Neural Information Processing Systems, 26. find
29. [350] Hong Jun Jeon, Smitha Milli, and Anca Dragan. 2020. Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 33:4415–4426. find
30. [351] Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. 2024a. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint arXiv:2402.02416. find
31. [352] Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2024b. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36.
32. [353] Jiaming Ji, Kaile Wang, Tianyi Qiu, Boyuan Chen, JiayiZhou ChangyeLi HantaoLou YaodongYang, and PKU-Alignment Team. 2024c. Language models resist alignment. arXiv preprint arXiv:2406.06144. — [arXiv:2406.06144](https://arxiv.org/abs/2406.06144) find
33. [354] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys (CSUR), 55(12):1–38.
34. [355] Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031.
35. [356] Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. 2021. Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34:1884–1897. find
36. [357] Zhijing Jin, Sydney Levine, Fernando Gonzalez Adauto, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Josh Tenenbaum, and Bernhard Schölkopf. 2022a. When to make exceptions: Exploring language models as accounts of human moral judgment. Advances in Neural Information Processing Syste… find
37. [358] Zhijing Jin, Sydney Levine, Fernando Gonzalez Adauto, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Josh Tenenbaum, and Bernhard Schölkopf. 2022b. When to make exceptions: Exploring language models as accounts of human moral judgment. In Advances in Neural Information Processing Sy… find
38. [359] Jonas DeGrave. 2022. Building a virtual machine inside chatgpt. https://www.engraved.blog/bu ilding-a-virtual-machine-inside. ↗
39. Ai safety via debate. arXiv preprint [360] Erik Jones, Anca D. Dragan, Aditi Raghunathan, and Jacob Steinhardt. 2023. Automatically auditing large language models via discrete optimization. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202… find
40. [361] Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. 2022. Linear connectivity reveals generalization strategies. In The Eleventh International Conference on Learning Representations. find
41. [362] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237–285. find
42. [363] Dimitris Kalimeris, Smriti Bhagat, Shankar Kalyanaraman, and Udi Weinsberg. 2021. Preference amplification in recommender systems. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 805–815. find
43. [364] Josh Kalin, Matthew Ciolino, David Noever, and Gerry Dozier. 2020. Black box to white box: Discover model characteristics based on strategic probing. In 2020 Third International Conference on Artificial Intelligence for Industries (AI4I), pages 60–63. IEEE. find
44. [365] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
45. [366] Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078.
46. [367] Michael Kearns and Aaron Roth. 2019. The ethical algorithm: The science of socially aware algorithm design. Oxford University Press. find
47. [368] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
48. [369] Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021. Alignment of language agents. arXiv preprint arXiv:2103.14659. find
49. [370] Zachary Kenton, Rohin Shah, David Lindner, Vikrant Varma, Victoria Krakovna, Mary Phuong, Ramana Kumar, and Elliot Catt. 2022. Threat model literature review. https://www.alignmentforum.org/p osts/wnnkD6P2k2TfHnNmt/threat-model-literature-review. ↗
50. [371] Zachary Kenton, Noah Y Siegel, János Kramár, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah D Goodman, et al. 2024. On scalable oversight with weak llms judging strong llms. arXiv preprint arXiv:2407.04622. — [arXiv:2407.04622](https://arxiv.org/abs/2407.04622) find
51. [372] Ben Kenward and Thomas Sinclair. 2021. Machine morality, moral progress, and the looming environmental disaster. Cognitive Computation and Systems, 3(2):83–90. find
52. [373] Cameron F Kerry, Joshua P Meltzer, Andrea Renda, Alex Engler, and Rosanna Fanni. 2021. Strengthening international cooperation on ai, progress report. https://www.brookings.edu/articles/strengt hening-international-cooperation-on-ai. ↗
53. [374] Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with more persuasive llms leads to more truthful answers. arXiv preprint arXiv:2402.06782. find
54. [375] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668–2677. PMLR. find
55. [376] Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. 2023. Preference Transformer: Modeling Human Preferences using Transformers for RL. In International Conference on Learning Representations (ICLR). find
56. [377] Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2024. Language models can solve computer tasks. Advances in Neural Information Processing Systems, 36.
57. [378] Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. 2021. Reward identification in inverse reinforcement learning. In International Conference on Machine Learning, pages 5496–5505. PMLR. find
58. [379] Pieter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne. 2018. Learning how to explain neural networks: Patternnet and patternattribution. In International Conference on Learning Representations. find
59. [380] Megan Kinniment, Lucas Jun Koba Sato, Haoxing Du, Brian Goodrich, Max Hasin, Lawrence Chan, Luke Harold Miles, Tao R Lin, Hjalmar Wijk, Joel Burget, Aaron Ho, Elizabeth Barnes, and Paul Christiano. 2023. Evaluating language-model agents on realistic autonomous tasks. https://evals.alignment. o… ↗
60. [381] Andrei Kirilenko, Albert S Kyle, Mehrdad Samadi, and Tugkan Tuzun. 2017. The flash crash: High-frequency trading in an electronic market. The Journal of Finance, 72(3):967–998. find
61. [382] Svetlana Kiritchenko and Saif M Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. arXiv preprint arXiv:1805.04508.
62. [383] Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. 2024. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and… find
63. [384] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-normalizing neural networks. Advances in Neural Information Processing Systems, 30. find
64. [385] Toryn Q Klassen, Sheila A McIlraith, Christian Muise, and Jarvis Xu. 2022. Planning to avoid side effects. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36(9), pages 9830–9839. find
65. [386] Franziska Klügl, Manuel Fehler, and Rainer Herrler. 2005. About the role of the environment in multi-agent simulations. In Environments for Multi-Agent Systems: First International Workshop, E4MAS 2004, New York, NY, July 19, 2004, Revised Selected Papers 1, pages 127–149. Springer. find
66. [387] W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. 2023. Reward (mis) design for autonomous driving. Artificial Intelligence, 316:103829. find
67. [388] W Bradley Knox and Peter Stone. 2008. Tamer: Training an agent manually via evaluative reinforcement. In 2008 7th IEEE international conference on development and learning, pages 292–297. IEEE. find
68. [389] W Bradley Knox and Peter Stone. 2012. Reinforcement learning from simultaneous human and mdp reward. In AAMAS, volume 1004, pages 475–482. Valencia. find
69. [390] W Bradley Knox and Peter Stone. 2013. Learning non-myopically from human-generated reward. In Proceedings of the 2013 international conference on Intelligent user interfaces, pages 191–202. find
70. [391] W Bradley Knox, Peter Stone, and Cynthia Breazeal. 2013. Training a robot via human feedback: A case study. In Social Robotics: 5th International Conference, ICSR 2013, Bristol, UK, October 27-29, 2013, Proceedings 5, pages 460–470. Springer. find
71. [392] William Bradley Knox. 2012. Learning from human-generated reward. find
72. [393] Leonie Koessler and Jonas Schuett. 2023. Risk assessment at agi companies: A review of popular risk assessment techniques from other safety-critical industries. arXiv preprint arXiv:2307.08823. — [arXiv:2307.08823](https://arxiv.org/abs/2307.08823) find
73. [394] Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885–1894. PMLR.
74. [395] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. 2024. Openassistant conversations-democratizing large language model alignment. Advances in Neural Information Processing Syst…
75. [396] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. 2023. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506–17533. PMLR.
76. [397] Peter Krafft, Chris Baker, Alex Pentland, and Joshua Tenenbaum. 2016. Modeling human ad hoc coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30–(1). find
77. [398] Victoria Krakovna. 2020. More instances about specification gaming. https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTi RRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml. ↗
78. [399] Victoria Krakovna. 2022. Paradigms of ai alignment: components and enablers. https://vkrakovna.wordpress.com/2022/06/02/paradigms-of-ai-alignment-components-and-enabler s. ↗
79. [400] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021. Gedi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4929–4952.
80. [401] Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. 2022. The disagreement problem in explainable machine learning: A practitioner’s perspective. arXiv preprint arXiv:2202.01602. — [arXiv:2202.01602](https://arxiv.org/abs/2202.01602) find
81. [402] Maya Krishnan. 2020. Against interpretability: a critical examination of the interpretability problem in machine learning. Philosophy & Technology, 33(3):487–502. find
82. [403] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. 2021. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pages 5815–5826. PMLR. find
83. [404] David Krueger, Tegan Maharaj, and Jan Leike. 2020. Hidden incentives for auto-induced distributional shift. arXiv preprint arXiv:2009.09153. — [arXiv:2009.09153](https://arxiv.org/abs/2009.09153) find
84. [405] Andras Kupcsik, Marc Deisenroth, Jan Peters, and Gerhard Neumann. 2013. Data-efficient generalization of robot skills with contextual policy search. In Proceedings of the AAAI conference on artificial intelligence, volume 27(1), pages 1401–1407. find
85. [406] Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Sam Gershman, and Finale Doshi-Velez. 2019. An evaluation of the human-interpretability of explanation. arXiv preprint arXiv:1902.00006. — [arXiv:1902.00006](https://arxiv.org/abs/1902.00006) find
86. [407] Isaac Lage, Andrew Ross, Samuel J Gershman, Been Kim, and Finale Doshi-Velez. 2018. Human-in-the-loop interpretability prior. Advances in Neural Information Processing Systems, 31. find
87. [408] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253. find
88. [409] Richard N Landers and Tara S Behrend. 2023. Auditing the ai auditors: A framework for evaluating fairness and bias in high stakes ai predictive models. American Psychologist, 78(1):36. find
89. [410] Leon Lang, Davis Foote, Stuart Russell, Anca Dragan, Erik Jenner, and Scott Emmons. 2024. When your ai deceives you: Challenges with partial observability of human evaluators in reward learning. arXiv preprint arXiv:2402.17747. — [arXiv:2402.17747](https://arxiv.org/abs/2402.17747) find
90. [411] Chan Lawrence, Adria Garriga-alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Thomas Nate. 2023. Causal Scrubbing: a method for rigorously testing interpretability hypotheses [Redwood Research]. https://www.alignmentforum.org/posts… ↗
91. [412] Yann LeCun. 2022. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62. find
92. [413] Deokjae Lee, Seungyong Moon, Junhyeok Lee, and Hyun Oh Song. 2022. Query-efficient and scalable black-box adversarial attacks on discrete sequential data via bayesian optimization. In International Conference on Machine Learning, pages 12478–12497. PMLR. find
93. [414] Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023a. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. find
94. [415] Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. 2017. Interactive visualization and manipulation of attention-based neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 121–126. find
95. [416] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. 2019. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274. find
96. [417] Sunok Lee, Minha Lee, and Sangsu Lee. 2023b. What if artificial intelligence become completely ambient in our daily lives? exploring future human-ai interaction through high fidelity illustrations. International Journal of Human–Computer Interaction, 39(7):1371–1389. find
97. [418] Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al. 2020. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial lif… find
98. [419] Joel Lehman, Kenneth O Stanley, et al. 2008. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE, pages 329–336. find
99. [420] Joel Z Leibo, Edgar A Dueñez-Guzman, Alexander Vezhnevets, John P Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charlie Beattie, Igor Mordatch, and Thore Graepel. 2021. Scalable evaluation of multi-agent reinforcement learning with melting pot. In International conference on machine lea… find
100. [421] Jan Leike. 2022. A proposal for improving societys values. find
101. [422] Jan Leike. 2023a. Combining weak-to-strong generalization with scalable oversight. https://aligne d.substack.com/p/combining-w2sg-with-scalable-oversight?utm_source=post-e mail-title&publication_id=328633&post_id=139945470&utm_campaign=email-pos t-title&isFreemail=true&r=2xbqf0. find
102. [423] Jan Leike. 2023b. A proposal for importing society’s values. https://aligned.substack.com/p/a-proposal-for-importing-societys-values. ↗
103. [424] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871. find
104. [425] Filippa Lentzos. 2022. Ai and biological weapons. In Armament, Arms Control and Artificial Intelligence: The Janus-faced Nature of Machine Learning in the Military Realm, pages 91–100. Springer. find
105. [426] Belinda Z Li, Maxwell Nye, and Jacob Andreas. 2021a. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: L… find
106. [427] Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. 2023a. Trustworthy ai: From principles to practices. ACM Computing Surveys (CSUR), 55(9):1–46. find
107. [428] Chao Li, Kelu Yao, Jin Wang, Boyu Diao, Yongjun Xu, and Quanshi Zhang. 2022a. Interpretable generative adversarial networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36(2), pages 1280–1288. find
108. [429] Jiawei Li, Yiming Li, Xingchun Xiang, Shu-Tao Xia, Siyi Dong, and Yun Cai. 2020. Tnt: An interpretable tree-network-tree learning framework using knowledge distillation. Entropy, 22(11):1203. find
109. [430] Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2022b. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations. find
110. [431] Mengxi Li, Alper Canberk, Dylan P Losey, and Dorsa Sadigh. 2021b. Learning human objectives from sequences of physical corrections. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 2877–2883. IEEE. find
111. [432] Tao Li and Suresh P Sethi. 2017. A review of dynamic stackelberg game models. Discrete & Continuous Dynamical Systems-B, 22(1):125. find
112. [433] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023b. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305.
113. [434] Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning, pages 6565–6576. PMLR.
114. [435] Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. 2023. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla.
115. [436] Wim BG Liebrand. 1984. The effect of social motives, communication and group size on behaviour in an n-person multi-stage mixed-motive game. European journal of social psychology, 14(3):239–264. find
116. [437] Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. Kagnet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (… find
117. [438] Fengming Lin, Xiaolei Fang, and Zheming Gao. 2022a. Distributionally robust optimization: A review on theory and applications. Numerical Algebra, Control and Optimization, 12(1):159–212. find
118. [439] Jessy Lin, Daniel Fried, Dan Klein, and Anca Dragan. 2022b. Inferring rewards from language in context. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8546–8560. find
119. [440] Stephanie Lin, Jacob Hilton, and Owain Evans. 2022c. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252.
120. [441] Tsung-Han Lin and Ping Tak Peter Tang. 2019. Sparse dictionary learning by dynamical neural networks. In International Conference on Learning Representations. find
121. [442] Yen-Ting Lin and Yun-Nung Chen. 2023. LLM-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. In Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023), pages 47–58.
122. [443] Zachary C Lipton. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31–57. find
123. [444] Qin Liu, Meng Zheng, Benjamin Planche, Srikrishna Karanam, Terrence Chen, Marc Niethammer, and Ziyan Wu. 2022. Pseudoclick: Interactive Image Segmentation with Click Imitation. In European Conference on Computer Vision (ECCV). find
124. [445] Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Diyi Yang, and Soroush Vosoughi. 2024a. Training socially aligned language models on simulated social interactions. In The Twelfth International Conference on Learning Representations. find
125. [446] Shusen Liu, Tao Li, Zhimin Li, Vivek Srikumar, Valerio Pascucci, and Peer-Timo Bremer. 2018. Visual interrogation of attention-based models for natural language inference and machine comprehension. Technical report, Lawrence Livermore National Lab.(LLNL), Livermore, CA (United States). find
126. [447] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024b. Agentbench: Evaluatin…
127. [448] Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. 2020. Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994. find
128. [449] Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860.
129. [450] Robert Loftin, Bei Peng, James MacGlashan, Michael L Littman, Matthew E Taylor, Jeff Huang, and David L Roberts. 2016. Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning. Autonomous Agents and Multi-Agent Systems, 30:30–59. find
130. [451] Dylan P Losey, Andrea Bajcsy, Marcia K O’Malley, and Anca D Dragan. 2022. Physical interaction as communication: Learning robot objectives online from human corrections. The International Journal of Robotics Research, 41(1):20–44. find
131. [452] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in Neural Information Processing Systems, 30. find
132. [453] Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. 2023. Mechanistic mode connectivity. In International Conference on Machine Learning, pages 22965–23004. PMLR. find
133. [454] Daniel D Lundstrom, Tianjian Huang, and Meisam Razaviyayn. 2022. A rigorous study of integrated gradients method and extensions to internal neuron attributions. In International Conference on Machine Learning, pages 14485–14508. PMLR. find
134. [455] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. 2020. Learning latent plans from play. In Conference on robot learning, pages 1113–1132. PMLR. find
135. [456] Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. 2023. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters. find
136. [457] Weiqin Ma, Pu Duan, Sanmin Liu, Guofei Gu, and Jyh-Charn Liu. 2012. Shadow attacks: automatically evading system-call-behavior based malware detection. Journal in Computer Virology, 8:1–13. find
137. [458] Zixian Ma, Rose Wang, Fei-Fei Li, Michael Bernstein, and Ranjay Krishna. 2022. Elign: Expectation alignment as a multi-agent intrinsic reward. Advances in Neural Information Processing Systems, 35:8304–8317. find
138. [459] Matthijs M Maas. 2021. Aligning ai regulation to sociotechnical change. Oxford Handbook on AI Governance (Oxford University Press, 2022 forthcoming). find
139. [460] Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11). find
140. [461] James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew E Taylor, and Michael L Littman. 2017. Interactive learning from policy-dependent human feedback. In International conference on machine learning, pages 2285–2294. PMLR. find
141. [462] Alasdair MacIntyre. 2013. After virtue. A&C Black. find
142. [463] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations.
143. [464] Andreas Madsen, Siva Reddy, and Sarath Chandar. 2022. Post-hoc interpretability for neural nlp: A survey. ACM Computing Surveys (CSUR), 55(8):1–42. find
144. [465] MAI. 2023. Introducing democratic fine-tuning. find
145. [466] Daniel J Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, Jean-Baptiste Lespiau, Alex Ahern, et al. 2023. Faster sorting algorithms discovered using deep reinforcement learning. Nature, 618(7964):257–263. find
146. [467] Aaron Mannes. 2020. Governance, risk, and artificial intelligence. AI Magazine, 41(1):61–69. find
147. [468] James Manyika, Michael Chui, Mehdi Miremadi, Jacques Bughin, Katy George, Paul Willmott, and Martin Dewhurst. 2017. A future that works: Ai, automation, employment, and productivity. McKinsey Global Institute Research, Tech. Rep, 60:1–135. find
148. [469] Xiaofeng Mao, Yuefeng Chen, Ranjie Duan, Yao Zhu, Gege Qi, Xiaodan Li, Rong Zhang, Hui Xue, et al. 2022. Enhance the visual representation via discrete adversarial training. Advances in Neural Information Processing Systems, 35:7520–7533. find
149. [470] Peter Marbach and John N Tsitsiklis. 2001. Simulation-based optimization of markov reward processes. IEEE Transactions on Automatic Control, 46(2):191–209. find
150. [471] Gary Marcus. 2018. Deep learning: A critical appraisal.
151. [472] David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar. 2018. Transparency by design: Closing the gap between performance and interpretability in visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4942–4950. find
152. [473] Charles G McClintock and Eddy Van Avermaet. 1982. Social values and rules of fairness: A theoretical perspective. Cooperation and helping behavior, pages 43–71. find
153. [474] Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. 2023. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2307.15771. find
154. [475] Kevin R McKee, Ian Gemp, Brian McWilliams, Edgar A Duèñez-Guzmán, Edward Hughes, and Joel Z Leibo. 2020. Social diversity and social preferences in mixed-motive reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 869–87… find
155. [476] Lev E McKinney, Yawen Duan, David Krueger, and Adam Gleave. 2022. On the fragility of learned reward functions. In NeurIPS ML Safety Workshop. find
156. [477] Scott McLean, Gemma JM Read, Jason Thompson, Chris Baber, Neville A Stanton, and Paul M Salmon. 2023. The risks associated with artificial general intelligence: A systematic review. Journal of Experimental & Theoretical Artificial Intelligence, 35(5):649–663. find
157. [478] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1–35.
158. [479] Bahar Memarian and Tenzin Doleck. 2023. Fairness, accountability, transparency, and ethics (fate) in artificial intelligence (ai), and higher education: A systematic review. Computers and Education: Artificial Intelligence, page 100152. find
159. [480] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372. find
160. [481] Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2022b. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations. find
161. [482] Alex Mennen. 2018. A comment on the ida-alphagozero metaphor; capabilities versus alignment. https://www.alignmentforum.org/posts/yXFKh2jGysQNfX2NM/a-comment-on-the-ida-alp hagozero-metaphor-capabilities. ↗
162. [483] Bruno Mermet and Gaële Simon. 2016. Formal verication of ethical properties in multiagent systems. In 1st Workshop on Ethics in the Design of Intelligent Agents. find
163. [484] David M Messick and Charles G McClintock. 1968. Motivational bases of choice in experimental games. Journal of experimental social psychology, 4(1):1–25. find
164. [485] Meta. 2023. Meta and microsoft introduce the next generation of llama. https://ai.meta.com/blog/llama-2. ↗
165. [486] Julian Michael, Ari Holtzman, Alicia Parrish, Aaron Mueller, Alex Wang, Angelica Chen, Divyam Madaan, Nikita Nangia, Richard Yuanzhe Pang, Jason Phang, et al. 2022. What do nlp researchers believe? results of the nlp community metasurvey. arXiv preprint arXiv:2208.12852. find
166. [487] Michaelcohen. 2020. the-ai-debate-debate. https://www.alignmentforum.org/posts/L3QDs 6of4Rb2TgpRD/the-ai-debate-debate. ↗
167. [488] Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267:1–38.
168. [489] Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys (CSUR), 56(2):1–40.
169. [490] Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi. 2024. Can LLMs keep a secret? testing privacy implications of language models via contextual integrity theory. In The Twelfth International Conference on Learning Representations. find
170. [491] Yisroel Mirsky, Ambra Demontis, Jaidip Kotak, Ram Shankar, Deng Gelei, Liu Yang, Xiangyu Zhang, Maura Pintor, Wenke Lee, Yuval Elovici, et al. 2023. The threat of offensive ai to organizations. Computers & Security, 124:103006. find
171. [492] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533. find
172. [493] Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. 2023. Model-based reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 16(1):1–118. find
173. [494] Sina Mohseni, Niloofar Zarei, and Eric D Ragan. 2021. A multidisciplinary survey and framework for design and evaluation of explainable ai systems. ACM Transactions on Interactive Intelligent Systems (TiiS), 11(3-4):1– 45. find
174. [495] Jakob Mökander, Jonas Schuett, Hannah Rose Kirk, and Luciano Floridi. 2023. Auditing large language models: a three-layered approach. AI and Ethics, pages 1–31. find
175. [496] Ari S Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick. 2018. On the importance of single directions for generalization. In International Conference on Learning Representations. find
176. [497] Alexander Mordvintsev, Chris Olah, and Mike Tyka. 2015. Inceptionism: Going deeper into neural networks. Google Research Blog. find
177. [498] Emad Mostaque. 2022. Democratizing ai, stable diffusion & generative models. https://exchange.s cale.com/public/videos/emad-mostaque-stability-ai-stable-diffusion-open-s ource. ↗
178. [499] Darius Muglich, Christian Schroeder de Witt, Elise van der Pol, Shimon Whiteson, and Jakob Foerster. 2022. Equivariant networks for zero-shot coordination. Advances in Neural Information Processing Systems, 35:6410–6423. find
179. [500] Gabriel Mukobi. 2022. Iterated distillation-amplification, gato, and proto-agi. https://www.lesswron g.com/posts/Evyk8eb6b7tFd6pxJ/iterated-distillation-amplification-gato-a nd-proto-agi-re. ↗
180. [501] Arslan Munir, Alexander Aved, and Erik Blasch. 2022. Situational awareness: techniques, challenges, and prospects. AI, 3(1):55–77. find
181. [502] Kevin P. Murphy. 2023. Probabilistic Machine Learning: Advanced Topics. MIT Press. find
182. [503] Ryan O Murphy and Kurt A Ackermann. 2014. Social value orientation: Theoretical and measurement issues in the study of social preferences. Personality and Social Psychology Review, 18(1):13–41. find
183. [504] Ryan O Murphy, Kurt A Ackermann, and Michel JJ Handgraaf. 2011. Measuring social value orientation. Judgment and Decision making, 6(8):771–781. find
184. [505] Grazia Murtarelli, Anne Gregory, and Stefania Romenti. 2021. A conversation-based perspective for shaping ethical human–machine interactions: The particular challenge of chatbots. Journal of Business Research, 129:927–935. find
185. [506] Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1…
186. [507] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.
187. [508] Neel Nanda. 2023a. Attribution patching: Activation patching at industrial scale. find
188. [509] Neel Nanda. 2023b. Othello-gpt: Future work i am excited about. https://www.alignmentforum.org/posts/qgK7smTvJ4DB8rZ6h/othello-gpt-future-work-i-am-excited-about. ↗
189. [510] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2022. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations.
190. [511] Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967.
191. [512] Ashutosh Nayyar, Aditya Mahajan, and Demosthenis Teneketzis. 2013. Decentralized stochastic control with partial history sharing: A common information approach. IEEE Transactions on Automatic Control, 58(7):1644–1658. find
192. [513] Michael Neely, Stefan F Schouten, Maurits JR Bleeker, and Ana Lucic. 2021. Order in the court: Explainable ai methods prone to disagreement. arXiv preprint arXiv:2105.03287. — [arXiv:2105.03287](https://arxiv.org/abs/2105.03287) find
193. [514] Andrew Y Ng, Daishi Harada, and Stuart Russell. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pages 278–287. Citeseer. find
194. [515] Andrew Y Ng, Stuart Russell, et al. 2000. Algorithms for inverse reinforcement learning. In Icml, volume 1, page 2. find
195. [516] Richard Ngo. 2020a. Agi safety from first principles. https://www.alignmentforum.org/s/mzg tmmTKKn5MuCzFJ. ↗
196. [517] Richard Ngo. 2020b. continuing-the-takeoffs-debate. https://www.alignmentforum.org/posts/Tpn2Fx9daLvj28kes/continuing-the-takeoffs-debate. ↗
197. [518] Richard Ngo. 2021. /why i m excited about debate. https://www.alignmentforum.org/posts/LDsSqXf9Dpu3J3gHD/why-i-m-excited-about-debate. ↗
198. [519] Richard Ngo, Lawrence Chan, and Sören Mindermann. 2024. The alignment problem from a deep learning perspective: A position paper. In The Twelfth International Conference on Learning Representations. find
199. [520] Anh Nguyen, Jason Yosinski, and Jeff Clune. 2015. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. find
200. [521] Anh Nguyen, Jason Yosinski, and Jeff Clune. 2016. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. arXiv preprint arXiv:1602.03616. — [arXiv:1602.03616](https://arxiv.org/abs/1602.03616) find
201. [522] Chi Nguyen. 2020. My understanding of paul christiano’s iterated amplification al safety research agenda. https://www.alignmentforum.org/posts/PT8vSxsusqWuN7JXp/my-understanding-o f-paul-christiano-s-iterated-amplification\#A_mathematical_way_of_solving _Go_is_impossible. ↗
202. [523] Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach. 2021. f-irl: Inverse reinforcement learning via state marginal matching. In Conference on Robot Learning, pages 529–551. PMLR. find
203. [524] Nassim Nicholas. 2008. The black swan: the impact of the highly improbable. Journal of the Management Training Institut, 36(3):56. find
204. [525] Safiya Umoja Noble. 2018. Algorithms of oppression. In Algorithms of oppression. New York university press. find
205. [526] Safiya Umoja Noble, Beatrice Dias, Sara Cole Stratton, Aimee van Wynsberghe, Carlos Affonso Souza, Ilene Carpenter, Alvaro Martin Enriquez, and Emily Ratté. 2021. Ai regulation through an intergenerational lens. https://www3.weforum.org/docs/WEF_AI_Regulation_through_an_Intergenera tional_Lens… ↗
206. [527] Ritesh Noothigattu, Snehalkumar Gaikwad, Edmond Awad, Sohan Dsouza, Iyad Rahwan, Pradeep Ravikumar, and Ariel Procaccia. 2018. A voting-based system for ethical decision making. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). find
207. [528] Eirini Ntoutsi, Pavlos Fafalios, Ujwal Gadiraju, Vasileios Iosifidis, Wolfgang Nejdl, Maria-Esther Vidal, Salvatore Ruggieri, Franco Turini, Symeon Papadopoulos, Emmanouil Krasanakis, et al. 2020. Bias in data-driven artificial intelligence systems—an introductory survey. Wiley Interdisciplina… find
208. [529] OECD. 2019. Oecd principles on artificial intelligence. https://oecd.ai/en/ai-principles. ↗
209. [530] OecdAI. 2021. Ai principles. https://oecd.ai/en/dashboards/ai-principles/P8. ↗
210. [531] Caspar Oesterheld. 2021. Approval-directed agency and the decision theory of newcomb-like problems. Synthese, 198(Suppl 27):6491–6504. find
211. [532] Caspar Oesterheld and Vincent Conitzer. 2022. Safe pareto improvements for delegated game playing. Autonomous Agents and Multi-Agent Systems, 36(2):46. find
212. [533] Chris Olah. 2014. Visualizing mnist: An exploration of dimensionality reduction. https://colah.gi thub.io/posts/2014-10-Visualizing-MNIST. ↗
213. [534] Chris Olah. 2015. Visualizing representations: Deep learning and human beings. https://colah.gith ub.io/posts/2015-01-Visualizing-Representations. ↗
214. [535] Chris Olah. 2023. Interpretability dreams. https://transformer-circuits.pub/2023/inter pretability-dreams/index.html\#larger-scale. ↗
215. [536] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill, 5(3):e00024–001. find
216. [537] Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. 2018. The building blocks of interpretability. Distill, 3(3):e10. find
217. [538] Chris Olah et al. 2017. Feature visualization. Distill. find
218. [539] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. In-context learning and induction heads. arXiv preprint arXiv:2209.11895.
219. [540] Stephen M Omohundro. 2008. The basic ai drives. In AGI, volume 171, pages 483–492. find
220. [541] OpenAI. 2021a. Curve detectors. https://distill.pub/2020/circuits/curve-detectors. ↗
221. [542] OpenAI. 2021b. Weight banding. https://distill.pub/2020/circuits/weight-banding. ↗
222. [543] OpenAI. 2023a. Gpt-4 technical report.
223. [544] OpenAI. 2023b. Gpt-4v(ision) system card. https://cdn.openai.com/papers/GPTV_System_Card.pdf. ↗
224. [545] OpenAI. 2023c. Introducing superalignment. https://openai.com/blog/introducing-super alignment. Accessed on July 5, 2023. ↗
225. [546] Robert Opp. 2023. Committing to bridging the digital divide in least developed countries. https://www.undp.org/blog/committing-bridging-digital-divide-least-developed-countri es. ↗
226. [547] Afshin Oroojlooy and Davood Hajinezhad. 2023. A review of cooperative multi-agent deep reinforcement learning. Applied Intelligence, 53(11):13677–13722. find
227. [548] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. 2018. An algorithmic perspective on imitation learning. Foundations and Trends® in Robotics, 7(1-2):1–179. find
228. [549] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
229. [550] Lorenzo Pacchiardi, Alex James Chan, Sören Mindermann, Ilan Moscovitz, Alexa Yue Pan, Yarin Gal, Owain Evans, and Jan M. Brauner. 2024. How to catch an AI liar: Lie detection in black-box LLMs by asking unrelated questions. In The Twelfth International Conference on Learning Representations.
230. [551] Malayandi Palan, Gleb Shevchuk, Nicholas Charles Landolfi, and Dorsa Sadigh. 2019. Learning reward functions by integrating human demonstrations and preferences. In Robotics: Science and Systems. find
231. [552] Alexander Pan, Kush Bhatia, and Jacob Steinhardt. 2021. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations. find
232. [553] Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Jonathan Ng, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. 2023a. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. ICML. find
233. [554] Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. 2023b. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In International Conference on… find
234. [555] Rahul Pandey, Hemant Purohit, Carlos Castillo, and Valerie L Shalin. 2022. Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learning. International Journal of Human-Computer Studies, 160:102772. find
235. [556] Paulina Karolina Pankowska. 2020. Framework on ethical aspects of artificial intelligence, robotics and related technologies. European Parliament. find
236. [557] Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael Wellman. 2016. Towards the science of security and privacy in machine learning. arXiv preprint arXiv:1611.03814. — [arXiv:1611.03814](https://arxiv.org/abs/1611.03814) find
237. [558] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. 2022. The unsurprising effectiveness of pre-trained vision models for control. In International Conference on Machine Learning, pages 17359–17371. PMLR. find
238. [559] Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023a. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22. find
239. [560] Peter S Park, Simon Goldstein, Aidan O’Gara, Michael Chen, and Dan Hendrycks. 2023b. Ai deception: A survey of examples, risks, and potential solutions. arXiv preprint arXiv:2308.14752. — [arXiv:2308.14752](https://arxiv.org/abs/2308.14752) find
240. [561] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. Bbq: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105.
241. [562] Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, 2(11). find
242. [563] Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
243. [564] Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pages 754–768. IEEE. find
244. [565] Will Pearce and Joseph Lucas. 2023. Nvidia ai red team: An introduction. https://developer.nvid ia.com/blog/nvidia-ai-red-team-an-introduction. ↗
245. [566] Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, and Ece Kamar. 2022. Investigations of performance and bias in human-ai teamwork in hiring. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36-11, pages 12089–12097. find
246. [567] Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. 2020. Performative prediction. In International Conference on Machine Learning, pages 7599–7609. PMLR. find
247. [568] Luís Moniz Pereira, Ari Saptawijaya, Luís Moniz Pereira, and Ari Saptawijaya. 2016a. Bridging two realms of machine ethics. Programming machine ethics, pages 159–165. find
248. [569] Luís Moniz Pereira, Ari Saptawijaya, et al. 2016b. Programming machine ethics, volume 26. Springer. find
249. [570] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419–3448…
250. [571] Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. 2023. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics…
251. [572] Julien Perolat, Joel Z Leibo, Vinicius Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel. 2017. A multi-agent reinforcement learning model of common-pool resource appropriation. Advances in Neural Information Processing Systems, 30. find
252. [573] Lucas Perry. 2020. Evan hubinger on inner alignment, outer alignment, and proposals for building safe advanced ai. https://www.alignmentforum.org/posts/qZGoHkRgANQpGHWnu/evan-hubin ger-on-inner-alignment-outer-alignment-and. ↗
253. [574] J Peters, Peter Buhlmann, and N Meinshausen. 2015. Causal inference using invariant prediction: identification and confidence intervals. arxiv. Methodology. find
254. [575] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. 2017. Elements of causal inference: foundations and learning algorithms. The MIT Press. find
255. [576] Steve Phelps and Yvan I. Russell. 2023. Investigating emergent goal-like behaviour in large language models using experimental economics. find
256. [577] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. 2017. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817–2826. PMLR. find
257. [578] James Pita, Manish Jain, Milind Tambe, Fernando Ordónez, and Sarit Kraus. 2010. Robust solutions to stackelberg games: Addressing bounded rationality and limited observations in human cognition. Artificial Intelligence, 174(15):1142–1171. find
258. [579] Fabrizio Pittorino, Antonio Ferraro, Gabriele Perugini, Christoph Feinauer, Carlo Baldassi, and Riccardo Zecchina. 2022. Deep networks on toroids: removing symmetries reveals the structure of flat regions in the landscape geometry. In International Conference on Machine Learning, pages 17759–1… find
259. [580] Robin L Plackett. 1975. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193–202. find
260. [581] Dean A Pomerleau. 1991. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88–97. find
261. [582] Karl Popper. 2005. The logic of scientific discovery. Routledge. find
262. [583] Omid Poursaeed, Tianxing Jiang, Harry Yang, Serge Belongie, and Ser-Nam Lim. 2021. Robustness and generalization via generative adversarial training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15711–15720. find
263. [584] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. 2022. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177. find
264. [585] Lutz Prechelt. 2002. Early stopping-but when? Springer. find
265. [586] Dale Purves, George J Augustine, David Fitzpatrick, Lawrence C Katz, Anthony-Samuel LaMantia, James O McNamara, and S Mark. Williams. 2001. Neuroscience, 2nd edition. Sinauer Associates. find
266. [587] Martin L Puterman. 2014. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons. find
267. [588] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations.
268. [589] Tianyi Qiu, Fanzhi Zeng, Jiaming Ji, Dong Yan, Kaile Wang, Jiayi Zhou, Han Yang, Josef Dai, Xuehai Pan, and Yaodong Yang. 2024. Rethinking information structures in rlhf: Reward generalization from a graph theory perspective. arXiv preprint arXiv:2402.10184. — [arXiv:2402.10184](https://arxiv.org/abs/2402.10184) find
269. [590] R Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried. 2005. Invariant visual representation by single neurons in the human brain. Nature, 435(7045):1102–1107. find
270. [591] Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, et al. 2023. Question decomposition improves the faithfulness of model-generated reasoning. arXiv preprint arXiv:2307.11768. find
271. [592] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.
272. [593] Hamed Rahimian and Sanjay Mehrotra. 2019. Distributionally robust optimization: A review. arXiv preprint arXiv:1908.05659. — [arXiv:1908.05659](https://arxiv.org/abs/1908.05659) find
273. [594] Dheeraj Rajagopal, Vidhisha Balachandran, Eduard H Hovy, and Yulia Tsvetkov. 2021. Selfexplain: A self-explaining architecture for neural text classifiers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 836–850. find
274. [595] Inioluwa Deborah Raji, Timnit Gebru, Margaret Mitchell, Joy Buolamwini, Joonseok Lee, and Emily Denton. 2020. Saving face: Investigating the ethical concerns of facial recognition auditing. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 145–151. find
275. [596] Ram Shankar Siva Kumar. 2023. Microsoft ai red team building future of safer ai. https://www.micr osoft.com/en-us/security/blog/2023/08/07/microsoft-ai-red-team-building-f uture-of-safer-ai. ↗
276. [597] Deepak Ramachandran and Eyal Amir. 2007. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pages 2586–2591. find
277. [598] Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. 2023. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 464–483. IEEE. find
278. [599] Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. 2022. Linear adversarial concept erasure. In International Conference on Machine Learning, pages 18400–18421. PMLR. find
279. [600] Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. 2020. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3:297–330. find
280. [601] Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. 2021. Probing the probing paradigm: Does probing accuracy entail task relevance?
281. [602] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389–5400. PMLR. find
282. In Neural Networks: Tricks of the trade, pages 55–69. [603] Siddharth Reddy, Anca Dragan, Sergey Levine, Shane Legg, and Jan Leike. 2020. Learning human objectives by evaluating hypothetical behavior. In International Conference on Machine Learning, pages 8020–8029. PMLR. find
283. [604] Siddharth Reddy, Anca D Dragan, and Sergey Levine. 2019. Sqil: Imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations. find
284. [605] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. 2022. A generalist agent. Transactions on Machine Learning Research. find
285. [606] Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. Advances in Neural Information Processing Systems, 32.
286. [607] Richard Ren, Steven Basart, Adam Khoja, Alice Gatti, Long Phan, Xuwang Yin, Mantas Mazeika, Alexander Pan, Gabriel Mukobi, Ryan H Kim, et al. 2024. Safetywashing: Do ai safety benchmarks actually measure safety progress? arXiv preprint arXiv:2407.21792. — [arXiv:2407.21792](https://arxiv.org/abs/2407.21792) find
287. [608] Yankun Ren, Jianbin Lin, Siliang Tang, Jun Zhou, Shuang Yang, Yuan Qi, and Xiang Ren. 2020. Generating natural language adversarial examples on a large scale with generative models. In ECAI 2020, pages 2156–2163. IOS Press. find
288. [609] Richard Ngo. 2022. Gradient hacking. https://www.alignmentforum.org/posts/EeAgytDZb DjRznPMA/gradient-hacking-definitions-and-examples. ↗
289. [610] Mattia Rigotti, Christoph Miksovic, Ioana Giurgiu, Thomas Gschwind, and Paolo Scotton. 2022. Attention-based interpretability with concept transformers. In International Conference on Learning Representations. find
290. [611] Mark Ring and Laurent Orseau. 2011. Delusion, survival, and intelligent agents. In Artificial General Intelligence: 4th International Conference, AGI 2011, Mountain View, CA, USA, August 3-6, 2011. Proceedings 4, pages 11–20. Springer. find
291. [612] Yuji Roh, Geon Heo, and Steven Euijong Whang. 2019. A survey on data collection for machine learning: a big data-ai integration perspective. IEEE Transactions on Knowledge and Data Engineering, 33(4):1328–1347. find
292. [613] Milton Rokeach. 1973. The nature of human values. Free press. find
293. [614] Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov. 2021. Solid: A large-scale semi-supervised dataset for offensive language identification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 915–928. find
294. [615] Andrew Ross, Isaac Lage, and Finale Doshi-Velez. 2017. The neural lasso: Local linear sparsity for interpretable explanations. In Workshop on Transparent and Interpretable Machine Learning in Safety Critical Environments, 31st Conference on Neural Information Processing Systems, volume 4. find
295. [616] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and Conference Pro… find
296. [617] Francesca Rossi, Kristen Brent Venable, and Toby Walsh. 2011. A Short Introduction to Preferences: Between AI and Social Choice. Morgan & Claypool Publishers. find
297. [618] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. 2004. " grabcut" interactive foreground extraction using iterated graph cuts. ACM transactions on graphics (TOG), 23(3):309–314. find
298. [619] Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206–215.
299. [620] Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pag…
300. [621] Tim Rudner and Helen Toner. 2021a. Key concepts in ai safety: Interpretability in machine learning. https://cset.georgetown.edu/publication/key-concepts-in-ai-safety-interpretab ility-in-machine-learning. ↗
301. [622] Tim Rudner and Helen Toner. 2021b. Key concepts in ai safety: Robustness and adversarial examples. https://cset.georgetown.edu/publication/key-concepts-in-ai-safety-robustn ess-and-adversarial-examples. ↗
302. [623] Stuart Russell. 2019. Human compatible: Artificial intelligence and the problem of control. Penguin. find
303. [624] Stuart Russell, Daniel Dewey, and Max Tegmark. 2015. Research priorities for robust and beneficial artificial intelligence. AI Magazine, 36(4):105–114. find
304. [625] Noveen Sachdeva and Julian McAuley. 2023. Data distillation: A survey. Transactions on Machine Learning Research. Survey Certification. find
305. [626] Joel L Sachs, Ulrich G Mueller, Thomas P Wilcox, and James J Bull. 2004. The evolution of cooperation. The Quarterly review of biology, 79(2):135–160. find
306. [627] Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. 2017. Active preference-based learning of reward functions. Escholarship. find
307. [628] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020. Distributionally robust neural networks. In International Conference on Learning Representations. find
308. [629] Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. 2022. A survey of evaluation metrics used for nlg systems. ACM Computing Surveys (CSUR), 55(2):1–39. find
309. [630] Pedro Saleiro, Benedict Kuester, Loren Hinkson, Jesse London, Abby Stevens, Ari Anisfeld, Kit T Rodolfa, and Rayid Ghani. 2018. Aequitas: A bias and fairness audit toolkit. arXiv preprint arXiv:1811.05577. — [arXiv:1811.05577](https://arxiv.org/abs/1811.05577) find
310. [631] Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller. 2019. Explainable AI: interpreting, explaining and visualizing deep learning, volume 11700. Springer Nature. find
311. [632] Jonas B Sandbrink. 2023. Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools. arXiv preprint arXiv:2306.13952. — [arXiv:2306.13952](https://arxiv.org/abs/2306.13952) find
312. [633] Lindsay Sanneman and Julie A Shah. 2020. A situation awareness-based framework for design and evaluation of explainable ai. In Explainable, Transparent Autonomous Agents and Multi-Agent Systems: Second International Workshop, EXTRAAMAS 2020, Auckland, New Zealand, May 9–13, 2020, Revised Selec… find
313. [634] Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Res…
314. [635] Beatrice Dias Sara Stratton. 2021. Why we must consider the intergenerational impacts of ai. https://www.weforum.org/agenda/2021/10/why-we-must-consider-the-intergeneration al-impact-of-ai. ↗
315. [636] Fumihiro Sasaki and Ryota Yamashina. 2020. Behavioral cloning from noisy demonstrations. In International Conference on Learning Representations. find
316. [637] William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802.
317. [638] Nripsuta Ani Saxena, Karen Huang, Evan DeFilippis, Goran Radanovic, David C Parkes, and Yang Liu. 2019. How do fairness definitions fare? examining public attitudes towards algorithmic definitions of fairness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 99–… find
318. [639] Stefan Schaal. 1996. Learning from demonstration. Advances in Neural Information Processing Systems, 9. find
319. [640] Stefan Schaal. 1999. Is imitation learning the route to humanoid robots? 3(6):233–242. find
320. [641] Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. 2024. Evaluating the moral beliefs encoded in llms. Advances in Neural Information Processing Systems, 36. find
321. [642] Johannes Schneider and Michalis Vlachos. 2021. Explaining neural networks by decoding layer activations. In Advances in Intelligent Data Analysis XIX: 19th International Symposium on Intelligent Data Analysis, IDA 2021, Porto, Portugal, April 26–28, 2021, Proceedings 19, pages 63–75. Springer. find
322. [643] Patrick Schramowski, Cigdem Turan, Sophie Jentzsch, Constantin Rothkopf, and Kristian Kersting. 2020. The moral choice machine. Frontiers in artificial intelligence, 3:516840. find
323. [644] Jonas Schuett, Noemi Dreksler, Markus Anderljung, David McCaffary, Lennart Heim, Emma Bluemke, and Ben Garfinkel. 2023. Towards best practices in agi safety and governance: A survey of expert opinion. arXiv preprint arXiv:2305.07153. — [arXiv:2305.07153](https://arxiv.org/abs/2305.07153) find
324. Trends in cognitive sciences, [645] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
325. [646] Peter Schuster and Karl Sigmund. 1983. Replicator dynamics. Journal of theoretical biology, 100(3):533– 538. find
326. [647] Shalom H Schwartz. 1992. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In Advances in experimental social psychology, volume 25, pages 1–65. Elsevier. find
327. [648] Shalom H Schwartz. 1994. Are there universal aspects in the structure and contents of human values? Journal of social issues, 50(4):19–45. find
328. [649] Elizabeth Seger, Noemi Dreksler, Richard Moulange, Emily Dardaman, Jonas Schuett, K Wei, Christoph Winter, Mackenzie Arnold, Seán Ó hÉigeartaigh, Anton Korinek, et al. 2023. Open-sourcing highly capable foundation models: An evaluation of risks, benefits, and alternative methods for pursuing o… — [arXiv:2311.09227](https://arxiv.org/abs/2311.09227) find
329. [650] Charbel-Raphael Segerie. 2023. Against almost every theory of impact of interpretability. https://www.alignmentforum.org/posts/LNA8mubrByG7SFacm/against-almost-every-theory-o f-impact-of-interpretability-1. ↗
330. [651] Amartya Sen. 1986. Social choice theory. Handbook of mathematical economics, 3:1073–1181. find
331. [652] Egemen Sert, Yaneer Bar-Yam, and Alfredo J Morales. 2020. Segregation dynamics with reinforcement learning and agent based modeling. Scientific Reports, 10(1):11771. find
332. [653] Rohin Shah, Pedro Freire, Neel Alex, Rachel Freedman, Dmitrii Krasheninnikov, Lawrence Chan, Michael D Dennis, Pieter Abbeel, Anca Dragan, and Stuart Russell. 2020. Benefits of assistance over reward learning. https://openreview.net/forum?id=DFIoGDZejIB. ↗
333. [654] Rohin Shah and Vikrant Varma. 2022. More examples of gmg. https://www.alignmentforum.org/posts/Cfe2LMmQC4hHTDZ8r/more-examples-of-goal-misgeneralization. ↗
334. [655] Rusheb Shah, Quentin Feuillade Montixi, Soroush Pour, Arush Tagade, and Javier Rando. 2023. Scalable and transferable black-box jailbreaks for language models via persona modulation. In Socially Responsible Language Modelling Research. find
335. [656] Ali Shahin Shamsabadi, Ricardo Sanchez-Matilla, and Andrea Cavallaro. 2020. Colorfool: Semantic adversarial colorization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1151–1160. find
336. [657] Jacob N Shapiro and David A Siegel. 2010. Is this paper dangerous? balancing secrecy and openness in counterterrorism. Security Studies, 19(1):66–98. find
337. [658] Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2024. Towards…
338. [659] Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, and Dieter Fox. 2022. Correcting robot plans with natural language feedback. arXiv preprint arXiv:2204.05186. — [arXiv:2204.05186](https://arxiv.org/abs/2204.05186) find
339. [660] Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. 2023. Videodex: Learning dexterity from internet videos. In Conference on Robot Learning, pages 654–665. PMLR. find
340. [661] Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2023. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825.
341. [662] Amit Sheth, Valerie L Shalin, and Ugur Kursuncu. 2022. Defining and detecting toxicity on social media: context and knowledge are key. Neurocomputing, 490:312–318. find
342. [663] Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, et al. 2023. Model evaluation for extreme risks. arXiv preprint arXiv:2305.15324. find
343. [664] Adam Shimi. 2022. How to diversify conceptual alignment: the model behind refine. https://www.al ignmentforum.org/posts/5uiQkyKdejX3aEHLM/how-to-diversify-conceptual-ali gnment-the-model-behind. ↗
344. [665] Daniel Shin, Anca D. Dragan, and Daniel S. Brown. 2023. Benchmarks and algorithms for offline preference-based reward learning. find
345. Trans. Mach. Learn. Res., 2023. find
346. [666] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235.
347. [667] Buck Shlegeris and Ryan Greenblatt. 2023. Meta-level adversarial evaluation of oversight techniques might allow robust measurement of their adequacy. In Alignment Forum. find
348. [668] Wai Man Si, Michael Backes, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Savvas Zannettou, and Yang Zhang. 2022. Why so toxic? measuring and triggering toxic behavior in open-domain chatbots. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Secu… find
349. [669] Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. 2023. Dual rl: Unification and new methods for reinforcement and imitation learning. In Sixteenth European Workshop on Reinforcement Learning. find
350. [670] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489. find
351. [671] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. 2017. Mastering the game of go without human knowledge. Nature, 550(7676):354–359. find
352. [672] David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. 2021. Reward is enough. Artificial Intelligence, 299:103535. find
353. [673] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034.
354. [674] Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. 2018. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In International Conference on Learning Representations. find
355. [675] Munindar P Singh. 2014. Norms as a basis for governing sociotechnical systems. ACM Transactions on Intelligent Systems and Technology (TIST), 5(1):1–23. find
356. [676] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460–9471. find
357. [677] Joar Max Viktor Skalse, Matthew Farrugia-Roberts, Stuart Russell, Alessandro Abate, and Adam Gleave. 2023. Invariance in policy optimisation and partial identifiability in reward learning. In International Conference on Machine Learning, pages 32033–32058. PMLR. find
358. [678] Nate Soares. 2018. The value learning problem. In Artificial intelligence safety and security, pages 89–97. Chapman and Hall/CRC. find
359. [679] Nate Soares and Benja Fallenstein. 2014. Aligning superintelligence with human interests: A technical research agenda. Machine Intelligence Research Institute (MIRI) technical report, 8. find
360. [680] Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. 2015. Corrigibility. In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence. find
361. [681] Nate Soares and Benya Fallenstein. 2017. Agent foundations for aligning machine intelligence with human interests: a technical research agenda. The technological singularity: Managing the journey, pages 103–125. find
362. [682] Emily H. Soice, Rafael Rocha, Kimberlee Cordova, Michael Specter, and Kevin M. Esvelt. 2023. Can large language models democratize access to dual-use biotechnology? find
363. [683] Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203.
364. [684] Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. 2018a. Multi-agent generative adversarial imitation learning. Advances in Neural Information Processing Systems, 31. find
365. [685] Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. 2018b. Constructing unrestricted adversarial examples with generative models. Advances in Neural Information Processing Systems, 31. find
366. [686] Atle Ottesen Søvik. 2022. What overarching ethical principle should a superintelligent ai follow? AI and Society: Knowledge Culture and Communication, 37(4):1505–1518. find
367. [687] Giovanni Spitale, Nikola Biller-Andorno, and Federico Germani. 2023. Ai model gpt-3 (dis)informs us better than humans. Science Advances, 9(26):eadh1850. find
368. [688] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Lea…
369. [689] Robin Staab, Mark Vero, Mislav Balunovic, and Martin Vechev. 2024. Beyond memorization: Violating privacy via inference with large language models. In The Twelfth International Conference on Learning Representations. find
370. [690] Bernd Carsten Stahl and Tonii Leach. 2023. Assessing the ethical and social concerns of artificial intelligence in neuroinformatics research: An empirical test of the european union assessment list for trustworthy ai (altai). AI and Ethics, 3(3):745–767. find
371. [691] Zach Stein-Perlman, Benjamin Weinstein-Raun, and Katja Grace. 2022. expert survey on progress in ai. AI Impacts. Available online at: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai (accessed December 7, 2022). ↗
372. [692] Jacob Steinhardt. 2015. https://jsteinhardt.wordpress.com/2015/06/24/long-ter m-and-short-term-challenges-to-ensuring-the-safety-of-ai-systems, title = Long-Term and Short-Term Challenges to Ensuring the Safety of AI Systems. ↗
373. [693] Jacob Steinhardt and Helen Toner. 2020. Why robustness is key to deploying ai. https://www.brooki ngs.edu/articles/why-robustness-is-key-to-deploying-ai. ↗
374. [694] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021. find
375. [695] Peter Stone, Gal Kaminka, Sarit Kraus, and Jeffrey Rosenschein. 2010. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1504–1509. find
376. [696] Hendrik Strobelt, Sebastian Gehrmann, Michael Behrisch, Adam Perer, Hanspeter Pfister, and Alexander M Rush. 2018. Seq2seq-vis: A visual debugging tool for sequence-to-sequence models. IEEE transactions on visualization and computer graphics, 25(1):353–363. find
377. [697] Simone Stumpf, Vidya Rajaram, Lida Li, Margaret Burnett, Thomas Dietterich, Erin Sullivan, Russell Drummond, and Jonathan Herlocker. 2007. Toward harnessing user feedback for machine learning. In Proceedings of the 12th international conference on Intelligent user interfaces, pages 82–91. find
378. [698] Simone Stumpf, Vidya Rajaram, Lida Li, Weng-Keen Wong, Margaret Burnett, Thomas Dietterich, Erin Sullivan, and Jonathan Herlocker. 2009. Interacting meaningfully with machine learning systems: Three experiments. International Journal of Human-Computer Studies, 67(8):639–662. find
379. [699] Theodore R Sumers, Mark K Ho, Robert D Hawkins, Karthik Narasimhan, and Thomas L Griffiths. 2021. Learning rewards from linguistic feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6002–6010. find
380. [700] AI Safety Summit. 2023. The bletchley declaration by countries attending the ai safety summit, 1-2 november 2023. https://www.gov.uk/government/publications/ai-safety-summit-2023-the -bletchley-declaration/the-bletchley-declaration-by-countries-attending-t he-ai-safety-summit-1-2-november-2023… ↗
381. [701] Chuangchuang Sun, Macheng Shen, and Jonathan P How. 2020. Scaling up multiagent reinforcement learning for robotic systems: Learn an adaptive sparse communication graph. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11755–11762. IEEE. find
382. [702] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2024. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36.
383. [703] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. 2018. Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th Int… find
384. [704] Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. 2021. Trafficsim: Learning to simulate realistic multi-agent behaviors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10400–10409. find
385. [705] Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press. find
386. [706] Justin Svegliato, Samer B Nashed, and Shlomo Zilberstein. 2021. Ethically compliant sequential decision making. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35(13), pages 11657–11665. find
387. [707] Shea Swaugerarchive. 2020. Software that monitors students during tests perpetuates inequality and violates their privacy. https://www.technologyreview.com/2020/08/07/1006132/software-alg orithms-proctoring-online-tests-ai-ethics. ↗
388. [708] Umar Syed, Michael Bowling, and Robert E Schapire. 2008. Apprenticeship learning using linear programming. In Proceedings of the 25th international conference on Machine learning, pages 1032–1039. find
389. [709] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199.
390. [710] Jonas Tallberg, Eva Erman, Markus Furendal, Johannes Geith, Mark Klamberg, and Magnus Lundgren. 2023. The global governance of artificial intelligence: Next steps for empirical and normative research. arXiv preprint arXiv:2305.11528. — [arXiv:2305.11528](https://arxiv.org/abs/2305.11528) find
391. [711] Kai Liang Tan, Yasaman Esfandiari, Xian Yeow Lee, Soumik Sarkar, et al. 2020. Robustifying reinforcement learning agents via action space adversarial training. In 2020 American control conference (ACC), pages 3959– 3964. IEEE. find
392. [712] Ming Tan. 1993. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pages 330–337. find
393. [713] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. find
394. [714] Annalisa T Taylor, Thomas A Berrueta, and Todd D Murphey. 2021. Active learning in robotics: A review of control principles. Mechatronics, 77:102576. find
395. [715] Max Tegmark and Steve Omohundro. 2023. Provably safe systems: the only path to controllable agi. arXiv preprint arXiv:2309.01933. — [arXiv:2309.01933](https://arxiv.org/abs/2309.01933) find
396. [716] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman… find
397. [717] The White House. 2023. Fact sheet: Biden-harris administration secures voluntary commitments from leading artificial intelligence companies to manage the risks posed by ai. https://www.whitehouse.gov/b riefing-room/statements-releases/2023/07/21/fact-sheet-biden-harris-admin istration-secures-… ↗
398. [718] Andrea L Thomaz and Cynthia Breazeal. 2008. Teachable robots: Understanding human teaching behavior to build more effective robot learners. Artificial Intelligence, 172(6-7):716–737. find
399. [719] Sunil Thulasidasan, Sushil Thapa, Sayera Dhaubhadel, Gopinath Chennupati, Tanmoy Bhattacharya, and Jeff Bilmes. 2021. An effective baseline for robustness to distributional shift. In 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 278–285. IEEE. find
400. [720] Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca Dragan, and Daniel S Brown. 2022. Causal confusion and reward misidentification in preference-based reward learning. In The Eleventh International Conference on Learning Representations. find
401. [721] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. 2017. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23–30. IEEE. find
402. [722] Suzanne Tolmeijer, Markus Kneer, Cristina Sarasua, Markus Christen, and Abraham Bernstein. 2020. Implementations in machine ethics: A survey. ACM Computing Surveys (CSUR), 53(6):1–38. find
403. [723] Faraz Torabi, Garrett Warnell, and Peter Stone. 2018. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4950–4957. find
404. [724] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
405. [725] Robert Trager, Ben Harack, Anka Reuel, Allison Carnegie, Lennart Heim, Lewis Ho, Sarah Kreps, Ranjit Lall, Owen Larter, Seán Ó hÉigeartaigh, et al. 2023. International governance of civilian ai: A jurisdictional certification approach. arXiv preprint arXiv:2308.15514. — [arXiv:2308.15514](https://arxiv.org/abs/2308.15514) find
406. [726] Johannes Treutlein, Michael Dennis, Caspar Oesterheld, and Jakob Foerster. 2021. A new formalism, method and open issues for zero-shot coordination. In International Conference on Machine Learning, pages 10413– 10423. PMLR. find
407. [727] Grigorios Tsoumakas and Ioannis Katakis. 2007. Multi-label classification: An overview. International Journal of Data Warehousing and Mining (IJDWM), 3(3):1–13. find
408. [728] Alexey Turchin and David Denkenberger. 2020. Classification of global catastrophic risks connected with artificial intelligence. Ai & Society, 35(1):147–163. find
409. [729] Alex Turner. 2022. Inner and outer alignment decompose one hard problem into two extremely hard problems. https://www.alignmentforum.org/posts/gHefoxiznGfsbiAu9/inner-and-out er-alignment-decompose-one-hard-problem-into. ↗
410. [730] Alex Turner, Neale Ratzlaff, and Prasad Tadepalli. 2020. Avoiding side effects in complex environments. Advances in Neural Information Processing Systems, 33:21406–21415. find
411. [731] Alex Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. 2021. Optimal policies tend to seek power. Advances in Neural Information Processing Systems, 34:23063–23074. find
412. [732] Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2024. Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36.
413. [733] UNESCO. 2021. Recommendation on the ethics of artificial intelligence. https://unesdoc.unesco.org/ark:/48223/pf0000381137. ↗
414. [734] UniteAI. 2023. What is ai capability control & why does it matter? https://www.unite.ai/what-i s-ai-capability-control-why-does-it-matter. ↗
415. [735] United Nations, ITU. 2023. Population of global offline continues steady decline to 2.6 billion people in 2023. ITU Press Release. find
416. [736] Fabio Urbina, Filippa Lentzos, Cédric Invernizzi, and Sean Ekins. 2022. Dual use of artificial-intelligence-powered drug discovery. Nature Machine Intelligence, 4(3):189–191. find
417. [737] Paul AM Van Lange, Ellen De Bruin, Wilma Otten, and Jeffrey A Joireman. 1997. Development of prosocial, individualistic, and competitive orientations: theory and preliminary evidence. Journal of personality and social psychology, 73(4):733. find
418. [738] Vladimir Vapnik. 1991. Principles of risk minimization for learning theory. Advances in Neural Information Processing Systems, 4. find
419. [739] Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019. Attention interpretability across nlp tasks. find
420. [740] Ajit Kumar Verma, Srividya Ajit, Durga Rao Karanki, et al. 2010. Reliability and safety engineering, volume 43. Springer. find
421. [741] Sahil Verma and Julia Rubin. 2018. Fairness definitions explained. In Proceedings of the international workshop on software fairness, pages 1–7. find
422. [742] Krakovna Victoria, Uesato Jonathan, Mikulik Vladimir, Rahtz Matthew, Everitt Tom, Kumar Ramana, Kenton Zac, Leike Jan, and Legg Shane. 2020. Specification gaming: the flip side of ai ingenuity. https://ww w.deepmind.com/blog/specification-gaming-the-flip-side-of-ai-ingenuity. find
423. [743] Jesse Vig. 2019. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 37–42.
424. [744] Ricardo Vinuesa, Hossein Azizpour, Iolanda Leite, Madeline Balaam, Virginia Dignum, Sami Domisch, Anna Felländer, Simone Daniela Langhans, Max Tegmark, and Francesco Fuso Nerini. 2020. The role of artificial intelligence in achieving the sustainable development goals. Nature Communications, 11… find
425. [745] Georg Henrik Von Wright. 1951. Deontic logic. Mind, 60(237):1–15. find
426. [746] Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language… find
427. [747] Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. 2024. The instruction hierarchy: Training llms to prioritize privileged instructions. arXiv preprint arXiv:2404.13208. find
428. [748] Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. 2024. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations. find
429. [749] Dilin Wang, Chengyue Gong, and Qiang Liu. 2019a. Improving neural language modeling via adversarial training. In International Conference on Machine Learning, pages 6555–6565. PMLR. find
430. [750] Fulton Wang and Cynthia Rudin. 2015. Falling rule lists. In Artificial intelligence and statistics, pages 1013–1022. PMLR. find
431. [751] Jiaqi Wang, Huafeng Liu, Xinyue Wang, and Liping Jing. 2021. Interpretable image recognition by constructing transparent embedding space. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 895–904. find
432. [752] Kaimeng Wang, Yu Zhao, and Ichiro Sakuma. 2023a. Learning robotic insertion tasks from human demonstration. IEEE Robotics and Automation Letters. find
433. [753] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations. find
434. [754] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2023b. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432.
435. [755] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. 2019b. Poet: open-ended coevolution of environments and their optimized solutions. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 142–151. find
436. [756] Woodrow Z Wang and Mark Beliaev. 2021. Emergent prosociality in multi-agent games through gifting. In 30th International Joint Conference on Artificial Intelligence (IJCAI). find
437. [757] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023c. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1… find
438. [758] Zhenyi Wang, Xiaoyang Wang, Bang An, Dong Yu, and Changyou Chen. 2020. Towards faithful neural table-to-text generation with content-matching constraints. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1072–1086.
439. [759] Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the gap: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617.
440. [760] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2024. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36.
441. [761] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
442. [762] Jörgen W Weibull. 1997. Evolutionary game theory. MIT press. find
443. [763] Laura Weidinger, Kevin R McKee, Richard Everett, Saffron Huang, Tina O Zhu, Martin J Chadwick, Christopher Summerfield, and Iason Gabriel. 2023. Using the veil of ignorance to align ai systems with principles of justice. Proceedings of the National Academy of Sciences, 120(18):e2213709120. find
444. [764] Lilian Weng. 2023a. Adversarial attacks on llms. https://lilianweng.github.io/posts/202 3-10-25-adv-attack-llm. ↗
445. [765] Lilian Weng. 2023b. Llm powered autonomous agents. https://lilianweng.github.io/posts/2023-06-23-agent. ↗
446. [766] Darrell M West. 2018. The future of work: Robots, AI, and automation. Brookings Institution Press. find
447. [767] Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698.
448. [768] White House. 2023. Ensuring safe, secure, and trustworthy ai. https://www.whitehouse.gov/w p-content/uploads/2023/07/Ensuring-Safe-Secure-and-Trustworthy-AI.pdf. ↗
449. [769] Erik Wijmans, Manolis Savva, Irfan Essa, Stefan Lee, Ari S. Morcos, and Dhruv Batra. 2023. Emergence of maps in the memories of blind navigation agents. In The Eleventh International Conference on Learning Representations. find
450. [770] wikipedia. 2023. Existential risk from artificial general intelligence. https://en.wikipedia.org/w iki/Existential_risk_from_artificial_general_intelligence. ↗
451. [771] Claus O Wilke, Jia Lan Wang, Charles Ofria, Richard E Lenski, and Christoph Adami. 2001. Evolution of digital organisms at high mutation rates leads to survival of the flattest. Nature, 412(6844):331–333. find
452. [772] Alan F Winfield, Katina Michael, Jeremy Pitt, and Vanessa Evers. 2019. Machine ethics: The design and governance of ethical ai and autonomous systems [scanning the issue]. Proceedings of the IEEE, 107(3):509– 517. find
453. [773] Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. 2017. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1–46. find
454. [774] Christian Wirth and Johannes Fürnkranz. 2013. Preference-based reinforcement learning: A preliminary survey. In Proceedings of the ECML/PKDD-13 Workshop on Reinforcement Learning from Generalized Feedback: Beyond Numeric Rewards. Citeseer. find
455. [775] Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback.
456. [776] Yueh-Hua Wu and Shou-De Lin. 2018. A low-cost ethics shaping approach for designing reinforcement learning agents. In Proceedings of the AAAI conference on artificial intelligence, volume 32. find
457. [777] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2024. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36.
458. [778] Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th international conference on world wide web, pages 1391–1399. find
459. [779] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864.
460. [780] Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5(12):1486– 1496. find
461. [781] Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. 2018a. Fairgan: Fairness-aware generative adversarial networks. In 2018 IEEE International Conference on Big Data (Big Data), pages 570–575. IEEE. find
462. [782] Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079.
463. [783] Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 295…
464. [784] Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Broeck. 2018b. A semantic loss function for deep learning with symbolic knowledge. In International conference on machine learning, pages 5502–5511. PMLR. find
465. [785] Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas S Huang. 2016. Deep interactive object selection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 373–381. find
466. [786] Mengjiao Yang, Sergey Levine, and Ofir Nachum. 2021. Trail: Near-optimal imitation learning with suboptimal data. In International Conference on Learning Representations. find
467. [787] Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. 2023a. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129. find
468. [788] Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023b. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949. find
469. [789] Huaxiu Yao, Xinyu Yang, Xinyi Pan, Shengchao Liu, Pang Wei Koh, and Chelsea Finn. 2024. Improving out-of-domain generalization with domain relations. In The Twelfth International Conference on Learning Representations. find
470. [790] Tianhe Yu Yevgen Chebotar. 2023. Rt-2: New model translates vision and language into action. https://www.deepmind.com/blog/rt-2-new-model-translates-vision-and-language-int o-action. ↗
471. [791] Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446.
472. [792] Jin Yong Yoo and Yanjun Qi. 2021. Towards improving adversarial training of nlp models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 945–956. find
473. [793] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. 2021. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1–36. find
474. [794] Han Yu, Zhiqi Shen, Chunyan Miao, Cyril Leung, Victor R Lesser, and Qiang Yang. 2018. Building ethics into artificial intelligence. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 5527–5533. find
475. [795] Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montserrat Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, brian ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. 2023. L… find
476. [796] Yang Yu. 2018. Towards sample efficient reinforcement learning. In IJCAI, pages 5739–5743. find
477. [797] Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2024. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36. find
478. [798] Luyao Yuan, Xiaofeng Gao, Zilong Zheng, Mark Edmonds, Ying Nian Wu, Federico Rossano, Hongjing Lu, Yixin Zhu, and Song-Chun Zhu. 2022. In situ bidirectional human-robot value alignment. Science Robotics, 7(68):eabm4183. find
479. [799] E Yudkowsky. 2018. Challenges to christiano’s capability amplification proposal. LessWrong. find
480. [800] Man-Ching Yuen, Irwin King, and Kwong-Sak Leung. 2011. A survey of crowdsourcing systems. In 2011 IEEE third international conference on privacy, security, risk and trust and 2011 IEEE third international conference on social computing, pages 766–773. IEEE. find
481. [801] Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann LeCun. 2021. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. arXiv preprint arXiv:2103.15949. find
482. [802] Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human L…
483. [803] Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combinatorial optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6066–6080.
484. [804] Wojciech Zaremb, Arka Dhar, Lama Ahmad, Tyna Eloundou, Shibani Santurkar, Sandhini Agarwal, and Jade Leung. 2023. Democratic inputs to ai. OpenAI Blog. find
485. [805] Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818–833. Springer.
486. [806] Rowan Zellers. 2019. Why we released grover. https://thegradient.pub/why-we-release d-grover. ↗
487. [807] Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018a. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335–340. find
488. [808] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2018b. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations.
489. [809] Quan-shi Zhang and Song-Chun Zhu. 2018. Visual interpretability for deep learning: a survey. Frontiers of Information Technology & Electronic Engineering, 19(1):27–39. find
490. [810] Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. 2018c. Interpretable convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8827–8836. find
491. [811] Shiyin Zhang, Jun Hao Liew, Yunchao Wei, Shikui Wei, and Yao Zhao. 2020a. Interactive object segmentation with inside-outside guidance. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12234–12244. find
492. [812] Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. 2018d. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5628–5635. IEEE. find
493. [813] Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, and Joseph E. Gonzalez. 2023a. The wisdom of hindsight makes language models better instruction followers. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of M… find
494. [814] Wenjia Zhang, Haoran Xu, Haoyi Niu, Peng Cheng, Ming Li, Heming Zhang, Guyue Zhou, and Xianyuan Zhan. 2023b. Discriminator-guided model-based offline imitation learning. In Conference on Robot Learning, pages 1266–1276. PMLR. find
495. [815] Yuan Zhang, Xiaoran Xu, Hanning Zhou, and Yan Zhang. 2020b. Distilling structured knowledge into embeddings for explainable and accurate recommendation. In Proceedings of the 13th international conference on web search and data mining, pages 735–743. find
496. [816] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023c. Siren’s song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219.
497. [817] Zhexin Zhang, Jiale Cheng, Hao Sun, Jiawen Deng, Fei Mi, Yasheng Wang, Lifeng Shang, and Minlie Huang. 2022. Constructing highly inductive contexts for dialogue safety through controllable reverse generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3684–… find
498. [818] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol…
499. [819] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.
500. [820] Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. 2024. On evaluating adversarial robustness of large vision-language models. Advances in Neural Information Processing Systems, 36. find
501. [821] Rui Zheng, Wei Shen, Yuan Hua, Wenbin Lai, Shihan Dou, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Haoran Huang, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Improving generalization of alignment with human preferences through group invariant learning. In The Twelfth International Conference on Learnin… find
502. [822] Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. 2016. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 4480–4488. find
503. [823] Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. 2018. Revisiting the importance of individual units in cnns via ablation. arXiv preprint arXiv:1806.02891. — [arXiv:1806.02891](https://arxiv.org/abs/1806.02891) find
504. [824] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36. find
505. [825] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. 2022. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
506. [826] Li Zhou and Kevin Small. 2021. Inverse reinforcement learning with natural language goals. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11116–11124. find
507. [827] Zhi-Hua Zhou. 2021. Machine learning. Springer Nature. find
508. [828] Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. 2019. The ingredients of real world robotic reinforcement learning. In International Conference on Learning Representations. find
509. [829] Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2023. Dyval: Graph-informed dynamic evaluation of large language models. arXiv preprint arXiv:2309.17167. find
510. [830] Simon Zhuang and Dylan Hadfield-Menell. 2020. Consequences of misaligned ai. Advances in Neural Information Processing Systems, 33:15763–15773. find
511. [831] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. 2008. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433–1438. Chicago, IL, USA. find
512. [832] Daniel Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Benjamin Weinstein-Raun, Daniel de Haas, et al. 2022. Adversarial training for high-stakes reliability. Advances in Neural Information Processing Systems, 35:9274–9286. find
513. [833] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. find
514. [834] Caleb Ziems, Jane Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2022. The moral integrity corpus: A benchmark for ethical dialogue systems. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3755–3773.
515. [835] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023a. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.
516. [836] Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. 2024a. Improving alignment and robustness with circuit breakers. find
517. [837] Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023b. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
518. [838] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. 2024b. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36. find

Updated 31 October 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ji-2023-ai-alignment-comprehensive-survey/