📄 Foundational Challenges in Assuring Alignment and Safety of Large Language Models

Paper · 2024 · Usman Anwar, Abulhair Saparov, Javier Rando et al.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

🧭 Fields, topics and concepts

Cite

Cite as Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Gunther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Zhang, Ruiqi Zhong, Sean O hEigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Aleksandar Petrov, Christian Schroeder de Witt, Sumeet Ramesh Motwan, Yoshua Bengio, Danqi Chen, Philip H.S. Torr, Samuel Albanie, Tegan Maharaj, Jakob Foerster, Florian Tramer, He He, Atoosa Kasirzadeh, Yejin Choi and David Krueger. 2024. Foundational Challenges in Assuring Alignment and Safety of Large Language Models. arXiv:2404.09932.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.09932 · arXiv 2404.09932

@misc{anwar2024foundational,
  title = {{Foundational Challenges in Assuring Alignment and Safety of Large Language Models}},
  author = {Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Gunther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Zhang, Ruiqi Zhong, Sean O hEigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Aleksandar Petrov, Christian Schroeder {de Witt}, Sumeet Ramesh Motwan, Yoshua Bengio, Danqi Chen, Philip H.S. Torr, Samuel Albanie, Tegan Maharaj, Jakob Foerster, Florian Tramer, He He, Atoosa Kasirzadeh, Yejin Choi and David Krueger},
  year = {2024},
  doi = {10.48550/arxiv.2404.09932},
  eprint = {2404.09932},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.09932}
}

References

1,093 references, 244 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jan Leike, John Schulman, and Jeffrey Wu. Our approach to alignment research, 2022. https://openai.com/blog/our-approach-to-alignment-research. ↗
2. Anthropic. Core Views on AI Safety: When, Why, What, and How, 2023a. https://www.anthropic.com/index/core-views-on-ai-safety. ↗
3. Frontier Model Forum. Introducing the Frontier Model Forum, 2023. https://www.frontiermodelf orum.org/updates/announcing-the-frontier-model-forum/. Accessed on: January 8, 2024. ↗
4. White House. Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. Presidential Actions, October 2023. URL https://www.whitehouse.gov/briefing-r oom/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustwo rthy-development-and-use-… ↗
5. UK PM Office. Prime Minister calls for global responsibility to take AI risks seriously and seize its opportunities, October 2023. https://www.gov.uk/government/news/prime-minister-calls-f or-global-responsibility-to-take-ai-risks-seriously-and-seize-its-opportunities. Accessed on: January 8, 2023. ↗
6. UN AI Advisory Board. Interim Report: Governing AI for Humanity, 2023. URL https://www.un.org/sites/un2.un.org/files/ai%5Fadvisory%5Fbody%5Finterim%5Freport.pdf. ↗
7. Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, et al. Managing AI risks in an era of rapid progress. arXiv preprint arXiv:2310.17688, 2023. find
8. FAccT. Statement on AI Harms and Policy, 2023. URL https://facctconference.org/2023/har m-policy. ↗
9. CAIS. Statement on AI Risk: AI experts and public figures express their concern about AI risk, 2023. https://www.safe.ai/statement-on-ai-risk. ↗
10. CHAI, Far.ai, and Ditchley Foundation. Prominent AI Scientists from China and the West Propose Joint Strategy to Mitigate Risks from AI, 2023. https://humancompatible.ai/news/2023/10/31/prominent-ai-scientists-from-china-and-the-west-propose-joint-strategy-to-mitigat e-risks-from-ai/. Accessed on:… ↗
11. Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective, 2023.
12. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
13. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
14. Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg. Levels of AGI: Operationalizing Progress on the Path to AGI. arXiv preprint arXiv:2311.02462, 2023. — [arXiv:2311.02462](https://arxiv.org/abs/2311.02462) find
15. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021.
16. Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and surprise in large generative models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1747–1764, 2022.
17. Abeba Birhane, Vinay Prabhu, Sang Han, and Vishnu Naresh Boddeti. On Hate Scaling Laws For Data-Swamps. arXiv preprint arXiv:2306.13141, 2023. find
18. Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, et al. Harms from Increasingly Agentic Algorithmic Systems. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transpar… find
19. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016.
20. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved Problems in ML Safety. arXiv preprint arXiv:2109.13916, 2021a.
21. Andrew Critch and David Krueger. AI research considerations for human existential safety (ARCHES). arXiv preprint arXiv:2006.04948, 2020. — [arXiv:2006.04948](https://arxiv.org/abs/2006.04948) find
22. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. Alignment of Language Agents, 2021.
23. Paul Christiano. Clarifying “AI Alignment”, April 7 2018. URL https://ai-alignment.com/clari fying-ai-alignment-cec47cd69dd6. ↗
24. Iason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411–437, 2020. find
25. Nancy G Leveson. Engineering a safer world: Systems thinking applied to safety. The MIT Press, 2016. find
26. Laura Weidinger, Maribeth Rauh, Nahema Marchal, Arianna Manzini, Lisa Anne Hendricks, Juan Mateos-Garcia, Stevie Bergman, Jackie Kay, Conor Griffin, Ben Bariach, et al. Sociotechnical Safety Evaluation of Generative AI Systems. arXiv preprint arXiv:2310.11986, 2023a. — [arXiv:2310.11986](https://arxiv.org/abs/2310.11986) find
27. Rob Ashmore, Radu Calinescu, and Colin Paterson. Assuring the machine learning lifecycle: Desiderata, methods, and challenges. ACM Computing Surveys (CSUR), 54(5):1–39, 2021. find
28. Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Marvin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Bau, Max Tegmark, ..., and… find
29. Ari Holtzman, Peter West, and Luke Zettlemoyer. Generative Models as a Complex Systems Science: How can we make sense of large language model behavior? arXiv preprint arXiv:2308.00189, 2023. find
30. Jacob Steinhardt. Complex Systems are Hard to Control, 2023. https://bounded-regret.ghost.io/complex-systems-are-hard-to-control/. ↗
31. Dan Hendrycks. Introduction to AI Safety, Ethics, and Society. 2023. find
32. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abilities of Large Language Models. Transactions on Machine…
33. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, Uist ’23, New York, NY, USA, aug 2023a… find
34. Kevin MacG Adams, Patrick T Hester, Joseph M Bradley, Thomas J Meyers, and Charles B Keating. Systems theory as the foundation for understanding systems. Systems Engineering, 17(1):112–123, 2014. find
35. Allen Newell and Herbert A. Simon. Computer science as empirical inquiry: symbols and search. Commun. ACM, 19(3):113–126, 1976. URL https://doi.org/10.1145/360018.360022. ↗
36. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. URL https://proceedings.neuri… ↗
37. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
38. Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning. arXiv preprint arXiv:2312.01552, 2023a.
39. Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387, 2023a.
40. Sophie Xhonneux, David Dobre, Jian Tang, Gauthier Gidel, and Dhanya Sridhar. In-context learning can re-learn forbidden tasks. arXiv preprint arXiv:2402.05723, 2024. — [arXiv:2402.05723](https://arxiv.org/abs/2402.05723) find
41. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv preprint arXiv: Arxiv-2305.16291, 2023a. find
42. Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental Limitations of Alignment in Large Language Models. arXiv preprint arXiv:2304.11082, 2023.
43. Raphaël Millière. The Alignment Problem in Context. arXiv preprint arXiv:2311.02147, 2023. — [arXiv:2311.02147](https://arxiv.org/abs/2311.02147) find
44. Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, Francesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan J Hubinger, Yuntao Bai,… find
45. Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018, 2024.
46. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An Explanation of In-context Learning as Implicit Bayesian Inference. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=RdJVFCHjUMI. ↗
47. Xinyi Wang, Wanrong Zhu, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. arXiv preprint arXiv:2301.11916, 2023b.
48. Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. arXiv preprint arXiv:2303.07895, 2023.
49. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022.
50. Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function Vectors in Large Language Models. arXiv preprint arXiv:2310.15213, 2023.
51. Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. arXiv preprint arXiv:2310.15916, 2023.
52. Eric J. Bigelow, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, and Tomer D. Ullman. In-Context Learning Dynamics with Random Binary Sequences, 2023. find
53. Sivaramakrishnan Swaminathan, Antoine Dedieu, Rajkumar Vasudeva Raju, Murray Shanahan, Miguel Lazaro-Gredilla, and Dileep George. Schema-learning and rebinding as mechanisms of in-context learning and emergence. arXiv preprint arXiv:2307.01201, 2023. — [arXiv:2307.01201](https://arxiv.org/abs/2307.01201) find
54. Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta. Flirt: Feedback loop in-context red teaming. arXiv preprint arXiv:2308.04265, 2023.
55. Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprint arXiv:2211.09066, 2022.
56. Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Seyed Mehran Kazemi, Najoung Kim, and He He. Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples. arXiv preprint arXiv:2305.15269, 2023.
57. Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization. arXiv preprint arXiv:2305.19420, 2023a. — [arXiv:2305.19420](https://arxiv.org/abs/2305.19420) find
58. Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820, 2019. — [arXiv:1906.01820](https://arxiv.org/abs/1906.01820) find
59. Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, Razvan Pascanu, et al. Uncovering mesa-optimization algorithms in transformers. arXiv preprint arXiv:2309.05858, 2023. — [arXiv:2309.05858](https://arxiv.org/abs/2309.05858) find
60. Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. General-purpose in-context learning by meta-learning transformers. arXiv preprint arXiv:2212.04458, 2022. — [arXiv:2212.04458](https://arxiv.org/abs/2212.04458) find
61. Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? Investigations with linear models. arXiv preprint arXiv:2211.15661, 2022a.
62. Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. arXiv preprint arXiv:2212.07677, 2022.
63. Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? A case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583–30598, 2022.
64. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball. arXiv preprint arXiv:2305.13534, 2023b.
65. Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. arXiv preprint arXiv:2306.00297, 2023. find
66. Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection. arXiv preprint arXiv:2306.04637, 2023a. — [arXiv:2306.04637](https://arxiv.org/abs/2306.04637) find
67. Abhishek Panigrahi, Sadhika Malladi, Mengzhou Xia, and Sanjeev Arora. Trainable transformer in transformer. arXiv preprint arXiv:2307.01189, 2023. — [arXiv:2307.01189](https://arxiv.org/abs/2307.01189) find
68. Licong Lin, Yu Bai, and Song Mei. Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining. arXiv preprint arXiv:2310.08566, 2023b. — [arXiv:2310.08566](https://arxiv.org/abs/2310.08566) find
69. Deqing Fu, Tian-Qi Chen, Robin Jia, and Vatsal Sharan. Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study with Linear Models. arXiv preprint arXiv:2310.17086, 2023a. find
70. Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. arXiv preprint arXiv:2306.17844, 2023a. find
71. Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations, 2019.
72. Zhong Li, Jiequn Han, E Weinan, and Qianxiao Li. Approximation and optimization theory for linear continuous-time recurrent neural networks. Journal of Machine Learning Research, 23(42):1–85, 2022a. find
73. Shida Wang and Beichen Xue. State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory. Advances in Neural Information Processing Systems, 36, 2023. find
74. Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, and Terry Lyons. Theoretical foundations of deep selective state-space models. arXiv preprint arXiv:2402.19047, 2024. — [arXiv:2402.19047](https://arxiv.org/abs/2402.19047) find
75. Yihan Wang, Jatin Chauhan, Wei Wang, and Cho-Jui Hsieh. Universality and limitations of prompt tuning. Advances in Neural Information Processing Systems, 36, 2023c. find
76. Aleksandar Petrov, Philip HS Torr, and Adel Bibi. Prompting a pretrained transformer can be a universal approximator. arXiv preprint arXiv:2402.14753, 2024. — [arXiv:2402.14753](https://arxiv.org/abs/2402.14753) find
77. Aleksandar Petrov, Philip Torr, and Adel Bibi. When do prompting and prefix-tuning work? a theory of capabilities and limitations. In The Twelfth International Conference on Learning Representations, 2023a. find
78. Ivan Lee, Nan Jiang, and Taylor Berg-Kirkpatrick. Exploring the relationship between model architecture and in-context learning ability. In The Twelfth International Conference on Learning Representations, 2023a. find
79. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, ..., and Chris Olah. In-co… — In-context Learning and Induction Heads
80. Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. arXiv preprint arXiv:2312.03002, 2023. — [arXiv:2312.03002](https://arxiv.org/abs/2312.03002) find
81. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language Models Implement Simple Word2Vec-style Vector Arithmetic. arXiv preprint arXiv:2305.16130, 2023a. find
82. Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. Overthinking the truth: Understanding how language models process false demonstrations. arXiv preprint arXiv:2307.09476, 2023. find
83. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846, 2023b.
84. Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression. arXiv preprint arXiv:2306.15063, 2023. — [arXiv:2306.15063](https://arxiv.org/abs/2306.15063) find
85. Stephanie CY Chan, Adam Santoro, Andrew K Lampinen, Jane X Wang, Aaditya Singh, Pierre H Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent few-shot learning in transformers. arXiv preprint arXiv:2205.05055, 2022. find
86. Michael Hahn and Navin Goyal. A theory of emergent in-context learning as implicit structure induction. arXiv preprint arXiv:2303.07971, 2023. — [arXiv:2303.07971](https://arxiv.org/abs/2303.07971) find
87. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint arXiv:2101.00027, 2020. find
88. Together Computer. RedPajama: an Open Dataset for Training Large Language Models, October 2023. URL https://github.com/togethercomputer/RedPajama-Data. ↗
89. Aaditya K Singh, Stephanie CY Chan, Ted Moskovitz, Erin Grant, Andrew M Saxe, and Felix Hill. The transient nature of emergent in-context learning in transformers. arXiv preprint arXiv:2311.08360, 2023. — [arXiv:2311.08360](https://arxiv.org/abs/2311.08360) find
90. Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, et al. Model evaluation for extreme risks. arXiv preprint arXiv:2305.15324, 2023.
91. Melanie Mitchell. How do we know how smart AI systems are?, 2023. find
92. Margot Kaminski. Regulating the Risks of AI. Boston University Law Review, 103:1347, 2023. find
93. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
94. R Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638, 2023. find
95. Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Technical report, Google DeepMind, 2024. https://storage.googleapis.com/deepmind-media/g emini/gemini_v1_5_report.pdf. ↗
96. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. A benchmark for learning to translate a new language from one grammar book. arXiv preprint arXiv:2309.16575, 2023. find
97. Nicholas Carlini. A GPT-4 Capability Forecasting Challenge, 2023a. https://nicholas.carlini.com/writing/llm-forecast/question/Capital-of-Paris. Accessed on: December 18, 2023. ↗
98. Ernest Davis. The limitations of standardized science tests as benchmarks for artificial intelligence research: Position paper. arXiv preprint arXiv:1411.1629, 2014. — [arXiv:1411.1629](https://arxiv.org/abs/1411.1629) find
99. Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399, 2023. find
100. Pei Zhou, Aman Madaan, Srividya Pranavi Potharaju, Aditya Gupta, Kevin R McKee, Ari Holtzman, Jay Pujara, Xiang Ren, Swaroop Mishra, Aida Nematzadeh, et al. How FaR Are Large Language Models From Agents with Theory-of-Mind? arXiv preprint arXiv:2310.03051, 2023a.
101. Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever hans or neural theory of mind? Stress testing social reasoning in large language models. arXiv preprint arXiv:2305.14763, 2023. find
102. Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions. arXiv preprint arXiv:2310.15421, 2023. find
103. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. find
104. Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktäschel, and David Scott Krueger. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks, 2023a. find
105. Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. Advances in neural information processing systems, 30, 2017a. find
106. Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017. — [arXiv:1703.11008](https://arxiv.org/abs/1703.11008) find
107. Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. arXiv preprint arXiv:1707.09564, 2017b. — [arXiv:1707.09564](https://arxiv.org/abs/1707.09564) find
108. Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning, pages 254–263. PMLR, 2018. find
109. Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, pages 1310–1320. Pmlr, 2019. find
110. Nicholas Carlini, Florian Tramer, Krishnamurthy Dj Dvijotham, Leslie Rice, Mingjie Sun, and J Zico Kolter. (certified!!) Adversarial robustness for free! arXiv preprint arXiv:2206.10550, 2022. — [arXiv:2206.10550](https://arxiv.org/abs/2206.10550) find
111. Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-Mix: A flexible and expandable family of evaluations for AI models. arXiv preprint arXiv:2310.17567, 2023a. — [arXiv:2310.17567](https://arxiv.org/abs/2310.17567) find
112. Ali Shirali, Rediet Abebe, and Moritz Hardt. A theory of dynamic benchmarks. arXiv preprint arXiv:2210.03165, 2022. — [arXiv:2210.03165](https://arxiv.org/abs/2210.03165) find
113. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024–001, 2020. find
114. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small. In The Eleventh International Conference on Learning Representations, sep 2022. URL https://openreview.net/f orum?id=Nps… ↗
115. John B Carroll. Human cognitive abilities: A survey of factor-analytic studies. Cambridge University Press, 1993. find
116. Ryan Burnell, Han Hao, Andrew RA Conway, and Jose Hernandez Orallo. Revealing the structure of language model capabilities. arXiv preprint arXiv:2306.10062, 2023a. find
117. Xiting Wang, Liming Jiang, Jose Hernandez-Orallo, Luning Sun, David Stillwell, Fang Luo, and Xing Xie. Evaluating General-Purpose AI with Psychometrics. arXiv preprint arXiv:2310.16379, 2023d. — [arXiv:2310.16379](https://arxiv.org/abs/2310.16379) find
118. Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei. Reading tea leaves: How humans interpret topic models. Advances in neural information processing systems, 22, 2009.
119. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021a.
120. Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint arXiv:2402.05162, 2024.
121. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models. arXiv preprint arXiv:2310.08744, 2023b. find
122. Damien Teney, Ehsan Abbasnejad, Kushal Kafle, Robik Shrestha, Christopher Kanan, and Anton Van Den Hengel. On the value of out-of-distribution testing: An example of goodhart’s law. Advances in neural information processing systems, 33:407–417, 2020. find
123. Ryan Burnell, Wout Schellaert, John Burden, Tomer D Ullman, Fernando Martinez-Plumed, Joshua B Tenenbaum, Danaja Rutar, Lucy G Cheke, Jascha Sohl-Dickstein, Melanie Mitchell, et al. Rethink reporting of evaluation results in AI. Science, 380(6641):136–138, 2023b. find
124. Qiming Bao, Gaël Gendron, Alex Yuxuan Peng, Wanjun Zhong, Neset Tan, Yang Chen, Michael Witbrock, and Jiamou Liu. A Systematic Evaluation of Large Language Models on Out-of-Distribution Logical Reasoning Tasks. arXiv preprint arXiv:2310.09430, 2023. — [arXiv:2310.09430](https://arxiv.org/abs/2310.09430) find
125. Emanuele La Malfa, Aleksandar Petrov, Simon Frieder, Christoph Weinhuber, Ryan Burnell, Anthony G Cohn, Nigel Shadbolt, and Michael Wooldridge. Language Models as a Service: Overview of a new paradigm and its challenges. arXiv preprint arXiv:2309.16573, 2023. — [arXiv:2309.16573](https://arxiv.org/abs/2309.16573) find
126. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020a.
127. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021b.
128. Minje Choi, Jiaxin Pei, Sagar Kumar, Chang Shu, and David Jurgens. Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark. arXiv preprint arXiv:2305.14938, 2023a. find
129. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023a.
130. Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the Tool Manipulation Capability of Open-source Large Language Models. arXiv preprint arXiv:2305.16504, 2023a.
131. Rahul Ramesh, Mikail Khona, Robert P Dick, Hidenori Tanaka, and Ekdeep Singh Lubana. How Capable Can a Transformer Become? A Study on Synthetic, Interpretable Tasks. arXiv preprint arXiv:2311.12997, 2023. — [arXiv:2311.12997](https://arxiv.org/abs/2311.12997) find
132. Pablo Antonio Moreno Casares, Bao Sheng Loe, John Burden, José Hernández-Orallo, et al. How general-purpose is a language model? Usefulness and safety with human prompters in the wild. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 5295–5303, 2022. find
133. Qinyuan Ye, Harvey Yiyun Fu, Xiang Ren, and Robin Jia. How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench. arXiv preprint arXiv:2305.14947, 2023. find
134. Edwin A Fleishman, Marilyn K Quaintance, and Laurie A Broedling. Taxonomies of human performance: The description of human tasks. Academic Press, 1984. find
135. OpenAI. GPT-4 technical report. arXiv, pages 2303–08774, 2023a.
136. Gemini Team. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
137. Anthropic. Introducing Claude, 2023b. https://www.anthropic.com/index/introducing-claude. Accessed on: June 21, 2023. ↗
138. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024a. find
139. W Joel Schneider and Kevin S McGrew. The Cattell-Horn-Carroll model of intelligence. In Contemporary intellectual assessment: Theories, tests, and issues, pages 99–144. The Guilford Press, 2012. find
140. Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36, 2024a. find
141. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023b. find
142. Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. GPT-4 Doesn’t Know It’s Wrong: An Analysis of Iterative Prompting for Reasoning Problems. arXiv preprint arXiv:2310.12397, 2023. — [arXiv:2310.12397](https://arxiv.org/abs/2310.12397) find
143. Zining Zhu and Frank Rudzicz. An information theoretic view on selecting linguistic probes. arXiv preprint arXiv:2009.07364, 2020. find
144. Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. Can Large Language Models Really Improve by Self-critiquing Their Own Plans? arXiv preprint arXiv:2310.08118, 2023. — [arXiv:2310.08118](https://arxiv.org/abs/2310.08118) find
145. Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842, 2023.
146. Yoav Shoham and Kevin Leyton-Brown. Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press, 2008. find
147. Samuel R Bowman. Eight things to know about large language models. arXiv:2304.00612, 2023. arXiv preprint Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701, 2021. find
148. Tom Viering and Marco Loog. The shape of learning curves: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. find
149. Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Learnability and the Vapnik-Chervonenkis dimension. Journal of the ACM (JACM), 36(4):929–965, 1989. find
150. Olivier Bousquet, Steve Hanneke, Shay Moran, Ramon Van Handel, and Amir Yehudayoff. A theory of universal learning. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 532–541, 2021. find
151. Hyunjune Sebastian Seung, Haim Sompolinsky, and Naftali Tishby. Statistical mechanics of learning from examples. Physical review A, 45(8):6056, 1992. find
152. Timothy LH Watkin, Albrecht Rau, and Michael Biehl. The statistical mechanics of learning a rule. Reviews of Modern Physics, 65(2):499, 1993. find
153. Shun-Ichi Amari. A universal theorem on learning curves. Neural networks, 6(2):161–166, 1993. find
154. David Haussler, H Sebastian Seung, Michael Kearns, and Naftali Tishby. Rigorous learning curve bounds from statistical mechanics. In Proceedings of the seventh annual conference on Computational learning theory, pages 76–87, 1994. find
155. Utkarsh Sharma and Jared Kaplan. Scaling Laws from the Data Manifold Dimension. J. Mach. Learn. Res., 23(1), jan 2022. ISSN 1532-4435. find
156. Gal Kaplun, Nikhil Ghosh, Saurabh Garg, Boaz Barak, and Preetum Nakkiran. Deconstructing Distributions: A Pointwise Framework of Learning, 2022. find
157. Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pages 1024–1034. Pmlr, 2020. find
158. Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data versus teacher–student paradigm. Journal of Statistical Mechanics: Theory and Experiment, 2020(12):124001, 2020. find
159. Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018. find
160. Marcus Hutter. Learning curve theory. arXiv preprint arXiv:2102.04074, 2021. — [arXiv:2102.04074](https://arxiv.org/abs/2102.04074) find
161. Eric J Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. arXiv preprint arXiv:2303.13506, 2023.
162. Łukasz Dębowski. A simplistic model of neural scaling laws: Multiperiodic Santa Fe processes. arXiv preprint arXiv:2302.09049, 2023. — [arXiv:2302.09049](https://arxiv.org/abs/2302.09049) find
163. Vivien Cabannes, Elvis Dohmatob, and Alberto Bietti. Scaling laws for associative memories. arXiv preprint arXiv:2310.02984, 2023. — [arXiv:2310.02984](https://arxiv.org/abs/2310.02984) find
164. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems,… find
165. Emmanuel Abbe, Enric Boix-Adsera, Matthew S Brennan, Guy Bresler, and Dheeraj Nagaraj. The staircase property: How hierarchical structure can guide deep learning. Advances in Neural Information Processing Systems, 34:26989–27002, 2021. find
166. Ibrahim Alabdulmohsin, Vinh Q Tran, and Mostafa Dehghani. Fractal Patterns May Unravel the Intelligence in Next-Token Prediction. arXiv preprint arXiv:2402.01825, 2024. — [arXiv:2402.01825](https://arxiv.org/abs/2402.01825) find
167. Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. arXiv preprint arXiv:2301.13442, 2023. find
168. Jens Tuyls, Dhruv Madeka, Kari Torkkola, Dean Foster, Karthik Narasimhan, and Sham Kakade. Scaling laws for imitation learning in nethack. arXiv preprint arXiv:2307.09423, 2023. — [arXiv:2307.09423](https://arxiv.org/abs/2307.09423) find
169. Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, et al. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608, 2023. find
170. Johan Obando-Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of Experts Unlock Parameter Scaling for Deep RL. arXiv preprint arXiv:2402.08609, 2024. — [arXiv:2402.08609](https://arxiv.org/abs/2402.08609) find
171. Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, et al. Predicting Emergent Abilities with Infinite Resolution Evaluation. arXiv e-prints, pages arXiv–2310, 2023a. find
172. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
173. Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken Neural Scaling Laws. In The Eleventh International Conference on Learning Representations, 2023. URL https://arxiv.org/ab s/2210.14891. ↗
174. Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. arXiv preprint arXiv:2302.03025, 2023. — [arXiv:2302.03025](https://arxiv.org/abs/2302.03025) find
175. R Thomas McCoy, Junghyun Min, and Tal Linzen. BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance. arXiv preprint arXiv:1911.02969, 2019. find
176. Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. Advances in neural information processing systems, 31, 2018. find
177. Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? arXiv preprint arXiv:1511.07543, 2015. — [arXiv:1511.07543](https://arxiv.org/abs/1511.07543) find
178. Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances in neural information processing systems, 34:225–236, 2021. find
179. Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild. arXiv preprint arXiv:2312.09230, 2023. find
180. Nikhil Vyas, Alexander Atanasov, Blake Bordelon, Depen Morwani, Sabarish Sainathan, and Cengiz Pehlevan. Feature-Learning Networks Are Consistent Across Widths At Realistic Scales. arXiv preprint arXiv:2305.18411, 2023. — [arXiv:2305.18411](https://arxiv.org/abs/2305.18411) find
181. Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth. In International Conference on Learning Representations, 2020. find
182. Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C Love, Erin Grant, Jascha Achterberg, Joshua B Tenenbaum, et al. Getting aligned on representational alignment. arXiv preprint arXiv:2310.13018, 2023. — [arXiv:2310.13018](https://arxiv.org/abs/2310.13018) find
183. Cheng-Han Chiang and Hung-yi Lee. Can Large Language Models Be an Alternative to Human Evaluations? arXiv preprint arXiv:2305.01937, 2023. find
184. J. Zhu, H. Yan, and T. L. Griffiths. Recovering Mental Representations from Large Language Models with Markov Chain Monte Carlo. arXiv preprint arXiv:2401.16657, 2024. — [arXiv:2401.16657](https://arxiv.org/abs/2401.16657) find
185. Gati Aher, Rosa I Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans. arXiv preprint arXiv:2208.10264, 2022. find
186. Lindia Tjuatja, Valerie Chen, Sherry Tongshuang Wu, Ameet Talwalkar, and Graham Neubig. Do LLMs exhibit human-like response biases? A case study in survey design. arXiv preprint arXiv:2311.04076, 2023.
187. Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT. Nature Computational Science, 3 (10):833–838, 2023. find
188. Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746–751, 2013.
189. Malvina Nissim, Rik van Noord, and Rob van der Goot. Fair is better than sensational: Man is to doctor as woman is to doctor. Computational Linguistics, 46(2):487–497, 2020.
190. Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geometry of Large Language Models. ArXiv, abs/2311.03658, 2023b. URL https://api.semanticscholar.org/CorpusID:265042984. ↗
191. Danny Hernandez and Tom B Brown. Measuring the algorithmic efficiency of neural networks. arXiv preprint arXiv:2005.04305, 2020. — [arXiv:2005.04305](https://arxiv.org/abs/2005.04305) find
192. Ege Erdil and Tamay Besiroglu. arXiv:2212.05153, 2022. — [arXiv:2212.05153](https://arxiv.org/abs/2212.05153) find
193. Algorithmic progress in computer vision. arXiv preprint Melanie Mitchell, Alessandro B. Palmarini, and Arseny Moskvichev. Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks. CoRR, abs/2311.09247, 2023. — [arXiv:2311.09247](https://arxiv.org/abs/2311.09247) find
194. Robert Kirk and David Scott Krueger. Causal confusion as an argument against the scaling hypothesis. AI Alignment Forum, 2022. https://www.alignmentforum.org/posts/FZL4ftXvcuKmmobmj/cau sal-confusion-as-an-argument-against-the-scaling. ↗
195. Matej Zecevic, Moritz Willig, Devendra Singh Dhami, and Kristian Kersting. Causal Parrots: Large Language Models May Talk Causality But Are Not Causal. CoRR, abs/2308.13067, 2023. find
196. Andrew Kyle Lampinen, Stephanie CY Chan, Ishita Dasgupta, Andrew J Nam, and Jane X Wang. Passive learning of active causal strategies in agents and language models. arXiv preprint arXiv:2305.16183, 2023. — [arXiv:2305.16183](https://arxiv.org/abs/2305.16183) find
197. Adam Tauman Kalai and Santosh S Vempala. Calibrated language models must hallucinate. arXiv preprint arXiv:2311.14648, 2023. — [arXiv:2311.14648](https://arxiv.org/abs/2311.14648) find
198. Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is Inevitable: An Innate Limitation of Large Language Models. arXiv preprint arXiv:2401.11817, 2024a.
199. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How Does LLM Safety Training Fail? arXiv preprint arXiv:2307.02483, 2023c.
200. Spencer Frei, Gal Vardi, Peter L Bartlett, and Nathan Srebro. The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networks. arXiv preprint arXiv:2303.01456, 2023. — [arXiv:2303.01456](https://arxiv.org/abs/2303.01456) find
201. Edoardo Debenedetti, Zishen Wan, Maksym Andriushchenko, Vikash Sehwag, Kshitij Bhardwaj, and Bhavya Kailkhura. Scaling Compute Is Not All You Need for Adversarial Robustness. arXiv preprint arXiv:2312.13131, 2023. — [arXiv:2312.13131](https://arxiv.org/abs/2312.13131) find
202. Ian R McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, et al. Inverse Scaling: When Bigger Isn’t Better. arXiv preprint arXiv:2306.09479, 2023.
203. Jason Wei, Yi Tay, and Quoc V Le. arXiv:2211.02011, 2022b. — [arXiv:2211.02011](https://arxiv.org/abs/2211.02011) find
204. Inverse scaling can become U-shaped. arXiv preprint Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv…
205. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of Large Language Models a mirage? arXiv preprint arXiv:2304.15004, 2023.
206. Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936, 2023. — [arXiv:2307.15936](https://arxiv.org/abs/2307.15936) find
207. Maya Okawa, Ekdeep Singh Lubana, Robert P Dick, and Hidenori Tanaka. Compositional abilities emerge multiplicatively: Exploring diffusion models on a synthetic task. arXiv preprint arXiv:2310.09336, 2023. — [arXiv:2310.09336](https://arxiv.org/abs/2310.09336) find
208. Boaz Barak. Emergent abilities and grokking: Fundamental, Mirage, or both?, 2023. https://windowso ntheory.org/2023/12/22/emergent-abilities-and-grokking-fundamental-mirage-or-both/. find
209. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, sep 2022. URL https://openreview.net/forum?id=9XFSbDPmdW. ↗
210. Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: SGD learns parities near the computational limit. Advances in Neural Information Processing Systems, 35:21750–21764, 2022. find
211. Jason Wei. 137 emergent abilities of large language models, 2022. https://www.jasonwei.net/blog/ emergence. Accessed on: October 20, 2023. ↗
212. Michael Hu, Angelica Chen, Naomi Saphra, and Kyunghyun Cho. Latent State Transitions in Training Dynamics. Transactions of Machine Learning Research (TMLR), 2023b. find
213. Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L. Leavitt, and Naomi Saphra. Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs. In The Twelfth International Conference on Learning Representations, 2024b. URL https://openreview .net/forum?id=MO5PiK… find
214. Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet. The developmental landscape of in-context learning. arXiv preprint arXiv:2402.02364, 2024. — [arXiv:2402.02364](https://arxiv.org/abs/2402.02364) find
215. Benjamin L Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis. The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains. arXiv preprint arXiv:2402.11004, 2024. — [arXiv:2402.11004](https://arxiv.org/abs/2402.11004) find
216. Jesse Hoogland, Alexander Gietelink Oldenziel, Daniel Murfet, and Stan van Wingerden. Towards Developmental Interpretability. AI Alignment Forum, 2023. https://www.alignmentforum.org/p osts/TjaeCWvLZtEDAS5Ex/towards-developmental-interpretability. Accessed on: 1 February, 2024. ↗
217. Susan Wei, Daniel Murfet, Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas Quella. Deep learning is singular, and that’s good. IEEE Transactions on Neural Networks and Learning Systems, 2022c. find
218. Sumio Watanabe. Recent advances in algebraic geometry and Bayesian statistics. Information Geometry, 2024. find
219. Kerson Huang. Statistical mechanics. John Wiley & Sons, 2008. find
220. Geoffrey Grimmett. Probability on graphs: random processes on graphs and lattices, volume 8. Cambridge University Press, 2018. find
221. Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300–22312, 2022. find
222. Shoaib Ahmed Siddiqui, Nitarshan Rajkumar, Tegan Maharaj, David Krueger, and Sara Hooker. Metadata archaeology: Unearthing data subsets by leveraging training dynamics. arXiv preprint arXiv:2209.10015, 2022. — [arXiv:2209.10015](https://arxiv.org/abs/2209.10015) find
223. Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Freeze-thaw Bayesian optimization. arXiv preprint arXiv:1406.3896, 2014. — [arXiv:1406.3896](https://arxiv.org/abs/1406.3896) find
224. Michael PH Stumpf and Mason A Porter. Critical truths about power laws. Science, 335(6069):665–666, 2012. find
225. Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7, 2021.
226. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022d.
227. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021.
228. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213, 2022.
229. Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023,…
230. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D Hwang, et al. Faith and Fate: Limits of Transformers on Compositionality. arXiv preprint arXiv:2305.18654, 2023.
231. Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large Language Models Still Can’t Plan (A Benchmark for LLMs on Planning and Reasoning about Change). arXiv preprint arXiv:2206.10498, 2022.
232. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks. CoRR, abs/2307.02477, 2023a. URL https://doi.org/10.48550/arXiv.2… ↗
233. Abulhair Saparov and He He. Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2022. URL https://openreview.net/pdf?id=qFVVBzXxR2V. ↗
234. Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”. CoRR, abs/2309.12288, 2023a. find
235. Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403, 2022.
236. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving Quantitative Reasoning Problems with Language Models. In NeurIPS, 2022.
237. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020.
238. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023a.
239. OpenAI. GPT-4 System Card. Technical report, OpenAI, 2023b. URL ^1^. find
240. Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, ..., and Dragomir Radev. FOLIO: Natural Language… — FOLIO: Natural Language Reasoning with First-Order Logic
241. Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning. CoRR, abs/2310.16049, 2023.
242. Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, and Mrinmaya Sachan. Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models. CoRR, abs/2310.14491, 2023. find
243. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive Commonsense Reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
244. Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. Understanding Social Reasoning in Language Models with Language Models. CoRR, abs/2306.15448, 2023. find
245. C. S. Peirce. Questions Concerning Certain Faculties Claimed for Man. The Journal of Speculative Philosophy, 2(2):103–114, 1868. find
246. Nathan D. Smith. Introduction to Philosophy. OpenStax, Rice University, Houston, Texas, 2022. find
247. Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement. CoRR, abs/2310.08559, 2023. find
248. Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D. Goodman. Hypothesis Search: Inductive Reasoning with Language Models. CoRR, abs/2309.05660, 2023f. find
249. Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large Language Models can Learn Rules. CoRR, abs/2310.07064, 2023. — [arXiv:2310.07064](https://arxiv.org/abs/2310.07064) find
250. Youssef Benchekroun, Megi Dervishi, Mark Ibrahim, Jean-Baptiste Gaya, Xavier Martinet, Grégoire Mialon, Thomas Scialom, Emmanuel Dupoux, Dieuwke Hupkes, and Pascal Vincent. WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models. CoRR, abs/2311.15930, 2023.
251. Wes Gurnee and Max Tegmark. Language Models Represent Space and Time. CoRR, abs/2310.02207, 2023. find
252. Jonathan Roberts, Timo Lüddecke, Sowmen Das, Kai Han, and Samuel Albanie. GPT4GEO: How a Language Model Sees the World’s Geography. arXiv preprint arXiv:2306.00020, 2023a. find
253. Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, and Katia Sycara. Theory of mind for multi-agent collaboration via large language models. arXiv preprint arXiv:2310.10701, 2023a.
254. Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez, Max Kleiman-Weiner, Mrinmaya Sachan, et al. CLADDER: Assessing Causal Reasoning in Language Models, 2023. find
255. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s Verify Step by Step. arXiv preprint arXiv:2305.20050, 2023.
256. Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching Small Language Models to Reason. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2:…
257. Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, et al. Orca 2: Teaching Small Language Models How to Reason. arXiv preprint arXiv:2311.11045, 2023.
258. Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language Models of Code are Few-Shot Commonsense Learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1384–1403.…
259. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hu… find
260. Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying Large Language Model Generalizatio… — Studying Large Language Model Generalization with Influence Functions
261. Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. On the Paradox of Learning to Reason from Data. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, pages 3365–3373, 202… find
262. William Merrill and Ashish Sabharwal. The parallelism tradeoff: Limitations of log-precision transformers. Transactions of the Association for Computational Linguistics, 11:531–545, 2023a. find
263. William Merrill, Nikolaos Tsilivis, and Aman Shukla. A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks. arXiv preprint arXiv:2303.11873, 2023. — [arXiv:2303.11873](https://arxiv.org/abs/2303.11873) find
264. William Merrill and Ashish Sabharwal. Transformers implement first-order logic with majority quantifiers. arXiv preprint arXiv:2210.02671, 2022. find
265. Lena Strobl. Average-Hard Attention Transformers are Constant-Depth Uniform Threshold Circuits. arXiv preprint arXiv:2308.03212, 2023. — [arXiv:2308.03212](https://arxiv.org/abs/2308.03212) find
266. Guhao Feng, Yuntian Gu, Bohang Zhang, Haotian Ye, Di He, and Liwei Wang. Towards Revealing the Mystery behind Chain of Thought: a Theoretical Perspective. arXiv preprint arXiv:2305.15408, 2023. find
267. William Merrill and Ashish Sabharwal. The Expresssive Power of Transformers with Chain of Thought. arXiv preprint arXiv:2310.07923, 2023b. find
268. Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking Like Transformers. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 11080–11090. find
269. Dana Angluin, David Chiang, and Andy Yang. Masked Hard-Attention Transformers and Boolean RASP Recognize Exactly the Star-Free Languages. arXiv preprint arXiv:2310.13897, 2023. — [arXiv:2310.13897](https://arxiv.org/abs/2310.13897) find
270. David Chiang, Peter Cholak, and Anand Pillay. Tighter bounds on the expressivity of transformer encoders. In International Conference on Machine Learning, pages 5544–5562. PMLR, 2023. find
271. William Merrill and Ashish Sabharwal. A logic for expressing log-precision transformers. Advances in Neural Information Processing Systems, 36, 2024. find
272. Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh M. Susskind, Samy Bengio, and Preetum Nakkiran. What Algorithms can Transformers Learn? A Study in Length Generalization. CoRR, abs/2310.16028, 2023c.
273. Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. Identifying the Risks of LM Agents with an LM-Emulated Sandbox. arXiv preprint arXiv:2309.15817, 2023.
274. ARC. ARC Evals, 2022. URL https://evals.alignment.org/. ↗
275. Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915, 2023a.
276. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022a. find
277. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, ..., and Andy Zeng. Do… — Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
278. Anton Osika. GPT Engineer, 2023. URL https://github.com/AntonOsika/gpt-engineer. Initial release: June 10, 2023. ↗
279. Michael Cohen, Marcus Hutter, and Michael Osborne. Advanced artificial agents intervene in the provision of reward. AI magazine, 43(3):282–293, 2022. find
280. Joar Skalse, Nikolaus HR Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. arXiv preprint arXiv:2209.13085, 2022. — [arXiv:2209.13085](https://arxiv.org/abs/2209.13085) find
281. Lauro Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning, pages 12004–12019. Pmlr, 2022. find
282. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! arXiv preprint arXiv:2310.03693, 2023a.
283. Herbert P Grice. Logic and conversation. In Speech acts, pages 41–58. Brill, 1975. find
284. Steven T Piantadosi, Harry Tily, and Edward Gibson. The communicative function of ambiguity in language. Cognition, 122(3):280–291, 2012. find
285. Murray Shanahan. The Frame Problem. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Spring 2016 edition, 2016. find
286. Alexander Nicholas D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jon Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormozdiari, Shaobo Hou, Neil Houlsby, Ghassen Jerfel, Alan Karthikesalingam, Mario Lučić, Yian Ma, Cory McLean, Diana Mincu,… find
287. Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. AI safety gridworlds. arXiv preprint arXiv:1711.09883, 2017. — [arXiv:1711.09883](https://arxiv.org/abs/1711.09883) find
288. Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander, Pieter Abbeel, and Anca Dragan. Preferences implicit in the state of the world. arXiv preprint arXiv:1902.04198, 2019. — [arXiv:1902.04198](https://arxiv.org/abs/1902.04198) find
289. Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, and Shane Legg. Avoiding side effects by considering future tasks. Advances in Neural Information Processing Systems, 33:19064–19074, 2020. find
290. Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29, 2016. find
291. Rohin Shah, Pedro Freire, Neel Alex, Rachel Freedman, Dmitrii Krasheninnikov, Lawrence Chan, Michael D Dennis, Pieter Abbeel, Anca Dragan, and Stuart Russell. Benefits of assistance over reward learning, 2020. find
292. Alexander Matt Turner, Dylan Hadfield-Menell, and Prasad Tadepalli. Conservative agency via attainable utility preservation. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 385–391, 2020. find
293. Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In International conference on machine learning, pages 7390–7399. Pmlr, 2021. find
294. Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, Chenxue Wang, Shichao Liu, and Qing Wang. ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarification. arXiv preprint arXiv:2310.10996, 2023. — [arXiv:2310.10996](https://arxiv.org/abs/2310.10996) find
295. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Clam: Selective clarification for ambiguous questions with large language models. arXiv preprint arXiv:2212.07769, 2022. find
296. Joshua Clymer, Garrett Baker, Rohan Subramani, and Sam Wang. Generalization Analogies (GE-NIES): A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains. arXiv preprint arXiv:2311.07723, 2023. — [arXiv:2311.07723](https://arxiv.org/abs/2311.07723) find
297. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, ..., and Jared Kapla… find
298. Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do Large Language Models Know What They Don’t Know? arXiv preprint arXiv:2305.18153, 2023. find
299. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664, 2023.
300. Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward Model Ensembles Help Mitigate Overoptimization. arXiv preprint arXiv:2310.02743, 2023. — [arXiv:2310.02743](https://arxiv.org/abs/2310.02743) find
301. Zaynah Javed, Daniel S Brown, Satvik Sharma, Jerry Zhu, Ashwin Balakrishna, Marek Petrik, Anca Dragan, and Ken Goldberg. Policy gradient bayesian robust optimization for imitation learning. In International Conference on Machine Learning, pages 4785–4796. PMLR, 2021. find
302. Dmitrii Krasheninnikov, Egor Krasheninnikov, and David Krueger. Assistance with large language models. In NeurIPS ML Safety Workshop, 2022. URL https://openreview.net/forum?id=OE9V 81spp6B. ↗
303. David Krueger, Jan Leike, Owain Evans, and John Salvatier. Active reinforcement learning: Observing rewards at a cost. arXiv preprint arXiv:2011.06709, 2020. — [arXiv:2011.06709](https://arxiv.org/abs/2011.06709) find
304. Francis Rhys Ward, Francesca Toni, Francesco Belardinelli, and Tom Everitt. Honesty Is the Best Policy: Defining and Mitigating AI Deception. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=EmxpDiPgRu. ↗
305. Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The Off-Switch Game. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17), 2017. find
306. Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Jonathan Ng, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark. Icml, 2023a. find
307. Jérémy Scheurer, Mikita Balesni, and Marius Hobbhahn. Technical Report: Large Language Models can Strategically Deceive their Users when Put Under Pressure, 2023a. find
308. Lorenzo Pacchiardi, Alex J Chan, Sören Mindermann, Ilan Moscovitz, Alexa Y Pan, Yarin Gal, Owain Evans, and Jan Brauner. How to catch an AI liar: Lie detection in black-box LLMs by asking unrelated questions. arXiv preprint arXiv:2309.15840, 2023.
309. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering Latent Knowledge in Language Models Without Supervision, dec 2022. URL http://arxiv.org/abs/2212.03827. arXiv:2212.03827 [cs]. ↗
310. Amos Azaria and Tom Mitchell. The Internal State of an LLM Knows When It’s Lying, oct 2023. URL http://arxiv.org/abs/2304.13734. arXiv:2304.13734 [cs]. ↗
311. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model, oct 2023b. URL http://arxiv.org/abs/2306.03341. arXiv:2306.03341 [cs]. ↗
312. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, ..., and Dan Hendrycks. Repres… — Representation Engineering: A Top-Down Approach to AI Transparency
313. Silen Naihin, David Atkinson, Marc Green, Merwane Hamadi, Craig Swift, Douglas Schonholtz, Adam Tauman Kalai, and David Bau. Testing Language Model Agents Safely in the Wild. arXiv preprint arXiv:2311.10538, 2023. — [arXiv:2311.10538](https://arxiv.org/abs/2311.10538) find
314. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544, 2022. find
315. Dmitrii Krasheninnikov, Egor Krasheninnikov, Bruno Mlodozeniec, and David Krueger. Meta-(out-of-context) learning in neural networks. arXiv preprint arXiv:2310.15047, 2023. — [arXiv:2310.15047](https://arxiv.org/abs/2310.15047) find
316. Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, and Owain Evans. Taken out of context: On measuring situational awareness in LLMs, 2023b. find
317. Alexander Meinke. LLMs can Spontaneously start Jailbreaking their Scoring Function, 2023. URL https://alignmentjam.com/project/jailbreaking-the-overseer. ↗
318. Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. AI Control: Improving Safety Despite Intentional Subversion, January 2024. URL http://arxiv.org/abs/2312.06942. arXiv:2312.06942 [cs] version: 3. — [arXiv:2312.06942](https://arxiv.org/abs/2312.06942) ↗
319. Christian Schroeder de Witt, Samuel Sokota, J. Zico Kolter, Jakob Nicolaus Foerster, and Martin Strohmeier. Perfectly Secure Steganography Using Minimum Entropy Coupling. September 2023. URL https://openreview.net/forum?id=HQ67mj5rJdR. ↗
320. Sumeet Ramesh Motwani, Mikhail Baranchuk, Martin Strohmeier, Vijay Bolina, Philip HS Torr, Lewis Hammond, and Christian Schroeder de Witt. Secret Collusion Among Generative AI Agents. arXiv:2402.07510, 2024. — [arXiv:2402.07510](https://arxiv.org/abs/2402.07510) find
321. Ian Goodfellow. A research agenda: Dynamic models to defend against correlated attacks. arXiv preprint arXiv:1903.06293, 2019. — [arXiv:1903.06293](https://arxiv.org/abs/1903.06293) find
322. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.
323. Toran Bruce Richards. Auto-GPT, 2023. URL https://github.com/Significant-Gravitas/Auto -GPT. ↗
324. Alexander Pan, Erik Jones, Meena Jagadeesan, and Jacob Steinhardt. Feedback Loops Drive In-Context Reward Hacking in LLMs. arXiv, 2024. find
325. Thomas C. Schelling. The Strategy of Conflict: With a New Preface by the Author. Harvard University Press, Cambridge, MA, may 1981. ISBN 9780674840317. find
326. John C. Harsanyi. A new theory of equilibrium selection for games with complete information. Games and Economic Behavior, 8(1):91–122, jan 1995. ISSN 0899-8256. doi: 10.1016/s0899-8256(05)80018-1. URL https://www.sciencedirect.com/science/article/pii/S0899825605800181. — [doi:10.1016/s0899-8256(05)80018-1](https://doi.org/10.1016/s0899-8256%2805%2980018-1) ↗
327. Tim Roughgarden. Selfish Routing and The Price of Anarchy. MIT Press, Cambridge, Mass, 2005. ISBN 9780262182430. OCLC: ocm56068857. find
328. Noam Nisan, editor. Algorithmic game theory. Cambridge University Press, Cambridge ; New York, 2007. ISBN 9780521872829. OCLC: ocn122526907. find
329. Francis M. Bator. The Anatomy of Market Failure. The Quarterly Journal of Economics, 72(3):351–379, 1958. ISSN 0033-5533. doi: 10.2307/1882231. URL https://www.jstor.org/stable/1882231. — [doi:10.2307/1882231](https://doi.org/10.2307/1882231) ↗
330. R. H. Coase. The Problem of Social Cost. The Journal of Law and Economics, 3:1–44, 1960. ISSN 0022-2186, 1537-5285. doi: 10.1086/466560. URL https://www.journals.uchicago.edu/doi/10 .1086/466560. — [doi:10.1086/466560](https://doi.org/10.1086/466560) ↗
331. James M. Buchanan and Wm. Craig Stubblebine. Externality. Economica, 29(116):371–384, 1962. ISSN 0013-0427. doi: 10.2307/2551386. URL https://www.jstor.org/stable/2551386. — [doi:10.2307/2551386](https://doi.org/10.2307/2551386) ↗
332. Israel M. Kirzner. Market Theory and the Price System. Liberty Fund, 1963. ISBN 9780865977600. Google-Books-ID: Pj96cgAACAAJ. find
333. Pradeep Dubey. Inefficiency of Nash Equilibria. Mathematics of Operations Research, 11(1):1–8, 1986. ISSN 0364-765x. URL https://www.jstor.org/stable/3690047. ↗
334. Adrien Ecoffet, Jeff Clune, and Joel Lehman. Open questions in creating safe open-ended AI: tensions between control and creativity. In Artificial Life Conference Proceedings 32, pages 27–35, 2020. find
335. Oliver Sourbut, Lewis Hammond, and Harriet Wood. Cooperation and control in delegation games. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-2024, 2024. doi: 10.24963/ijcai.2024/26. — [doi:10.24963/ijcai.2024/26](https://doi.org/10.24963/ijcai.2024/26) find
336. Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R. McKee, Joel Z. Leibo, Kate Larson, and Thore Graepel. Open Problems in Cooperative AI, dec 2020. URL http://arxiv.org/ abs/2012.08630. arXiv:2012.08630 [cs]. — [arXiv:2012.08630](https://arxiv.org/abs/2012.08630) ↗
337. Vincent Conitzer and Caspar Oesterheld. Foundations of cooperative AI. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15359–15367, 2023. find
338. Lewis Hammond et al. Multi-Agent Risks from Advanced AI, 2024. Forthcoming. find
339. Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.04658, 2023b.
340. Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from AI feedback. arXiv preprint arXiv:2305.10142, 2023b.
341. Gabriel Mukobi, Hannah Erlebach, Niklas Lauffer, Lewis Hammond, Alan Chan, and Jesse Clifton. Welfare Diplomacy: Benchmarking Language Model Cooperation. arXiv:2310.08901, oct 2023. doi: 10.48550/arxiv.2310.08901. — [doi:10.48550/arxiv.2310.08901](https://doi.org/10.48550/arxiv.2310.08901) find
342. Alan Chan, Maxime Riché, and Jesse Clifton. Towards the Scalable Evaluation of Cooperativeness in Language Models. arXiv:2303.13360, mar 2023b. doi: 10.48550/arxiv.2303.13360. — [doi:10.48550/arxiv.2303.13360](https://doi.org/10.48550/arxiv.2303.13360) find
343. Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with Large Language Models. arXiv:2305.16867, may 2023. doi: 10.48550/arxiv.230 5.16867. — [doi:10.48550/arxiv.230](https://doi.org/10.48550/arxiv.230) find
344. Andrew Critch, Michael Dennis, and Stuart Russell. Cooperative and uncooperative institution designs: Surprises and problems in open-source game theory, aug 2022. URL http://arxiv.org/abs/2208 .07006. arXiv:2208.07006 [cs]. — [arXiv:2208.07006](https://arxiv.org/abs/2208.07006) ↗
345. Caspar Oesterheld, Johannes Treutlein, Roger Grosse, Vincent Conitzer, and Jakob Foerster. Similarity-based cooperative equilibrium, nov 2023. URL http://arxiv.org/abs/2211.14468. arXiv:2211.14468 [cs]. — [arXiv:2211.14468](https://arxiv.org/abs/2211.14468) ↗
346. Rishi Bommasani, Kathleen A Creel, Ananya Kumar, Dan Jurafsky, and Percy S Liang. Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? Advances in Neural Information Processing Systems, 35:3663–3678, 2022. find
347. Rusheb Shah, Quentin Feuillade-Montixi, Soroush Pour, Arush Tagade, Stephen Casper, and Javier Rando. Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation, 2023. find
348. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models, 2023b.
349. Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role play with large language models. Nature, pages 1–6, 2023. — Role-Play with Large Language Models
350. Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, et al. RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models. arXiv preprint arXiv:2310.00746, 2023g.
351. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022a.
352. Herbie Bradley, Andrew Dai, Hannah Teufel, Jenny Zhang, Koen Oostermeijer, Marco Bellagente, Jeff Clune, Kenneth Stanley, Grégory Schott, and Joel Lehman. Quality-Diversity through AI Feedback. arXiv preprint arXiv:2310.13032, 2023.
353. Li Ding, Jenny Zhang, Jeff Clune, Lee Spector, and Joel Lehman. Quality Diversity through Human Feedback. arXiv preprint arXiv:2310.12103, 2023. — [arXiv:2310.12103](https://arxiv.org/abs/2310.12103) find
354. Jakob Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with Opponent-learning Awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, Aamas ’18, pages 122–130, Richland, SC, 2018. Internatio… find
355. Michael Bradley Johanson, Edward Hughes, Finbarr Timbers, and Joel Z Leibo. Emergent bartering behaviour in multi-agent reinforcement learning. arXiv preprint arXiv:2205.06760, 2022. — [arXiv:2205.06760](https://arxiv.org/abs/2205.06760) find
356. Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent Tool Use From Multi-Agent Autocurricula. arXiv:1909.07528, sep 2019. doi: 10.48550/arxiv.1909.07528. — [doi:10.48550/arxiv.1909.07528](https://doi.org/10.48550/arxiv.1909.07528) find
357. Gurusha Juneja, Subhabrata Dutta, Soumen Chakrabarti, Sunny Manchanda, and Tanmoy Chakraborty. Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoning. arXiv preprint arXiv:2310.18338, 2023. find
358. Vyacheslav I Yukalov and Didier Sornette. Self-organization in complex systems as decision making. Advances in Complex Systems, 17(03n04):1450016, 2014. find
359. Zachary Kenton, Ramana Kumar, Sebastian Farquhar, Jonathan Richens, Matt MacDermott, and Tom Everitt. Discovering Agents. arXiv:2208.08345, August 2022. — [arXiv:2208.08345](https://arxiv.org/abs/2208.08345) find
360. Laurent Orseau, Simon McGregor McGill, and Shane Legg. Agents and devices: A relative definition of agency. arXiv preprint arXiv:1805.12387, 2018. — [arXiv:1805.12387](https://arxiv.org/abs/1805.12387) find
361. Florian E Dorner. Algorithmic collusion: A critical review. arXiv preprint arXiv:2110.04740, 2021. — [arXiv:2110.04740](https://arxiv.org/abs/2110.04740) find
362. Vitalik Buterin. On Collusion, 2019. URL https://vitalik.eth.limo/general/2019/04/03/coll usion.html. ↗
363. Stephanie Assad, Robert Clark, Daniel Ershov, and Lei Xu. Algorithmic Pricing and Competition: Empirical Evidence from the German Retail Gasoline Market. Working Paper 1438, Economics Department, Queen’s University, Aug 2020. URL https://ideas.repec.org/p/qed/wpaper/143 8.html. ↗
364. Marcel Wieting and Geza Sapi. Algorithms in the Marketplace: An Empirical Analysis of Automated Pricing in E-Commerce. Technical Report 21-06, NET Institute, 2021. find
365. Zach Y. Brown and Alexander MacKay. Competition in Pricing Algorithms. American Economic Journal: Microeconomics, 15(2):109–56, 5 2023. doi: 10.1257/mic.20210158. URL https://www.ae aweb.org/articles?id=10.1257/mic.20210158. — [doi:10.1257/mic.20210158](https://doi.org/10.1257/mic.20210158) ↗
366. Emilio Calvano, Giacomo Calzolari, Vincenzo Denicolò, and Sergio Pastorello. Artificial Intelligence, Algorithmic Pricing, and Collusion. American Economic Review, 110(10):3267–97, 10 2020. doi: 10.1257/aer.20190623. URL https://www.aeaweb.org/articles?id=10.1257/aer.20190623. — [doi:10.1257/aer.20190623](https://doi.org/10.1257/aer.20190623) ↗
367. Timo Klein. Autonomous algorithmic collusion: Q-learning under sequential pricing. The RAND Journal of Economics, 52(3):538–558, 2021. doi: https://doi.org/10.1111/1756-2171.12383. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/1756-2171.12383. — [doi:10.1111/1756-2171.12383](https://doi.org/10.1111/1756-2171.12383) ↗
368. Fabien Roger and Ryan Greenblatt. Preventing Language Models From Hiding Their Reasoning. arXiv preprint arXiv:2310.18512, 2023. find
369. Stuart Russell. Human compatible: Artificial intelligence and the problem of control. Penguin, 2019. find
370. Yali Du, Joel Z Leibo, Usman Islam, Richard Willis, and Peter Sunehag. A Review of Cooperation in Multi-agent Learning. arXiv preprint arXiv:2312.05162, 2023a. — [arXiv:2312.05162](https://arxiv.org/abs/2312.05162) find
371. Adam Kalai and Ehud Kalai. Cooperation In Strategic Games Revisited. The Quarterly Journal of Economics, 128(2):917–66, 2013. URL https://econweb.ucsd.edu/~jwatson/PAPERS/SWET2012/K alai-paper.pdf. ↗
372. Andrei Lupu and Doina Precup. Gifting in Multi-Agent Reinforcement Learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 789–797, 2020. find
373. Jiachen Yang, Ang Li, Mehrdad Farajtabar, Peter Sunehag, Edward Hughes, and Hongyuan Zha. Learning to Incentivize Other Learning Agents. In Adaptive and Learning Agents Workshop at AAMAS, 2020. URL https://ala2020.vub.ac.be/papers/ALA2020%5Fpaper%5F27.pdf. ↗
374. Phillip J. K. Christoffersen, Andreas A. Haupt, and Dylan Hadfield-Menell. Get It in Writing: Formal Contracts Mitigate Social Dilemmas in Multi-Agent RL, aug 2023. URL http://arxiv.org/abs/ 2208.10469. arXiv:2208.10469 [cs, econ]. — [arXiv:2208.10469](https://arxiv.org/abs/2208.10469) ↗
375. Manfred Milinski, Dirk Semmann, and Hans-Jürgen Krambeck. Reputation helps solve the ‘tragedy of the commons’. Nature, 415(6870):424–426, jan 2002. ISSN 1476-4687. doi: 10.1038/415424a. URL https://www.nature.com/articles/415424a. — [doi:10.1038/415424a](https://doi.org/10.1038/415424a) ↗
376. Joseph Henrich. Cooperation, Punishment, and the Evolution of Human Institutions. Science, 312 (5770):60–61, apr 2006. ISSN 0036-8075, 1095-9203. doi: 10.1126/science.1126398. URL https://www.science.org/doi/10.1126/science.1126398. — [doi:10.1126/science.1126398](https://doi.org/10.1126/science.1126398) ↗
377. Robert Boyd, Herbert Gintis, and Samuel Bowles. Coordinated Punishment of Defectors Sustains Cooperation and Can Proliferate When Rare. Science, 328(5978):617–620, apr 2010. ISSN 0036-8075, 1095-9203. doi: 10.1126/science.1183665. URL https://www.science.org/doi/10.1126/sc ience.1183665. — [doi:10.1126/science.1183665](https://doi.org/10.1126/science.1183665) ↗
378. Catherine Moon and Vincent Conitzer. Maximal Cooperation in Repeated Games on Social Networks. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJ-CAI), Buenos Aires, Argentina, 2015. URL https://www.ijcai.org/Proceedings/15/Papers/03 7.pdf. ↗
379. David Pardoe, Peter Stone, Maytal Saar-Tsechansky, and Kerem Tomak. Adaptive mechanism design. In Proceedings of the 8th international conference on Electronic commerce The new e-commerce: innovations for conquering current barriers, obstacles and limitations to conducting successful business on the… — [doi:10.1145/1151454.1151480](https://doi.org/10.1145/1151454.1151480) find
380. Jiachen Yang, Ethan Wang, Rakshit Trivedi, Tuo Zhao, and Hongyuan Zha. Adaptive Incentive Design with Multi-Agent Meta-Gradient Reinforcement Learning. arXiv:2112.10859, December 2021. — [arXiv:2112.10859](https://arxiv.org/abs/2112.10859) find
381. Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C. Parkes, and Richard Socher. The AI Economist: Taxation policy design via two-level deep multiagent reinforcement learning. Science Advances, 8(18), may 2022. doi: 10.1126/sciadv.abk2607. find
382. Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Çaglar Gülçehre, Pedro A. Ortega, DJ Strouse, Joel Z. Leibo, and Nando de Freitas. Social Influence As Intrinsic Motivation for Multi-agent Deep Reinforcement Learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th… find
383. Pmlr, 2019. URL http://proceedings.mlr.press/v97/jaques19a.html. ↗
384. Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Dueñez Guzman, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, Heather Roff, and Thore Graepel. Inequity Aversion Improves Cooperation in Intertemporal Social Dilemmas. In S. Bengio, H. Wallach, H. Laroch… find
385. Jane X. Wang, Edward Hughes, Chrisantha Fernando, Wojciech M. Czarnecki, Edgar A. Duéñez Guzmán, and Joel Z. Leibo. Evolving Intrinsic Motivations for Altruistic Behavior. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, Aamas ’19, pages 683–692, Richl… find
386. Kevin R. McKee, Ian Gemp, Brian McWilliams, Edgar A. Duèñez Guzmán, Edward Hughes, and Joel Z. Leibo. Social Diversity and Social Preferences in Mixed-Motive Reinforcement Learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, Aamas ’20, page 869–8… find
387. Julian Yocum, Phillip Christoffersen, Mehul Damani, Justin Svegliato, Dylan Hadfield-Menell, and Stuart Russell. Mitigating Generative Agent Social Dilemmas. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. URL https://openreview.net/forum?id=5TIdOk7XQ6. ↗
388. Erik Jones and Jacob Steinhardt. Capturing failures of large language models via human cognitive biases. Advances in Neural Information Processing Systems, 35:11785–11799, 2022. find
389. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking Cognitive Biases in Large Language Models as Evaluators. arXiv preprint arXiv:2309.17012, 2023. find
390. Hengyuan Hu, Adam Lerer, Brandon Cui, David Wu, Luis Pineda, Noam Brown, and Jakob Foerster. Off-Belief Learning, aug 2021. URL http://arxiv.org/abs/2103.04000. arXiv:2103.04000 [cs]. — [arXiv:2103.04000](https://arxiv.org/abs/2103.04000) ↗
391. Dan Hendrycks and Mantas Mazeika. X-Risk Analysis for AI Research, 2022. find
392. Paul Christiano. Current Work in AI Alignment. EA Global: San Francisco, 2019. find
393. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. find
394. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730– 27744, 2022.
395. Craig Barratt and Stephen Boyd. Example of exact trade-offs in linear controller design. IEEE Control Systems Magazine, 9(1):46–52, 1989. doi: 10.1109/37.16750. — [doi:10.1109/37.16750](https://doi.org/10.1109/37.16750) find
396. Bart De Moor, Johan David, Joos Vandewalle, Maarten De Moor, and Daniel Berckmans. Trade-offs in linear control system design: A practical example. Optimal Control Applications and Methods, 13 (2):121–144, 1992. find
397. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022b. find
398. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the Effects of RLHF on LLM Generalisation and Diversity. arXiv preprint arXiv:2310.06452, 2023a.
399. Heidy Khlaaf. Toward Comprehensive Risk Assessments and Assurance of AI-Based Systems. Trail of Bits, 2023. find
400. Daniel Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Benjamin Weinstein-Raun, Daniel de Haas, et al. Adversarial training for high-stakes reliability. Advances in Neural Information Processing Systems, 35:9274–9286, 2022. find
401. Anthropic. Model Card and Evaluations for Claude Models, 2023c. https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf. ↗
402. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback, 2023.
403. Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. Elo Uncovered: Robustness and Best Practices in Language Model Evaluation. arXiv preprint arXiv:2311.17295, 2023. find
404. David Glukhov, Ilia Shumailov, Yarin Gal, Nicolas Papernot, and Vardan Papyan. LLM Censorship: A Machine Learning Challenge or a Computer Security Problem? arXiv preprint arXiv:2307.10719, 2023. — [arXiv:2307.10719](https://arxiv.org/abs/2307.10719) find
405. Kellin Pelrine, Mohammad Taufeeque, Michał Zając, Euan McLean, and Adam Gleave. Exploiting Novel GPT-4 APIs. arXiv preprint arXiv:2312.14302, 2023. — [arXiv:2312.14302](https://arxiv.org/abs/2312.14302) find
406. Jan Leike. Distinguishing three alignment taxes, 2022a. URL https://aligned.substack.com/p/t hree-alignment-taxes. ↗
407. Tong Wang. Gaining Free or Low-Cost Interpretability with Interpretable Partial Substitute. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6505–6514. Pmlr, 09–… find
408. George Baryannis, Samir Dani, and Grigoris Antoniou. Predicting supply chain risks using machine learning: The trade-off between performance and interpretability. Future Generation Computer Systems, 101:993–1004, 2019. ISSN 0167-739x. doi: https://doi.org/10.1016/j.future.2019.07.059. URL https://ww… — [doi:10.1016/j.future.2019.07.059](https://doi.org/10.1016/j.future.2019.07.059) ↗
409. Gintare Karolina Dziugaite, Shai Ben-David, and Daniel M. Roy. Enforcing Interpretability and its Statistical Impacts: Trade-offs between Accuracy and Interpretability, 2020. find
410. Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, ..., and Christopher Olah. Softm… find
411. Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness May Be at Odds with Accuracy. In International Conference on Learning Representations, 2019. find
412. Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically Principled Trade-off between Robustness and Accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of… find
413. Florian Tramer, Jens Behrmann, Nicholas Carlini, Nicolas Papernot, and Joern-Henrik Jacobsen. Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume… find
414. Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. RobustBench: a standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Tr… find
415. Gwern Branwen. Why Tool AIs Want to Be Agent AIs. Gwern.net, 2016. https://gwern.net/tool-ai. Accessed on: 3 January, 2024. ↗
416. Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv preprint arXiv:2401.05566, 2024.
417. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623, 2021. find
418. Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP), pages 1314–1331. find
419. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023a.
420. Helen Ngo, Cooper Raterink, João G. M. Araújo, Ivan Zhang, Carol Chen, Adrien Morisot, and Nicholas Frosst. Mitigating harm in language models with conditional-likelihood filtration, 2021. URL https://arxiv.org/abs/2108.07790. — [arXiv:2108.07790](https://arxiv.org/abs/2108.07790) ↗
421. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res., 21(1), jun 2022. ISSN 1532-4435. find
422. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online, nov 2020. Association for Computational Lingu…
423. Irene Solaiman and Christy Dennison. Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc., 2021. URL ht…
424. Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in Detoxifying Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2447–2469, Pu…
425. Maribeth Rauh, John Mellor, Jonathan Uesato, Po-Sen Huang, Johannes Welbl, Laura Weidinger, Sumanth Dathathri, Amelia Glaese, Geoffrey Irving, Iason Gabriel, William Isaac, and Lisa Anne Hendricks. Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models, 2022. find
426. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758, 2021. find
427. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistic…
428. Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. Detoxifying Language Models Risks Marginalizing Minority Voices. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p…
429. Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797, 2023.
430. Gabriel Stanovsky, Noah A Smith, and Luke Zettlemoyer. Evaluating gender bias in machine translation. arXiv preprint arXiv:1906.00591, 2019.
431. Sourojit Ghosh and Aylin Caliskan. ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five other Low-Resource Languages. arXiv preprint arXiv:2305.10510, 2023. — [arXiv:2305.10510](https://arxiv.org/abs/2305.10510) find
432. Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. Low-Resource Languages Jailbreak GPT-4, 2023.
433. Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, 2(11), 2021. find
434. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12): 86–92, 2021. find
435. Margaret Mitchell, Alexandra Sasha Luccioni, Nathan Lambert, Marissa Gerchick, Angelina McMillan-Major, Ezinwanne Ozoani, Nazneen Rajani, Tristan Thrush, Yacine Jernite, and Douwe Kiela. Measuring data. arXiv preprint arXiv:2212.05129, 2022. — [arXiv:2212.05129](https://arxiv.org/abs/2212.05129) find
436. Angelina McMillan-Major, Emily M Bender, and Batya Friedman. Data Statements: From Technical Concept to Community Practice. ACM Journal on Responsible Computing, 2023. find
437. Marc Marone and Benjamin Van Durme. Data portraits: Recording foundation model training data. arXiv preprint arXiv:2303.03919, 2023. — [arXiv:2303.03919](https://arxiv.org/abs/2303.03919) find
438. Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. What’s In My Big Data? ArXiv, abs/2310.20707, 2023. URL https://api.semanticscholar.org/CorpusID:264803575. ↗
439. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating Training Data Makes Language Models Better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Associ… find
440. Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697–10707. Pmlr, 2022.
441. Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, and Eneko Agirre. Did ChatGPT cheat on your test?, June 2023. https://hitz-zentroa.github.io/lm-contamination/blog/. ↗
442. Chirag Agarwal, Daniel D’souza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10368–10378, 2022. find
443. Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920– 19930, 2020.
444. Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31, 2018. find
445. Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Predicting predictions from training data. arXiv preprint arXiv:2202.00622, 2022. — [arXiv:2202.00622](https://arxiv.org/abs/2202.00622) find
446. Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186, 2023c. — [arXiv:2303.14186](https://arxiv.org/abs/2303.14186) find
447. Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Towards tracing knowledge in language models back to the training data. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2429–2446, 2022b. find
448. Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If Influence Functions are the Answer, Then What is the Question? Advances in Neural Information Processing Systems, 35: 17953–17967, 2022. find
449. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. Pretraining Language Models with Human Preferences, 2023.
450. Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591–27609, 2022.
451. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021b. find
452. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, ..… — PaLM 2 Technical Report
453. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019.
454. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv preprint arXiv:2305.18290, 2023.
455. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model Alignment as Prospect Theoretic Optimization. arXiv preprint arXiv:2402.01306, 2024. find
456. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, p… find
457. Yangsibo Huang, Samyak Gupta, Zexuan Zhong, Kai Li, and Danqi Chen. Privacy Implications of Retrieval-Based Language Models. arXiv preprint arXiv:2305.14888, 2023a. find
458. Dan Friedman, Alexander Wettig, and Danqi Chen. Learning Transformer Programs. arXiv preprint arXiv:2306.01128, 2023. find
459. Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah D. Goodman, and Christopher Potts. Inducing Causal Structure for Interpretable Neural Networks, jul 2022. URL http://arxiv.org/abs/2112.00826. arXiv:2112.00826 [cs]. ↗
460. Peter Henderson, Eric Mitchell, Christopher Manning, Dan Jurafsky, and Chelsea Finn. Self-destructing models: Increasing the costs of harmful dual uses of foundation models. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 287–296, 2023a. find
461. Abian Pedregosa and Eleni Triantafillou. Announcing the First Machine Unlearning Challenge, 2023. https://blog.research.google/2023/06/announcing-first-machine-unlearning.html. Accessed on: 3 January, 2024. ↗
462. Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. Rethinking Machine Unlearning for Large Language Models. arXiv preprint arXiv:2402.08787, 2024a. find
463. Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models, 2023a.
464. Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B, 2023. find
465. Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing RLHF Protections in GPT-4 via Fine-Tuning, 2023. find
466. Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. “Kelly is a Warm Person, Joseph is a Role Model”: Gender Biases in LLM-Generated Reference Letters. arXiv preprint arXiv:2310.09219, 2023a.
467. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar… find
468. Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. Mechanistic mode connectivity. In International Conference on Machine Learning, pages 22965–23004. Pmlr, 2023. find
469. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity. arXiv preprint arXiv:2401.01967, 2024.
470. Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3 (4):128–135, 1999. find
471. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13): 3521–3526… find
472. Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Continual-t0: Progressively instructing 50+ tasks to language models without forgetting. arXiv preprint arXiv:2205.12393, 2022. find
473. Andrea Cossu, Tinne Tuytelaars, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, and Davide Bacciu. Continual pre-training mitigates forgetting in language and vision. arXiv preprint arXiv:2205.09357, 2022. — [arXiv:2205.09357](https://arxiv.org/abs/2205.09357) find
474. Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding Catastrophic Forgetting in Language Models via Implicit Inference, 2023.
475. Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=GhVS8%5FyPeEa. ↗
476. Duo Li, Guimei Cao, Yunlu Xu, Zhanzhan Cheng, and Yi Niu. Technical report for iccv 2021 challenge sslad-track3b: Transformers are better continual learners. arXiv preprint arXiv:2201.04924, 2022b. — [arXiv:2201.04924](https://arxiv.org/abs/2201.04924) find
477. Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connectivity reveals generalization strategies. arXiv preprint arXiv:2205.12411, 2022. find
478. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023b.
479. Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. The history and risks of reinforcement learning and human feedback. arXiv e-prints, pages arXiv–2310, 2023. find
480. Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain Generalization: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4396–4415, 2023d. doi: 10.1109/tpami.2022.3195549. — [doi:10.1109/tpami.2022.3195549](https://doi.org/10.1109/tpami.2022.3195549) find
481. Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
482. Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731, 2019. find
483. David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pages 5815–5826. Pmlr, 2021. find
484. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-Grained Human Feedback Gives Better Rewards for Language Model Training. arXiv preprint arXiv:2306.01693, 2023b.
485. Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale. arXiv preprint arXiv:2303.16755, 2023b. find
486. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 2021. find
487. Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. The Journal of Machine Learning Research, 23(1):… find
488. Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding. Communications of the ACM, 67(1):110–120, 2023b. find
489. Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding: A survey. arXiv preprint arXiv:2208.11857, 2022. find
490. Zhaowei Zhang, Fengshuo Bai, Jun Gao, and Yaodong Yang. Measuring Value Understanding in Language Models through Discriminator-Critique Gap. arXiv preprint arXiv:2310.00378, 2023d. — [arXiv:2310.00378](https://arxiv.org/abs/2310.00378) find
491. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022.
492. Andreas Stuhlmüller and Jungwon Byun. Supervise Process, not Outcomes, 2022. URL https://ought.org/updates/2022-04-06-process. ↗
493. James Chua, Edward Rees, Hunar Batra, Samuel R Bowman, Julian Michael, Ethan Perez, and Miles Turpin. Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought. arXiv preprint arXiv:2403.05518, 2024. find
494. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979–1993, 2018. find
495. Evan Hubinger. Relaxed adversarial training for inner alignment. AI Alignment Forum, 2019. https://www.alignmentforum.org/posts/9Dy5YRaoCxH9zuJqa/relaxed-adversarial-training-for-i nner-alignment. ↗
496. Nupur Kumari, Mayank Singh, Abhishek Sinha, Harshitha Machiraju, Balaji Krishnamurthy, and Vineeth N Balasubramanian. Harnessing the vulnerability of latent layers in adversarially trained models. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 2779–2785,… find
497. Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Hadfield-Menell. Defending Against Unforeseen Failure Modes with Latent Adversarial Training. arXiv preprint arXiv:2403.05030, 2024b. — [arXiv:2403.05030](https://arxiv.org/abs/2403.05030) find
498. Yilun Kuang and Yash Bharti. Scale-invariant-Fine-Tuning (SiFT) for Improved Generalization in Classification, 2021. find
499. Florian Tramer and Dan Boneh. Adversarial training and robustness for multiple perturbations. Advances in neural information processing systems, 32, 2019. find
500. Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017. find
501. W Jeffrey Johnston and Stefano Fusi. Abstract representations emerge naturally in neural networks trained to perform multiple tasks. Nature Communications, 14(1):1040, 2023. find
502. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. arXiv preprint arXiv:2110.11309, 2021. find
503. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022. find
504. Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept erasure. In International Conference on Machine Learning, pages 18400–18421. Pmlr, 2022a. find
505. Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Adversarial concept erasure in kernel space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6034–6055, 2022b. find
506. Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form, oct 2023. URL http://arxiv.org/ abs/2306.03819. arXiv:2306.03819 [cs]. — [arXiv:2306.03819](https://arxiv.org/abs/2306.03819) ↗
507. Guanghao Li, Li Shen, Yan Sun, Yue Hu, Han Hu, and Dacheng Tao. Subspace based Federated Unlearning, 2023c. find
508. Sangamesh Kodge, Gobinda Saha, and Kaushik Roy. Deep Unlearning: Fast and Efficient Training-free Approach to Controlled Forgetting, 2023. find
509. Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation Addition: Steering Language Models Without Optimization, nov 2023a. URL http://arxiv.org/abs/2308.10248. arXiv:2308.10248 [cs]. ↗
510. Alex Turner, Monte MacDiarmid, David Udell, Lisa Thiergart, and Ulisse Mini. Steering GPT-2 XL by Adding an Activation Vector. AI Alignment Forum, 2023b. https://www.alignmentforum.org/posts/5spBue2z2tw4JuDCx/steering-gpt-2-xl-by-adding-an-activation-vector. Accessed on: October 20, 2023. ↗
511. Maximilian Li, Xander Davies, and Max Nadeau. Circuit breaking: Removing model behaviors with targeted ablation. arXiv preprint arXiv:2309.05973, 2023d. — [arXiv:2309.05973](https://arxiv.org/abs/2309.05973) find
512. Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models. arXiv preprint arXiv:2311.12092, 2023. — [arXiv:2311.12092](https://arxiv.org/abs/2311.12092) find
513. Vaidehi Patil, Peter Hase, and Mohit Bansal. Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks. ArXiv, abs/2309.17410, 2023. URL https://api.semanticscholar.org/CorpusID:263311025. ↗
514. Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. Entity cloze by date: What LMs know about unseen entities. arXiv preprint arXiv:2205.02832, 2022.
515. Yasumasa Onoe, Michael JQ Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can lms learn new entities from descriptions? Challenges in propagating injected knowledge. arXiv preprint arXiv:2305.01651, 2023. find
516. Yiming Zhang and Daphne Ippolito. Prompts should not be seen as secrets: Systematically measuring prompt extraction attack success. arXiv preprint arXiv:2307.06865, 2023. — [arXiv:2307.06865](https://arxiv.org/abs/2307.06865) find
517. Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable Extraction of Training Data from (Production) Language Models. arXiv preprint arXiv:2311.17035, 2023.
518. Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. find
519. Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint arXiv:2209.02299, 2022. — [arXiv:2209.02299](https://arxiv.org/abs/2209.02299) find
520. Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:18075–18086, 2021. find
521. Shashwat Goel, Ameya Prabhu, Philip Torr, Ponnurangam Kumaraguru, and Amartya Sanyal. Corrective Machine Unlearning, 2024. find
522. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge Unlearning for Mitigating Privacy Risks in Language Models, 2022.
523. Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large Language Model Unlearning, 2023.
524. Ronen Eldan and Mark Russinovich. Who’s Harry Potter? Approximate Unlearning in LLMs, 2023. — Who's Harry Potter? Approximate Unlearning in LLMs
525. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting Pretraining Data from Large Language Models. ArXiv, abs/2310.16789, 2023. URL https://api.semanticscholar.org/CorpusID:264451585. ↗
526. Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight Methods to Evaluate Robust Unlearning in LLMs, 2024. find
527. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning. arXiv preprint arXiv:2403.03218, 2024.
528. Leyla Bilge and Tudor Dumitraş. Before we knew it: an empirical study of zero-day attacks in the real world. In Proceedings of the 2012 ACM conference on Computer and communications security, pages 833–844, 2012. find
529. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with LLMs via cipher. arXiv preprint arXiv:2308.06463, 2023b.
530. Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus. Lost in pruning: The effects of pruning neural networks beyond test accuracy. Proceedings of Machine Learning and Systems, 3:93–138, 2021. find
531. Mengnan Du, Subhabrata Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, and Ahmed Hassan Awadallah. Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding. arXiv preprint arXiv:2110.08419, 2021. find
532. Svetlana Pavlitska, Hannes Grolig, and J. Marius Zöllner. Relationship between Model Compression and Adversarial Robustness: A Review of Current Evidence, 2023. find
533. Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. arXiv preprint arXiv:2101.05930, 2021. — [arXiv:2101.05930](https://arxiv.org/abs/2101.05930) find
534. Lijun Sheng, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. AdaptGuard: Defending Against Universal Attacks for Model Adaptation. arXiv preprint arXiv:2303.10594, 2023. — [arXiv:2303.10594](https://arxiv.org/abs/2303.10594) find
535. Lu Pang, Tao Sun, Haibin Ling, and Chao Chen. Backdoor cleansing with unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12218–12227, 2023. find
536. Inioluwa Deborah Raji, Emily M Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. AI and the everything in the whole wide world benchmark. arXiv preprint arXiv:2111.15366, 2021. find
537. Samuel R Bowman and George E Dahl. What will it take to fix benchmarking in natural language understanding? arXiv preprint arXiv:2104.02145, 2021. find
538. Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?… ↗
539. Ben Hutchinson, Negar Rostamzadeh, Christina Greer, Katherine Heller, and Vinodkumar Prabhakaran. Evaluation gaps in machine learning practice. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1859–1876, 2022. find
540. Sayash Kapoor and Arvind Narayanan. Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 2023. find
541. Timothy R McIntosh, Teo Susnjak, Tong Liu, Paul Watters, and Malka N Halgamuge. Inadequacies of large language model benchmarks in the era of generative artificial intelligence. arXiv preprint arXiv:2402.09880, 2024. — [arXiv:2402.09880](https://arxiv.org/abs/2402.09880) find
542. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324, 2023. find
543. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of What Art? A Call for Multi-Prompt LLM Evaluation. arXiv preprint arXiv:2401.00595, 2023. find
544. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921, 2023e.
545. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
546. Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts. arXiv preprint arXiv:2104.06599, 2021.
547. Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
548. Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048– 2056. Pmlr, 2020. find
549. Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274–38290, 2022.
550. Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. Data Contamination Through the Lens of Time. arXiv preprint arXiv:2310.10628, 2023b. find
551. Inbal Magar and Roy Schwartz. Data Contamination: From Memorization to Exploitation. ArXiv, abs/2203.08242, 2022. URL https://api.semanticscholar.org/CorpusID:247475929. ↗
552. Shahriar Golchin and Mihai Surdeanu. Time Travel in LLMs: Tracing Data Contamination in Large Language Models. ArXiv, abs/2308.08493, 2023. URL https://api.semanticscholar.org/Corp usID:260925501. ↗
553. Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B Hashimoto. Proving test set contamination in black box language models. arXiv preprint arXiv:2310.17623, 2023.
554. Yucheng Li. An open source data contamination report for llama series models. arXiv preprint arXiv:2310.17589, 2023. find
555. Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J Nay, Kshitij Gupta, and Aran Komatsuzaki. Arb: Advanced reasoning benchmark for large language models. arXiv preprint arXiv:2307.13692, 2023.
556. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv preprint arXiv:2311.12022, 2023.
557. Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs. arXiv preprint arXiv:2311.04892, 2023.
558. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking Foundation Models with Language-Model-as-an-Examiner. arXiv preprint arXiv:2306.04181, 2023b.
559. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena, 2023.
560. Terry Yue Zhuo. Large Language Models Are State-of-the-Art Evaluators of Code Generation. arXiv preprint arXiv:2304.14317, 2023. find
561. Minghao Wu and Alham Fikri Aji. Style over substance: Evaluation biases for large language models. arXiv preprint arXiv:2307.03025, 2023.
562. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022b.
563. Sandipan Kundu, Yuntao Bai, Saurav Kadavath, Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie, Avital Balwit, Azalia Mirhoseini, Brayden McLean, et al. Specific versus General Principles for Constitutional AI. arXiv preprint arXiv:2310.13798, 2023. find
564. Tom Hosking, Phil Blunsom, and Max Bartolo. Human Feedback is not Gold Standard, 2023. find
565. Rahul Pandey, Hemant Purohit, Carlos Castillo, and Valerie L Shalin. Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learning. International Journal of Human-Computer Studies, 160:102772, 2022. find
566. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning, 2023.
567. Kawin Ethayarajh and Dan Jurafsky. The authenticity gap in human evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6056–6070, 2022. find
568. Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A Smith. All that’s’ human’is not gold: Evaluating human evaluation of generated text. arXiv preprint arXiv:2107.00061, 2021. find
569. Ziang Xiao, Susu Zhang, Vivian Lai, and Q Vera Liao. Evaluating NLG Evaluation Metrics: A Measurement Theory Perspective. arXiv preprint arXiv:2305.14889, 2023. — [arXiv:2305.14889](https://arxiv.org/abs/2305.14889) find
570. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators, jun 2022. URL http://arxiv.org/abs/2206 .05802. arXiv:2206.05802 [cs]. ↗
571. Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. find
572. Cassidy Laidlaw and Anca Dragan. The boltzmann policy distribution: Accounting for systematic suboptimality in human models. arXiv preprint arXiv:2204.10759, 2022. — [arXiv:2204.10759](https://arxiv.org/abs/2204.10759) find
573. David Lindner and Mennatallah El-Assady. Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning. arXiv preprint arXiv:2206.13316, 2022. — [arXiv:2206.13316](https://arxiv.org/abs/2206.13316) find
574. Julien Fageot, Sadegh Farhadkhani, Lê Nguyên Hoang, and Oscar Villemaud. Generalized Bradley-Terry Models for Score Estimation from Paired Comparisons. arXiv preprint arXiv:2308.08644, 2023. — [arXiv:2308.08644](https://arxiv.org/abs/2308.08644) find
575. Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks. ArXiv, abs/2306.07899, 2023. URL https://api.semanticscholar.org/CorpusID:259145373. ↗
576. Jeremias Prassl and Martin Risak. The legal protection of crowdworkers: four avenues for workers’ rights in the virtual realm. Policy implications of virtual work, pages 273–295, 2017. find
577. Nihar Shah, Dengyong Zhou, and Yuval Peres. Approval voting and incentives in crowdsourcing. In International conference on machine learning, pages 10–19. Pmlr, 2015. find
578. Natalie Schluter. The glass ceiling in NLP. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2793–2798, 2018. find
579. Ali Akbar Septiandri, Marios Constantinides, Mohammad Tahaei, and Daniele Quercia. WEIRD FAc-cTs: How Western, Educated, Industrialized, Rich, and Democratic is FAccT? In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 160–171, 2023. find
580. Su Lin Blodgett, Q Vera Liao, Alexandra Olteanu, Rada Mihalcea, Michael Muller, Morgan Klaus Scheuerman, Chenhao Tan, and Qian Yang. Responsible language technologies: Foreseeing and mitigating harms. In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pages 1–3, 2022. find
581. Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, ..., and Jared Ka… — Measuring Progress on Scalable Oversight for Large Language Models
582. Lukas Fluri, Daniel Paleka, and Florian Tramèr. Evaluating Superhuman Models with Consistency Checks. arXiv preprint arXiv:2306.09983, 2023. — [arXiv:2306.09983](https://arxiv.org/abs/2306.09983) find
583. Geoffrey Irving, Paul Christiano, and Dario Amodei. arXiv:1805.00899, 2018.
584. AI safety via debate. arXiv preprint Sam Bowman and Tamera Lanham. The Anthropic–NYU Debate Agenda, 2023. https://docs.goo gle.com/document/d/173SpCyspboHBp3bHqWvUiduzatbuuv7QAvWVamwcGbk/edit. Accessed on: March 10, 2024. ↗
585. Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018. find
586. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. arXiv preprint arXiv:2312.09390, 2023.
587. Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe. The Unreasonable Effectiveness of Easy Training Data for Hard Tasks, 2024. URL https://arxiv.org/pdf/2401.06751.pdf. ↗
588. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018.
589. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862, 2021.
590. David Krueger. AI alignment and generalization in deep learning, 2023. find
591. Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. Debating with More Persuasive LLMs Leads to More Truthful Answers. arXiv preprint arXiv:2402.06782, 2024.
592. Justin Reppert, Ben Rachbach, Charlie George, Luke Stebbing, Jungwon Byun, Maggie Appleton, and Andreas Stuhlmüller. Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes, jan 2023. URL http://arxiv.org/abs/2301.01751. arXiv:2301.01751 [cs]. — [arXiv:2301.01751](https://arxiv.org/abs/2301.01751) ↗
593. Beth Barnes, Paul Christiano, William Saunders, Joe Collman, Mark Xu, Chris Painter, Mihnea Maftei, and Ronny Fernandez. Debate update: Obfuscated arguments problem. AI Alignment Forum, 2020. https://www.alignmentforum.org/posts/PJLABqQ962hZEqhdB/debateupdate-obfuscated-arg uments-problem. Accessed… ↗
594. Jonah Brown-Cohen, Geoffrey Irving, and Georgios Piliouras. Scalable AI safety via doubly-efficient debate. arXiv preprint arXiv:2311.14125, 2023. — [arXiv:2311.14125](https://arxiv.org/abs/2311.14125) find
595. Ruiqi Zhong, Charlie Snell, Dan Klein, and Jason Eisner. Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQL. In Proceedings of EMNLP, December 2023b. find
596. Tamera Lanham. Externalized reasoning oversight: a research direction for language model alignment. AI Alignment Forum, 2022. https://www.alignmentforum.org/posts/FRRb6Gqem8k69ocbi/ext ernalized-reasoning-oversight-a-research-direction-for. Accessed on: 3 January, 2024. ↗
597. Lisa Schut, Nenad Tomasev, Tom McGrath, Demis Hassabis, Ulrich Paquet, and Been Kim. Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero, oct 2023. URL http://arxiv.org/abs/2310.16410. arXiv:2310.16410 [cs, stat]. — [arXiv:2310.16410](https://arxiv.org/abs/2310.16410) ↗
598. Marco Tulio Ribeiro. “Why Should I Trust You?” Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016. URL — "Why Should I Trust You?": Explaining the Predictions of Any Classifier https://api.semanticscholar.org/CorpusID:260555761. ↗
599. Scott M. Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions. In Neural Information Processing Systems, 2017. URL https://api.semanticscholar.org/CorpusID: 21889700. ↗
600. Blair Bilodeau, Natasha Jaques, Pang Wei Koh, and Been Kim. Impossibility Theorems for Feature Attribution. ArXiv, abs/2212.11870, 2022. URL https://api.semanticscholar.org/CorpusID: 254974246. — [arXiv:2212.11870](https://arxiv.org/abs/2212.11870) ↗
601. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks. In International Conference on Machine Learning, 2017. URL https://api.semanticscholar.org/ CorpusID:16747630. ↗
602. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? Analyzing individual neurons in deep NLP models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6309–6317, 2019. find
603. David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 117(48):30071–30078, 2020. find
604. Ludwig Schubert, Chelsea Voss, Nick Cammarata, Gabriel Goh, and Chris Olah. High-low frequency detectors. Distill, 2021. find
605. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy Models of Superposition, sep 2022b. URL ht… find
606. Omer Antverg and Yonatan Belinkov. On the Pitfalls of Analyzing Individual Neurons in Language Models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=8uz0EWPQIMu. ↗
607. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203.14680, 2022.
608. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D Goodman. Finding alignments between interpretable causal variables and distributed neural representations. arXiv preprint arXiv:2303.02536, 2023.
609. Sebastian Farquhar, Vikrant Varma, Zachary Kenton, Johannes Gasteiger, Vladimir Mikulik, and Rohin Shah. Challenges with unsupervised LLM knowledge discovery. arXiv preprint arXiv:2312.10029, 2023. — [arXiv:2312.10029](https://arxiv.org/abs/2312.10029) find
610. BA Levinstein and Daniel A Herrmann. Still no lie detector for language models: Probing empirical and conceptual roadblocks. arXiv preprint arXiv:2307.00175, 2023. — [arXiv:2307.00175](https://arxiv.org/abs/2307.00175) find
611. Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2307.15771, 2023. find
612. Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016. find
613. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do Llamas Work in English? On the Latent Language of Multilingual Transformers. arXiv preprint arXiv:2402.10588, 2024.
614. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574–9586, 2021. find
615. Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314, 2021. — [arXiv:2106.13314](https://arxiv.org/abs/2106.13314) find
616. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In International Conference on Machine Learning, 2017. URL https://api.semanticscho… ↗
617. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017.
618. Andreea Bobu, Andi Peng, Pulkit Agrawal, Julie Shah, and Anca D Dragan. Aligning Robot and Human Representations. arXiv preprint arXiv:2302.01928, 2023. — [arXiv:2302.01928](https://arxiv.org/abs/2302.01928) find
619. Lukas Muttenthaler, Lorenz Linhardt, Jonas Dippel, Robert A Vandermeulen, Katherine Hermann, Andrew Lampinen, and Simon Kornblith. Improving neural network representations using human similarity judgments. Advances in Neural Information Processing Systems, 36, 2023. find
620. Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael A. Specter, and Lalana Kagal. Explaining Explanations: An Overview of Interpretability of Machine Learning. 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pages 80–89, 2018. URL https://api.semant… ↗
621. Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim. Sanity Checks for Saliency Maps. In Neural Information Processing Systems, 2018. URL https://api.semanticscholar.org/CorpusID:52938797. ↗
622. Peter Hase and Mohit Bansal. Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? In Annual Meeting of the Association for Computational Linguistics, 2020. URL https://api.semanticscholar.org/CorpusID:218502350. ↗
623. Julius Adebayo, Michael Muelly, Ilaria Liccardi, and Been Kim. Debugging Tests for Model Explanations. ArXiv, abs/2011.05429, 2020. URL https://api.semanticscholar.org/CorpusID: 226299635. — [arXiv:2011.05429](https://arxiv.org/abs/2011.05429) ↗
624. Stephen Casper, Yuxiao Li, Jiawei Li, Tong Bu, Kevin Zhang, Kaivalya Hariharan, and Dylan Hadfield-Menell. Red Teaming Deep Neural Networks with Feature Synthesis Tools, sep 2023b. URL http://arxiv.org/abs/2302.10894. arXiv:2302.10894 [cs]. — [arXiv:2302.10894](https://arxiv.org/abs/2302.10894) ↗
625. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023. ↗
626. Jing Huang, Atticus Geiger, Karel D’Oosterlinck, Zhengxuan Wu, and Christopher Potts. Rigorously Assessing Natural Language Explanations of Neurons. arXiv preprint arXiv:2309.10312, 2023b.
627. Finale Doshi-Velez and Been Kim. Towards A Rigorous Science of Interpretable Machine Learning, mar 2017. URL http://arxiv.org/abs/1702.08608. arXiv:1702.08608 [cs, stat]. ↗
628. Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, 267:1–38, 2019.
629. Maya Krishnan. Against Interpretability: a Critical Examination of the Interpretability Problem in Machine Learning. Philosophy & Technology, 33(3):487–502, sep 2020. ISSN 2210-5441. doi: 10.100 7/s13347-019-00372-9. URL https://doi.org/10.1007/s13347-019-00372-9. — [doi:10.1007/s13347-019-00372-9](https://doi.org/10.1007/s13347-019-00372-9) ↗
630. Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks, aug 2023. URL http://arxiv.org/abs/2207.13243. arXiv:2207.13243 [cs]. — [arXiv:2207.13243](https://arxiv.org/abs/2207.13243) ↗
631. Alon Jacovi and Yoav Goldberg. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205, Online, jul 2020. Association for Computational Linguistics. doi:…
632. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-Precision Model-Agnostic Explanations. In AAAI Conference on Artificial Intelligence, 2018. URL https://api.semanticsc holar.org/CorpusID:3366554. ↗
633. Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, jenny, Ansh Radhakrishnan, Buck Shlegaris, and Nate Thomas. Causal Scrubbing: a method for rigorously testing interpretability hypotheses, 2023c. https://static1.squarespace.com/static/6114773bd7f9917 b7ae4ef8d/t/6364a036… ↗
634. Sarah Schwettmann, Tamar Rott Shaham, Joanna Materzynska, Neil Chowdhury, Shuang Li, Jacob Andreas, David Bau, and Antonio Torralba. A Function Interpretation Benchmark for Evaluating Interpretability Methods. ArXiv, abs/2309.03886, 2023. URL https://api.semanticscholar.org/CorpusID:261582916. — [arXiv:2309.03886](https://arxiv.org/abs/2309.03886) ↗
635. Center for AI Safety. The Trojan Detection Challenge 2023 (LLM Edition) - The Trojan Detection Challenge, 2023. URL https://trojandetection.ai/. ↗
636. Javier Rando, Francesco Croce, Krystof Mitka, Stepan Shabalin, Maksym Andriushchenko, Nicolas Flammarion, and Florian Tramèr. Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMs, 2024. find
637. Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pages 177–190. Springer, 2005. find
638. Ellie Pavlick and Tom Kwiatkowski. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics, 7:677–694, 11 2019. ISSN 2307-387x. doi: 10.116 2/tacl_a_00293. URL https://doi.org/10.1162/tacl%5Fa%5F00293. ↗
639. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, ..… — Measuring Faithfulness in Chain-of-Thought Reasoning
640. Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, and Kathleen McKeown. Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations, jul 2023b. URL http://arxiv.org/abs/2307.08678. arXiv:2307.08678 [cs]. ↗
641. Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Eckersley. Explainable machine learning in deployment. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 648–657, 2020. find
642. Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and Editing Knowledge Representations in Language Models, may 2023a. URL http://arxiv.org/abs/2304.00740. arXiv:2304.00740 [cs]. ↗
643. Chenmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language models via meta learning. arXiv preprint arXiv:2311.04661, 2023. — [arXiv:2311.04661](https://arxiv.org/abs/2311.04661) find
644. Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, et al. Knowledge editing for large language models: A survey. arXiv preprint arXiv:2310.16218, 2023h.
645. Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv:1703.03717, 2017. find
646. Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. In Proceedings of the European conference on computer vision (ECCV), pages 771–787, 2018. find
647. Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu. Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. In International conference on machine learning, pages 8116–8126. PMLR, 2020. find
648. Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3619–3629, 2021. find
649. Casey Chu, Andrey Zhmoginov, and Mark Sandler. Cyclegan, a master of steganography. arXiv preprint arXiv:1712.02950, 2017. — [arXiv:1712.02950](https://arxiv.org/abs/1712.02950) find
650. David Manheim and Scott Garrabrant. Categorizing variants of Goodhart’s Law. arXiv preprint arXiv:1803.04585, 2018. — [arXiv:1803.04585](https://arxiv.org/abs/1803.04585) find
651. Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research c… find
652. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv, nov 2018. http://arxiv.org/abs/1610.01644. ↗
653. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A Primer in BERTology: What we know about how BERT works, nov 2020. URL http://arxiv.org/abs/2002.12327. arXiv:2002.12327 [cs]. ↗
654. Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept Algebra for (Score-Based) Text-Controlled Generative Models, oct 2023i. URL http://arxiv.org/abs/2302.03693. arXiv:2302.03693 [cs, stat]. — [arXiv:2302.03693](https://arxiv.org/abs/2302.03693) ↗
655. Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of Relation Decoding in Transformer Language Models, aug 2023b. URL http://arxiv.org/abs/2308.09124. arXiv:2308.09124 [cs]. ↗
656. Lee Sharkey, Dan Braun, and beren. [Interim research report] Taking features out of superposition with sparse autoencoders. AI Alignment Forum, 2023. https://www.alignmentforum.org/posts/z6Q QJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition. ↗
657. Mara Graziani, Laura Mahony, An phi Nguyen, Henning Muller, and Vincent Andrearczyk. Uncovering Unique Concept Vectors through Latent Space Decomposition. ArXiv, abs/2307.06913, 2023. URL https://api.semanticscholar.org/CorpusID:259847592. — [arXiv:2307.06913](https://arxiv.org/abs/2307.06913) ↗
658. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, ..., and Christopher O… find
659. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse Autoencoders Find Highly Interpretable Features in Language Models, oct 2023. URL http://arxiv.org/abs/23 09.08600. arXiv:2309.08600 [cs]. ↗
660. Vikram V. Ramaswamy, Sunnie S. Y. Kim, Ruth C. Fong, and Olga Russakovsky. Overlooked Factors in Concept-Based Explanations: Dataset Choice, Concept Learnability, and Human Capability. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10932–10941, 2022. URL https://ap… find
661. Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Vi’egas, and Martin Wattenberg. An Interpretability Illusion for BERT. ArXiv, abs/2104.07143, 2021. URL https://api.semanticscholar.org/CorpusID:233241181. — [arXiv:2104.07143](https://arxiv.org/abs/2104.07143) ↗
662. Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy Suppression: Comprehensively Understanding an Attention Head. arXiv preprint arXiv:2310.04625, 2023. find
663. Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2021. find
664. Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969, 2023.
665. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards Automated Circuit Discovery for Mechanistic Interpretability. CoRR, abs/2304.14997, 2023. URL https://doi.org/10.48550/arXiv.2304.14997. ↗
666. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting, may 2023. URL http://arxiv.org/abs/2305.04388. arXiv:2305.04388 [cs]. ↗
667. Jacob Eisenstein, Daniel Andor, Bernd Bohnet, Michael Collins, and David Mimno. Honest students from untrusted teachers: Learning an interpretable question-answering pipeline from a pretrained language model. arXiv preprint arXiv:2210.02498, 2022. — [arXiv:2210.02498](https://arxiv.org/abs/2210.02498) find
668. Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Sam McCandlish, Sheer El Showk, Tamera Lanham, Tim Maxwell, Venkatesa Chandras… find
669. Paulo Tabuada. Verification and control of hybrid systems: a symbolic approach. Springer Science & Business Media, 2009. find
670. C.A.R. Hoare, Jayadev Misra, Gary T. Leavens, and Natarajan Shankar. The verified software initiative: A manifesto. ACM Comput. Surv., 2009. find
671. Max Tegmark and Steve Omohundro. Provably safe systems: the only path to controllable AGI. arXiv preprint arXiv:2309.01933, 2023. — [arXiv:2309.01933](https://arxiv.org/abs/2309.01933) find
672. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
673. Yuhuai Wu, Albert Qiaochu Jiang, Wenda Li, Markus Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. Autoformalization with large language models. Advances in Neural Information Processing Systems, 35:32353–32368, 2022.
674. John M Zelle and Raymond J Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pages 1050–1055, 1996.
675. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, 2013.
676. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods…
677. Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch. Towards Faithful Model Explanation in NLP: A Survey. ArXiv, 2022. doi: 10.48550/arxiv.2209.11326. URL https://arxiv.org/abs/2209 .11326. Publisher: arXiv Version Number: 2. ↗
678. Tanmay Gupta and Aniruddha Kembhavi. Visual Programming: Compositional visual reasoning without training. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14953–14962, 2022. URL https://api.semanticscholar.org/CorpusID:253734854. ↗
679. D’idac Sur’is, Sachit Menon, and Carl Vondrick. ViperGPT: Visual Inference via Python Execution for Reasoning. ArXiv, abs/2303.08128, 2023. URL https://api.semanticscholar.org/CorpusID: 257505358. ↗
680. Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu-Ghazaleh. Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv preprint arXiv:2310.10844, 2023.
681. Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein. Coercing LLMs to do and reveal (almost) anything, 2024. find
682. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. Prompt Injection attack against LLM-integrated Applications. arXiv preprint arXiv:2306.05499, 2023a.
683. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. More than you’ve asked for: A Comprehensive Analysis of Novel Prompt Injection Threats to Application-Integrated Large Language Models. arXiv preprint arXiv:2302.12173, 2023a.
684. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study, 2023b.
685. Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs. arXiv preprint arXiv:2402.11753, 2024b.
686. Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. Image Hijacking: Adversarial Images can Control Generative Models at Runtime. arXiv preprint arXiv:2309.00236, 2023. — [arXiv:2309.00236](https://arxiv.org/abs/2309.00236) find
687. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. “Do Anything Now”: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. arXiv preprint arXiv:2308.03825, 2023.
688. Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choudhury. Tricking LLMs into Disobedience: Understanding, Analyzing, and Preventing Jailbreaks, 2023. find
689. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source LLMs via exploiting generation. arXiv preprint arXiv:2310.06987, 2023c.
690. Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, et al. Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game. arXiv preprint arXiv:2311.01011, 2023. find
691. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
692. Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE symposium on security and privacy (sp), pages 39–57. find
693. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
694. Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447, 2023a.
695. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv:1712.06751, 2017.
696. Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in neural information processing systems, 33:1633–1645, 2020b. find
697. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline Defenses for Adversarial Attacks Against Aligned Language Models. arXiv preprint arXiv:2309.00614, 2023b.
698. Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically Auditing Large Language Models via Discrete Optimization. arXiv preprint arXiv:2303.04381, 2023.
699. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual Jailbreak Challenges in Large Language Models. arXiv preprint arXiv:2310.06474, 2023a.
700. Cheng Li, Jindong Wang, Kaijie Zhu, Yixuan Zhang, Wenxin Hou, Jianxun Lian, and Xing Xie. Emotionprompt: Leveraging psychology for large language models enhancement via emotional stimulus. arXiv preprint arXiv:2307.11760, 2023e.
701. Jacob Andreas. Language models as agent models. arXiv preprint arXiv:2212.01681, 2022.
702. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022b.
703. Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. Explore, Establish, Exploit: Red Teaming Language Models from Scratch. arXiv preprint arXiv:2306.09442, 2023c.
704. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980, 2020.
705. Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-based instruction search for prompting large language models. arXiv preprint arXiv:2203.07281, 2022.
706. Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. Rlprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205.12548, 2022.
707. Raz Lapid, Ron Langberg, and Moshe Sipper. Open Sesame! Universal Black Box Jailbreaking of Large Language Models. arXiv preprint arXiv:2309.01446, 2023. — [arXiv:2309.01446](https://arxiv.org/abs/2309.01446) find
708. Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277, 2016. — [arXiv:1605.07277](https://arxiv.org/abs/1605.07277) find
709. OpenAI. GPT-4V(ision) System Card, 2023c. https://cdn.openai.com/papers/GPTV%5FSystem%5F Card.pdf. ↗
710. Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. Visual Adversarial Examples Jailbreak Large Language Models. arXiv preprint arXiv:2306.13213, 2023b.
711. Simon Willison. Multi-modal prompt injection, 2023a. https://simonwillison.net/2023/Oct/14/ multi-modal-prompt-injection/. Accessed on: October 20, 2023. ↗
712. Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts. arXiv preprint arXiv:2311.05608, 2023.
713. Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, and Vitaly Shmatikov. (Ab)using Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs. arXiv preprint arXiv:2307.10490, 2023. — [arXiv:2307.10490](https://arxiv.org/abs/2307.10490) find
714. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
715. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023c.
716. Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar. Introducing our Multimodal Models, 2023. URL https://www.adept.ai/blog/fu yu-8b. ↗
717. Alec Helbling, Mansi Phute, Matthew Hull, and Duen Horng Chau. LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked. arXiv preprint arXiv:2308.07308, 2023.
718. Simon Willison. You can’t solve AI security problems with more AI, 2022. https://simonwillison.net/2022/Sep/17/prompt-injection-more-ai/. Accessed on: October 20, 2023. ↗
719. Chris Cundy and Stefano Ermon. SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking. arXiv preprint arXiv:2306.05426, 2023. find
720. Egor Zverev, Sahar Abdelnabi, Mario Fritz, and Christoph H Lampert. Can LLMs Separate Instructions From Data? And What Do We Even Mean By That? arXiv preprint arXiv:2403.06833, 2024. — [arXiv:2403.06833](https://arxiv.org/abs/2403.06833) find
721. Jiahao Yu, Yuhang Wu, Dong Shu, Mingyu Jin, and Xinyu Xing. Assessing Prompt Injection Risks in 200+ Custom GPTs. arXiv preprint arXiv:2311.11538, 2023b. — [arXiv:2311.11538](https://arxiv.org/abs/2311.11538) find
722. Yao Qiang, Xiangyu Zhou, and Dongxiao Zhu. Hijacking Large Language Models via Adversarial In-Context Learning. arXiv preprint arXiv:2311.09948, 2023. — [arXiv:2311.09948](https://arxiv.org/abs/2311.09948) find
723. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection, 2023b. — Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
724. Simon Willison. The Dual LLM pattern for building AI assistants that can resist prompt injection, 2023b. https://simonwillison.net/2023/Apr/25/dual-llm-pattern. Accessed on: October 20, 2023. ↗
725. Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546, 2022. — [arXiv:2210.17546](https://arxiv.org/abs/2210.17546) find
726. Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv preprint arXiv:1206.6389, 2012. — [arXiv:1206.6389](https://arxiv.org/abs/1206.6389) find
727. Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical. arXiv preprint arXiv:2302.10149, 2023b. — [arXiv:2302.10149](https://arxiv.org/abs/2302.10149) find
728. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on M… find
729. Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. LLM360: Towards Fully Transparent Open-Source LLMs. arXiv preprint arXiv:2312.06550, 2023d. find
730. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. In Proceedings of the 28th USENIX Conference on Security Symposium, Sec’19, page 267–284, Usa, 2019. USENIX Association. ISBN 9781939133069.
731. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. find
732. Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning Language Models During Instruction Tuning. arXiv preprint arXiv:2305.00944, 2023b. find
733. Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang. Composite Backdoor Attacks Against Large Language Models. arXiv preprint arXiv:2310.07676, 2023d. find
734. Javier Rando and Florian Tramèr. Universal Jailbreak Backdoors from Poisoned Human Feedback. arXiv preprint arXiv:2311.14455, 2023. — [arXiv:2311.14455](https://arxiv.org/abs/2311.14455) find
735. Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorobeychik, and Chaowei Xiao. On the Exploitability of Reinforcement Learning with Human Feedback for Large Language Models. arXiv preprint arXiv:2311.09641, 2023j. — [arXiv:2311.09641](https://arxiv.org/abs/2311.09641) find
736. Ziqing Yang, Xinlei He, Zheng Li, Michael Backes, Mathias Humbert, Pascal Berrang, and Yang Zhang. Data poisoning attacks against multimodal encoders. In International Conference on Machine Learning, pages 39299–39313. Pmlr, 2023b. find
737. Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. arXiv preprint arXiv:2106.09667, 2021. — [arXiv:2106.09667](https://arxiv.org/abs/2106.09667) find
738. Changjiang Li, Ren Pang, Zhaohan Xi, Tianyu Du, Shouling Ji, Yuan Yao, and Ting Wang. An Embarrassingly Simple Backdoor Attack on Self-supervised Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4367–4378, October 2023f. find
739. Aniruddha Saha, Ajinkya Tejankar, Soroush Abbasi Koohpayegani, and Hamed Pirsiavash. Backdoor Attacks on Self-Supervised Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13337–13346, June 2022. find
740. Zhen Xiang, David J Miller, and George Kesidis. Revealing backdoors, post-training, in DNN classifiers via novel inference on optimized perturbations inducing group misclassification. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3827–38… find
741. Yinpeng Dong, Xiao Yang, Zhijie Deng, Tianyu Pang, Zihao Xiao, Hang Su, and Jun Zhu. Black-box detection of backdoor attacks with limited information and data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16482–16491, 2021. find
742. Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707–723. find
743. Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 1265–1282, 2019. find
744. Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting AI trojans using meta neural analysis. In 2021 IEEE Symposium on Security and Privacy (SP), pages 103–120. find
745. Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 301–310, 2020. find
746. Lauro Langosco, Neel Alex, William Baker, David Quarel, Herbie Bradley, and David Krueger. Detecting Backdoors with Meta-Models. In NeurIPS 2023 Workshop on Backdoors in Deep Learning - The Good, the Bad, and the Ugly, 2024. URL https://openreview.net/forum?id=cmJiEqniEc. ↗
747. Shafi Goldwasser, Michael P Kim, Vinod Vaikuntanathan, and Or Zamir. Planting undetectable backdoors in machine learning models. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS), pages 931–942. find
748. Sanghyun Hong, Nicholas Carlini, and Alexey Kurakin. Handcrafted backdoors in deep neural networks. Advances in Neural Information Processing Systems, 35:8068–8080, 2022. find
749. Laura Sartori and Andreas Theodorou. A sociotechnical perspective for the future of AI: narratives, inequalities, and human control. Ethics and Information Technology, 24(1):4, 2022. find
750. Seth Lazar and Alondra Nelson. AI safety on whose terms?, 2023. find
751. Atoosa Kasirzadeh and James Stewart. Sociotechnical AI safety: a multidisciplinary perspective, 2024. find
752. Atoosa Kasirzadeh and Iason Gabriel. In conversation with Artificial Intelligence: aligning language models with human values. Philosophy & Technology, 36(2):1–24, 2023. find
753. Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. Truthful AI: Developing and Governing AI That Does Not Lie. arXiv:2110.06674, October 2021. doi: 10.48550/arxiv.2110.06674. — [doi:10.48550/arxiv.2110.06674](https://doi.org/10.48550/arxiv.2110.06674) find
754. Jacob Hilton. Truthful LMs as a warm-up for aligned AGI. AI Alignment Forum, 2022. https://www.al ignmentforum.org/posts/jWkqACmDes6SoAiyE/truthful-lms-as-a-warm-up-for-aligned-agi. ↗
755. Vinodkumar Prabhakaran, Margaret Mitchell, Timnit Gebru, and Iason Gabriel. A human rights-based approach to responsible AI. arXiv preprint arXiv:2210.02667, 2022a. find
756. Allan Dafoe, Yoram Bachrach, Gillian Hadfield, Eric Horvitz, Kate Larson, and Thore Graepel. Cooperative AI: machines must learn to find common ground. Nature, 593(7857):33–36, may 2021. doi: 10.1038/d41586-021-01170-0. URL https://www.nature.com/articles/d41586-021-01170-0. — [doi:10.1038/d41586-021-01170-0](https://doi.org/10.1038/d41586-021-01170-0) ↗
757. Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. In Workshops at the twenty-ninth AAAI conference on artificial intelligence, 2015. find
758. Betty Li Hou and Brian Patrick Green. Foundational Moral Values for AI Alignment. arXiv preprint arXiv:2311.17017, 2023. — [arXiv:2311.17017](https://arxiv.org/abs/2311.17017) find
759. Taylor Sorensen, Liwei Jiang, Jena Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, et al. Value Kaleidoscope: Engaging AI with pluralistic human values, rights, and duties. arXiv preprint arXiv:2309.00779, 2023.
760. Esin Durmus, Karina Nyugen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388, 2023.
761. Jan-Philipp Fränken, Sam Kwok, Peixuan Ye, Kanishk Gandhi, Dilip Arumugam, Jared Moore, Alex Tamkin, Tobias Gerstenberg, and Noah D Goodman. Social Contract AI: Aligning AI Assistants with Implicit Group Norms. arXiv preprint arXiv:2310.17769, 2023. — [arXiv:2310.17769](https://arxiv.org/abs/2310.17769) find
762. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. find
763. Atoosa Kasirzadeh. Chatgpt, large language technologies, and the bumpy road of benefiting humanity. arXiv preprint arXiv:2304.11163, 2023. — [arXiv:2304.11163](https://arxiv.org/abs/2304.11163) find
764. Ryan Liu, Theodore R Sumers, Ishita Dasgupta, and Thomas L Griffiths. How do large language models navigate conflicts between honesty and helpfulness? arXiv preprint arXiv:2402.07282, 2024b. — [arXiv:2402.07282](https://arxiv.org/abs/2402.07282) find
765. Vinodkumar Prabhakaran, Rida Qadri, and Ben Hutchinson. Cultural incongruencies in artificial intelligence. arXiv preprint arXiv:2211.13069, 2022b. — [arXiv:2211.13069](https://arxiv.org/abs/2211.13069) find
766. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, et al. Challenges and strategies in cross-cultural NLP. arXiv preprint arXiv:2203.10020, 2022.
767. Sarah Keren, Avigdor Gal, and Erez Karpas. Goal recognition design. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 24, pages 154–162, 2014. find
768. Malek Mechergui and Sarath Sreedharan. Goal Alignment: Re-analyzing Value Alignment Problems Using Human-Aware AI. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10110–10118, 2024. find
769. Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Sean Kirmani, Isabel Leal, Edward Lee, Sergey Levine, Yao Lu, Sharath Maddineni, Kanishka Rao, Dorsa Sadigh, Pannag Sanketi, ..., and Zh… find
770. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92–110, 2022.
771. Marc Serramia, Maite López-Sánchez, Stefano Moretti, and Juan Antonio Rodríguez-Aguilar. On the dominant set selection problem and its application to value alignment. Autonomous Agents and Multi-Agent Systems, 35(2):42, 2021. find
772. Mohammad Atari, Mona J Xue, Peter S Park, Damián Blasi, and Joseph Henrich. Which humans? 2023. find
773. Rebecca L Johnson, Giada Pistilli, Natalia Menédez-González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv preprint arXiv:2203.07785, 2022.
774. Shaily Bhatt, Sunipa Dev, Partha Talukdar, Shachi Dave, and Vinodkumar Prabhakaran. Cultural Re-contextualization of Fairness Research in Language Technologies in India. arXiv preprint arXiv:2211.11206, 2022. — [arXiv:2211.11206](https://arxiv.org/abs/2211.11206) find
775. Michal Shur-Ofry. Multiplicity as an AI Governance Principle. Available at SSRN 4444354, 2023. find
776. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. A Roadmap to Pluralistic Alignment. arXiv preprint arXiv:2402.05070, 2024.
777. Roel Dobbe, Thomas Krendl Gilbert, and Yonatan Mintz. Hard choices in artificial intelligence. Artificial Intelligence, 300:103555, 2021. find
778. Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano. Participation is not a design fix for machine learning. In Proceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1–6, 2022. find
779. Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark Diaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. Power to the people? Opportunities and challenges for participatory AI. Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1–8, 2022. find
780. Meaning Alignment Institute. Democratic Finetuning, 2023. URL https://meaningalignment.sub stack.com/p/the-first-moral-graph. ↗
781. Deliberation at Scale. First Report: Democratic Inputs to AI, 2023. URL https://findcommongrou nd.online/top-level-pages/about-the-consortium. find
782. Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Jaime Diez-Medrano, Milena Lagos, Pippa Norris, Eduard Ponarin, and Bianca Puranen. World Values Survey: Round Seven – Country-Pooled Datafile Version 5.0.0, 2022. find
783. Shalom H Schwartz. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In Advances in experimental social psychology, volume 25, pages 1–65. Elsevier, 1992. find
784. Shalom H Schwartz. Are there universal aspects in the structure and contents of human values? Journal of social issues, 50(4):19–45, 1994. find
785. Laura Weidinger, Kevin R McKee, Richard Everett, Saffron Huang, Tina O Zhu, Martin J Chadwick, Christopher Summerfield, and Iason Gabriel. Using the Veil of Ignorance to align AI systems with principles of justice. Proceedings of the National Academy of Sciences, 120(18):e2213709120, 2023b. find
786. Tanusree Sharma, Jongwon Park, Yujin Kwon, Yiren Liu, Yun Huang, Sunny Liu, Dawn Song, Jeff Hancock, and Yang Wang. Inclusive.ai: Engaging Underserved Populations In Democratic Decision-making On Ai, 2023. find
787. Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing… find
788. Sara Fish, Paul Gölz, David C Parkes, Ariel D Procaccia, Gili Rusak, Itai Shapira, and Manuel Wüthrich. Generative Social Choice. arXiv preprint arXiv:2309.01291, 2023. — [arXiv:2309.01291](https://arxiv.org/abs/2309.01291) find
789. Sara Hooker. The hardware lottery. Communications of the ACM, 64(12):58–65, 2021. find
790. Cesare Carissimo and Marcin Korecki. Limits of optimization. Minds and Machines, pages 1–21, 2023. find
791. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning AI with shared human values. arXiv preprint arXiv:2008.02275, 2020b.
792. Nino Scherrer, Claudia Shi, Amir Feder, and David M Blei. Evaluating the moral beliefs encoded in LLMs. arXiv preprint arXiv:2307.14324, 2023.
793. Arnav Arora, Lucie-Aimée Kaffee, and Isabelle Augenstein. Probing pre-trained language models for cross-cultural differences in values. arXiv preprint arXiv:2203.13722, 2022.
794. Leon Yin, Davey Alba, and Leonardo Nicoletti. OpenAI’s GPT Is a Recruiter’s Dream Tool. Tests Show There’s Racial Bias. Bloomberg, 2024. https://www.bloomberg.com/graphics/2024-opena i-gpt-hiring-racial-discrimination. Accessed on: 9 March, 2024. ↗
795. Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. A word on machine ethics: A response to Jiang et al.(2021). arXiv preprint arXiv:2111.04158, 2021. — [arXiv:2111.04158](https://arxiv.org/abs/2111.04158) find
796. Gabriel Simmons. Moral mimicry: Large language models produce moral rationalizations tailored to political identity. arXiv preprint arXiv:2209.12106, 2022. find
797. Jan Leike. What could a solution to the alignment problem look like? Musings on the Alignment Problem, 2022b. https://aligned.substack.com/p/alignment-solution. Accessed on: 3 January, 2024. ↗
798. A. P. Brogan. Philosophy and the Problem of Value. Proceedings and Addresses of the American Philosophical Association, 6:105–129, 1952. URL https://doi.org/10.2307/1483020. — [doi:10.2307/1483020](https://doi.org/10.2307/1483020) ↗
799. Elselijn Kingma and Natalie Banner. Liberating Practice from Philosophy – A Critical Examination of Values-Based Practice and Its Underpinnings. In Michael Loughlin, editor, Debates in Values-Based Practice: Arguments For and Against, pages 37–49. Cambridge University Press, 2014. find
800. Mark Schroeder. Value Theory. The Stanford Encyclopedia of Philosophy (Fall 2021 Edition), 2021. URL https://plato.stanford.edu/archives/fall2021/entries/value-theory/. ↗
801. Regina Polak and Patrick Rohs. Values–Politics–Religion: The European Values Study: In-depth Analysis–Interdisciplinary Perspectives–Future Prospects. Springer Nature, 2023. find
802. Matthias Kaiser. The idea of a theory of values and the metaphor of value-landscapes. Humanities and Social Sciences Communications, 11(1):1–10, 2024. find
803. Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. The empty signifier problem: Towards clearer paradigms for operationalising" alignment" in large language models. arXiv preprint arXiv:2310.02457, 2023b. — [arXiv:2310.02457](https://arxiv.org/abs/2310.02457) find
804. Neel Guha, Christie Lawrence, Lindsey A Gailmard, Kit Rodolfa, Faiz Surani, Rishi Bommasani, Inioluwa Raji, Mariano-Florentino Cuéllar, Colleen Honigsberg, Percy Liang, et al. Ai regulation has its own alignment problem: The technical and institutional feasibility of disclosure, registration, licens… find
805. Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, et al. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint arXiv:1802.07228, 2018. find
806. Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An Overview of Catastrophic AI Risks. arXiv preprint arXiv:2306.12001, 2023. — [arXiv:2306.12001](https://arxiv.org/abs/2306.12001) find
807. Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D Griffin. Use of LLMs for illicit purposes: Threats, prevention measures, and vulnerabilities. arXiv preprint arXiv:2308.12833, 2023. — [arXiv:2308.12833](https://arxiv.org/abs/2308.12833) find
808. Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min-Yen Kan, and William Yang Wang. On the Risk of Misinformation Pollution with Large Language Models. arXiv preprint arXiv:2305.13661, 2023b.
809. Giovanni Spitale, Nikola Biller-Andorno, and Federico Germani. AI model GPT-3 (dis) informs us better than humans. arXiv preprint arXiv:2301.11924, 2023. — [arXiv:2301.11924](https://arxiv.org/abs/2301.11924) find
810. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019.
811. Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G Parker, and Munmun De Choudhury. Synthetic lies: Understanding ai-generated misinformation and evaluating algorithmic and human solutions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–20, 2023f. find
812. Josh A Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. Generative language models and automated influence operations: Emerging threats and potential mitigations. arXiv preprint arXiv:2301.04246, 2023. — [arXiv:2301.04246](https://arxiv.org/abs/2301.04246) find
813. Kai-Cheng Yang and Filippo Menczer. Anatomy of an AI-powered malicious social botnet. arXiv preprint arXiv:2307.16336, 2023. — [arXiv:2307.16336](https://arxiv.org/abs/2307.16336) find
814. Eyal Aharoni, Sharlene Fernandes, Daniel J. Brady, Caelan Alexander, Michael Criner, Kara Queen, Javier Rando, Eddy Nahmias, and Victor Crespo. Attributions toward artificial agents in a modified moral turing test. Scientific Reports, 14(1):8458, Apr 2024. ISSN 2045-2322. doi: 10.1038/s41598-0 24-58… — [doi:10.1038/s41598-0](https://doi.org/10.1038/s41598-0) find
815. Micah Musser. A cost analysis of generative language models and influence operations. arXiv preprint arXiv:2308.03740, 2023. — [arXiv:2308.03740](https://arxiv.org/abs/2308.03740) find
816. Ben Buchanan, Andrew Lohn, and Micah Musser. Truth, lies, and automation: How language models could change disinformation. Center for Security and Emerging Technology, 2021. find
817. Eugene Bagdasaryan and Vitaly Shmatikov. Spinning language models: Risks of propaganda-as-a-service and countermeasures. In 2022 IEEE Symposium on Security and Privacy (SP), pages 769–786. find
818. Emilio Ferrara. GenAI against humanity: Nefarious applications of generative artificial intelligence and large language models. arXiv preprint arXiv:2310.00737, 2023. find
819. Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. arXiv preprint arXiv:2303.05453, 2023c.
820. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 214–… find
821. Bobby Chesney and Danielle Citron. Deep fakes: A looming challenge for privacy, democracy, and national security. Calif. L. Rev., 107:1753, 2019. find
822. Solcyré Burga. How a New Bill Could Protect Against Deepfakes. Time, 2024. https://time.com/6 590711/deepfake-protection-federal-bill/. ↗
823. Nicolas Pröllochs. Community-based fact-checking on Twitter’s Birdwatch platform. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 794–805, 2022. find
824. Beizhe Hu, Qiang Sheng, Juan Cao, Yuhui Shi, Yang Li, Danding Wang, and Peng Qi. Bad actor, good advisor: Exploring the role of large language models in fake news detection. arXiv preprint arXiv:2309.12247, 2023c. find
825. Canyu Chen and Kai Shu. Combating misinformation in the age of llms: Opportunities and challenges. arXiv preprint arXiv:2311.05656, 2023.
826. Steve Newman. Cybersecurity and ai: The evolving security landscape, 2024. URL https://www.sa fe.ai/blog/cybersecurity-and-ai-the-evolving-security-landscape. ↗
827. Rabimba Karanjai. Targeted phishing campaigns using large scale language models. arXiv preprint arXiv:2301.00665, 2022. — [arXiv:2301.00665](https://arxiv.org/abs/2301.00665) find
828. Julian Hazell. Large language models can be used to effectively scale spear phishing campaigns. arXiv preprint arXiv:2305.06972, 2023. — [arXiv:2305.06972](https://arxiv.org/abs/2305.06972) find
829. Miloslova Plachkinova and Chris Maurer. Security breach at target. Journal of Information Systems Education, 29(1):11–20, 2018. find
830. F. Salahdine and N. Kaabouch. Social Engineering Attacks: A Survey. Future Internet, 11(4), 2019. doi: https://doi.org/10.3390/fi11040089. — [doi:10.3390/fi11040089](https://doi.org/10.3390/fi11040089) ↗
831. Checkpoint Research. OPWNAI: AI That Can Save the Day or Hack It Away, 2022. https://rese arch.checkpoint.com/2022/opwnai-ai-that-can-save-the-day-or-hack-it-away/. Accessed on: 30 January, 2024. find
832. Checkpoint Research. OPWNAI: Cybercriminals Starting to Use ChatGPT, 2023. https://research.checkpoint.com/2023/opwnai-cybercriminals-starting-to-use-chatgpt/. Accessed on: 30 January, 2024. ↗
833. Arthur Erzberger. WormGPT and FraudGPT – The Rise of Malicious LLMs. https://www.trustwav e.com/en-us/resources/blogs/spiderlabs-blog/wormgpt-and-fraudgpt-the-rise-of-malic ious-llms/, 2023. ↗
834. Ehsan Aghaei, Xi Niu, Waseem Shadid, and Ehab Al-Shaer. SecureBERT: A Domain-Specific Language Model for Cybersecurity. In International Conference on Security and Privacy in Communication Systems, pages 39–56. Springer, 2022. find
835. Mohamed Amine Ferrag, Mthandazo Ndhlovu, Norbert Tihanyi, Lucas C Cordeiro, Merouane Debbah, and Thierry Lestable. Revolutionizing Cyber Threat Detection with Large Language Models. arXiv preprint arXiv:2306.14263, 2023. — [arXiv:2306.14263](https://arxiv.org/abs/2306.14263) find
836. Nicholas Carlini. A LLM assisted exploitation of AI-Guardian. arXiv preprint arXiv:2307.15008, 2023b. — [arXiv:2307.15008](https://arxiv.org/abs/2307.15008) find
837. Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. PentestGPT: An LLM-empowered automatic penetration testing tool. arXiv preprint arXiv:2308.06782, 2023b. — [arXiv:2308.06782](https://arxiv.org/abs/2308.06782) find
838. Xinyuan Sun, Davide Crapis, Matt Stephenson, Barnabé Monnot, Thomas Thiery, and Jonathan Passerat-Palmbach. Cooperative ai via decentralized commitment devices. arXiv preprint arXiv:2311.07815, 2023. — [arXiv:2311.07815](https://arxiv.org/abs/2311.07815) find
839. Michael Roytman and Ed Bellis. Modern Vulnerability Management: Predictive Cybersecurity. Artech House, 2023. find
840. Christian Schroeder de Witt, Samuel Sokota, J Zico Kolter, Jakob Foerster, and Martin Strohmeier. Perfectly secure steganography using minimum entropy coupling. arXiv preprint arXiv:2210.14889, 2022. — [arXiv:2210.14889](https://arxiv.org/abs/2210.14889) find
841. Tim Franzmeyer, Stephen Marcus McAleer, Joao F Henriques, Jakob Nicolaus Foerster, Philip Torr, Adel Bibi, and Christian Schroeder de Witt. Illusory attacks: Detectability matters in adversarial attacks on sequential decision-makers. In The Twelfth International Conference on Learning Representation… find
842. Jiacen Xu, Jack W Stokes, Geoff McDonald, Xuesong Bai, David Marshall, Siyue Wang, Adith Swaminathan, and Zhou Li. Autoattacker: A large language model guided system to implement automatic cyber-attacks. arXiv preprint arXiv:2403.01038, 2024b. — [arXiv:2403.01038](https://arxiv.org/abs/2403.01038) find
843. Matteo Boffa, Giulia Milan, Luca Vassio, Idilio Drago, Marco Mellia, and Zied Ben Houidi. Towards nlp-based processing of honeypot logs. In 2022 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pages 314–321. find
844. Schroeder de Witt, Hawra Milani, Klaudia Krawiecka, Swapneel Mehta, Carla Cremer, and Martin Strohmeier. Multi-Agent Security Workshop at NeurIPS’23, 2023. URL https://neurips.cc/vir tual/2023/workshop/66520. ↗
845. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking Black Box Large Language Models in Twenty Queries. arXiv preprint arXiv:2310.08419, 2023.
846. Lilian Weng, Vik Goel, and Andrea Vallone. Using GPT-4 for content moderation, 2023. https://openai.com/blog/using-gpt-4-for-content-moderation. Accessed on: 30 January, 2024. ↗
847. Ethan Edwards. Large Language Models will be Great for Censorship, 2023. https://ethanedwards.substack.com/p/large-language-models-will-be-great. Accessed on: January 30, 2024. ↗
848. Mikal Hem. Evading the censors: Critical journalism in authoritarian states. Reuters Institute Fellowship Paper University of Oxford, 2014. find
849. Jeffrey Knockel, Christopher Parsons, Lotus Ruan, Ruohan Xiong, Jedidiah Crandall, and Ron Deibert. We chat, they watch: How international users unwittingly build up WeChat’s Chinese censorship apparatus. Citizen Lab Research Report 127, University of Toronto, 2020. find
850. Steven Feldstein. The global expansion of AI surveillance. Carnegie Endowment for International Peace Washington, DC, 2019. find
851. Zack Whittaker. NSA improperly collected Americans’ phone records for a second time, documents reveal, 2019. https://techcrunch.com/2019/06/26/nsa-improper-phone-records-collectio n/. Accessed on: 30th January, 2024. ↗
852. Sahar Aziz. A domestic terror law could quash political dissent in the US. Al Jazeera, 2020. https://www.aljazeera.com/opinions/2020/6/13/a-domestic-terror-law-could-quash-political -dissent-in-the-us. Accessed on: 9 March, 2024. ↗
853. Omar Zahzah. Digital Apartheid: Palestinians Being Silenced on Social Media. Al Jazeera, 2021. https://www.aljazeera.com/opinions/2021/5/13/social-media-companies-are-trying-t o-silence-palestinian-voices. ↗
854. Neil M Richards. The dangers of surveillance. Harv. L. Rev., 126:1934, 2012. find
855. Pratyusha Ria Kalluri, William Agnew, Myra Cheng, Kentrell Owens, Luca Soldaini, and Abeba Birhane. The Surveillance AI Pipeline. arXiv preprint arXiv:2309.15084, 2023. — [arXiv:2309.15084](https://arxiv.org/abs/2309.15084) find
856. Hannah Ellis-Petersen. WhatsApp sues Indian government over ‘mass surveillance’ internet laws, 2021. https://www.theguardian.com/world/2021/may/26/whatsapp-sues-indian-government-ove r-mass-surveillance-internet-laws. Accessed on: 30th January, 2024. ↗
857. Morgan Meaker. Ukraine’s War Brings Autonomous Weapons to the Front Lines. Wired, 2023. https://www.wired.co.uk/article/ukraine-war-autonomous-weapons-frontlines. Accessed on: 1 February, 2024. ↗
858. David Hambling. Ukrainian AI Attack Drones May Be Killing Without Human Oversight. New Scientist, 2023. https://www.newscientist.com/article/2397389-ukrainian-ai-attack-drones-m ay-be-killing-without-human-oversight/. Accessed on: 1 February, 2024. ↗
859. Amnesty International. Israel and Occupied Palestinian Territories: Automated Apartheid: How facial recognition fragments, segregates and controls Palestinians in the OPT, 2023. https://www.amnest y.org/en/documents/mde15/6701/2023/en/. Accessed on: 1 February, 2024. ↗
860. Andrew Tarantola. Palantir shows off an AI that can go to war. Engadget, 2023. URL https://www.engadget.com/palantir-shows-off-an-ai-that-can-go-to-war-180513781.html. ↗
861. Frank Sauer and Niklas Schörnig. Killer drones: The ‘silver bullet’ of democratic warfare? Security Dialogue, 43(4):363–380, 2012. find
862. Noel Sharkey. Saying ‘no!’to lethal autonomous targeting. In Military Ethics and Emerging Technologies, pages 132–146. Routledge, 2016. find
863. Paul Scharre. Autonomous weapons and operational risk, 2016. find
864. Heather M Roff and Richard Moyes. Meaningful human control, artificial intelligence and autonomous weapons. In Briefing Paper Prepared for the Informal Meeting of Experts on Lethal Au-Tonomous Weapons Systems, UN Convention on Certain Conventional Weapons, 2016. find
865. Future of Life Institute. Autonomous Weapons Open Letter: AI & Robotics Researchers, 2016. https://futureoflife.org/open-letter/open-letter-autonomous-weapons-ai-robotics/. Accessed on: January 30, 2024. ↗
866. Jakob N. Foerster. Open letter against armed drones to the German Social Democratic Party, 2020. find
867. Boston Dynamics. General Purpose Robots Should Not Be Weaponized, 2022. https://bostondyna mics.com/news/general-purpose-robots-should-not-be-weaponized/. Accessed on: January 30, 2024. find
868. Caroline Christie. After dropping ‘don’t be evil,’ Google looks for new words to justify its military projects, 2018. https://www.documentjournal.com/2018/06/google-quietly-abandons-empha sis-on-their-dont-be-evil-motto/. Accessed on: January 27, 2024. ↗
869. Sam Biddle. OpenAI Quietly Deletes Ban on Using ChatGPT for “Military and Warfare”, 2024. https://theintercept.com/2024/01/12/open-ai-military-ban-chatgpt/. Accessed on: January 27, 2024. ↗
870. Michael A Skinnider, R Greg Stacey, David S Wishart, and Leonard J Foster. Chemical language models enable navigation in sparsely populated chemical space. Nature Machine Intelligence, 2021. find
871. Michael Moret, Irene Pachon Angona, Leandro Cotos, Shen Yan, Kenneth Atz, Cyrill Brunner, Martin Baumgartner, Francesca Grisoni, and Gisbert Schneider. Leveraging molecular structure and bioactivity with chemical language models for de novo drug design. Nature Communications, 2023. find
872. Jonas B Sandbrink. Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools. arXiv preprint arXiv:2306.13952, 2023. — [arXiv:2306.13952](https://arxiv.org/abs/2306.13952) find
873. Anjali Gopal, Nathan Helm-Burger, Lenni Justen, Emily H Soice, Tiffany Tzeng, Geetha Jeyapragasan, Simon Grimm, Benjamin Mueller, and Kevin M Esvelt. Will releasing the weights of large language models grant widespread access to pandemic agents? arXiv preprint arXiv:2310.18233, 2023. — [arXiv:2310.18233](https://arxiv.org/abs/2310.18233) find
874. Emily H Soice, Rafael Rocha, Kimberlee Cordova, Michael Specter, and Kevin M Esvelt. Can large language models democratize access to dual-use biotechnology? arXiv preprint arXiv:2306.03809, 2023. — [arXiv:2306.03809](https://arxiv.org/abs/2306.03809) find
875. Christopher A. Mouton, Caleb Lucas, and Ella Guest. The Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study, 2024. https://www.rand.org/pubs/research_r eports/RRA2977-2.html. ↗
876. Tejal Patwardhan, Kevin Liu, Todor Markov, Neil Chowdhury, Dillon Leet, Natalie Cone, Caitlin Maltbie, Joost Huizinga, Carroll Wainwright, Shawn Jackson, Steven Adler, Rocco Casagrande, and Aleksander Madry. Building an early warning system for LLM-aided biological threat creation, 2024. https://ope… find
877. Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 2024.
878. Gary Marcus. When looked at carefully, OpenAI’s new study on GPT-4 and bioweapons is deeply worrisome, 2024. https://garymarcus.substack.com/p/when-looked-at-carefully-openais. Accessed on: 9 March, 2024. ↗
879. Jaime M Yassif, Shayna Korol, and Angela Kane. Guarding against catastrophic biological risks: preventing state biological weapon development and use by shaping intentions. Health security, 2023. find
880. Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.10130, 2023. — [arXiv:2303.10130](https://arxiv.org/abs/2303.10130) find
881. Chloe Xiang. Startup Uses AI Chatbot to Provide Mental Health Counseling and Then Realizes It ‘Feels Weird’. Vice, 2023a. https://www.vice.com/en/article/4ax9yw/startup-uses-ai-cha tbot-to-provide-mental-health-counseling-and-then-realizes-it-feels-weird. Accessed on: 30 January, 2024. ↗
882. Debby RE Cotton, Peter A Cotton, and J Reuben Shipway. Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, pages 1–12, 2023. find
883. World Health Organization. Ethics and Governance of Artificial Intelligence for Health: WHO Guidance. World Health Organization, Geneva, 2021. find
884. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the Reliability of Watermarks for Large Language Models. arXiv preprint arXiv:2306.04634, 2023.
885. Hanlin Zhang, Benjamin L Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak. Watermarks in the sand: Impossibility of strong watermarking for generative models. arXiv preprint arXiv:2311.04378, 2023e. — [arXiv:2311.04378](https://arxiv.org/abs/2311.04378) find
886. Baihe Huang, Banghua Zhu, Hanlin Zhu, Jason D Lee, Jiantao Jiao, and Michael I Jordan. Towards Optimal Statistical Watermarking. arXiv preprint arXiv:2312.07930, 2023e. — [arXiv:2312.07930](https://arxiv.org/abs/2312.07930) find
887. Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, and Heng Huang. Unbiased watermark for large language models. arXiv preprint arXiv:2310.10669, 2023d. — [arXiv:2310.10669](https://arxiv.org/abs/2310.10669) find
888. Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, and Alon Halevy. Factuality challenges in the era of large language models. arXiv preprint arXiv:2310.05189, 2023.
889. Elizabeth Seger, Noemi Dreksler, Richard Moulange, Emily Dardaman, Jonas Schuett, K Wei, Christoph Winter, Mackenzie Arnold, Seán Ó hÉigeartaigh, and Anton Korinek. Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source obje… — [arXiv:2311.09227](https://arxiv.org/abs/2311.09227) find
890. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, ..., and Thom… — Llama 2: Open Foundation and Fine-Tuned Chat Models
891. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. DecodingTrust: A Comprehensive Assessment of Trus… — DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
892. Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J. Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias. ArXiv, abs/2306.15895, 2023c. URL https://api.semanticscholar.org/CorpusID: 259275123. ↗
893. Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. Dialect prejudice predicts AI decisions about people’s character, employability, and criminality. arXiv preprint arXiv:2403.00742, 2024.
894. Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. Unveiling the Implicit Toxicity in Large Language Models. arXiv preprint arXiv:2311.17391, 2023.
895. Yau-Shian Wang and Yingshan Chang. Toxicity Detection with Generative Prompt-based Inference, 2022. find
896. Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, and Markus Pauly. The Self-Perception and Political Biases of ChatGPT, 2023. find
897. Rida Qadri, Renee Shelby, Cynthia L Bennett, and Emily Denton. AI’s Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 506–517, 2023. find
898. Akshita Jha, Vinodkumar Prabhakaran, Remi Denton, Sarah Laszlo, Shachi Dave, Rida Qadri, Chandan K Reddy, and Sunipa Dev. Beyond the Surface: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation. arXiv preprint arXiv:2401.06310, 2024. — [arXiv:2401.06310](https://arxiv.org/abs/2401.06310) find
899. Nathan Lambert. Big Tech’s LLM evals are just marketing, 2023. URL https://www.interconnect s.ai/p/evals-are-marketing. ↗
900. Andrew Blair-Stanek, Nils Holzenberger, and Benjamin Van Durme. OpenAI Cribbed Our Tax Example, But Can GPT-4 Really Do Tax? Tax Notes, August 14 2023. https://www.taxnotes.com/fea tured-analysis/openai-cribbed-our-tax-example-can-gpt-4-really-do-tax/2023/08/11/ 7h0hc. ↗
901. Anthropic. Long context prompting for Claude 2.1, 2023d. https://www.anthropic.com/index/cl aude-2-1-prompting. ↗
902. Sara Merken. New York lawyers sanctioned for using fake ChatGPT cases in legal brief, 2023. URL https://www.reuters.com/legal/new-york-lawyers-sanctioned-using-fake-chatgpt-cases -legal-brief-2023-06-22/. ↗
903. Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. LM-Polygraph: Uncertainty Estimation for Language Models, 2023. find
904. Cordula Kupfer, Rita Prassl, Jürgen Fleiß, Christine Malin, Stefan Thalmann, and Bettina Kubicek. Check the box! How to deal with automation bias in AI-based personnel selection. Frontiers in Psychology, 14:1118723, 2023. find
905. Linda J Skitka, Kathleen L Mosier, and Mark Burdick. Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5):991–1006, 1999. find
906. Lucía Vicente and Helena Matute. Humans inherit artificial intelligence biases. Scientific Reports, 13 (1):15737, 2023. find
907. Celeste Kidd and Abeba Birhane. How AI can distort human beliefs. Science, 380(6651):1222–1223, 2023. find
908. Helen Nissenbaum. Privacy in context: Technology, policy, and the integrity of social life. Stanford University Press, 2020. find
909. Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi. Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory. arXiv preprint arXiv:2310.17884, 2023. — [arXiv:2310.17884](https://arxiv.org/abs/2310.17884) find
910. David H Autor. Why are there still so many jobs? The history and future of workplace automation. Journal of economic perspectives, 29(3):3–30, 2015. find
911. Angus Maddison. The World Economy: Historical Statistics. OECD Development Centre, 2004. find
912. Anton Korinek and Joseph E Stiglitz. Artificial intelligence and its implications for income distribution and unemployment. In The Economics of Artificial Intelligence: An agenda, pages 349–390. University of Chicago Press, 2019. find
913. Daniel Susskind. Technological Unemployment. In Justin B. Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew M. Young, and Baobao Zhang, editors, The Oxford Handbook of AI Governance. Oxford Academic, 2023. doi: 10.1093/oxfordhb/9780197579329.013.42. URL https://do… — [doi:10.1093/oxfordhb/9780197579329.013.42](https://doi.org/10.1093/oxfordhb/9780197579329.013.42) find
914. David H Autor. Work of the Past, Work of the Future. In AEA Papers and Proceedings, volume 109, pages 1–32. American Economic Association, 2019. find
915. Xiang Hui, Oren Reshef, and Luofeng Zhou. The Short-Term Effects of Generative Artificial Intelligence on Employment: Evidence from an Online Labor Market. Available at SSRN 4527336, 2023. find
916. Anton Korinek and Megan Juelfs. Preparing for the (Non-Existent?) Future of Work. In Justin B. Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew M. Young, and Baobao Zhang, editors, The Oxford Handbook of AI Governance. Oxford Academic, 2023. doi: 10.1093/oxfordhb… — [doi:10.1093/oxfordhb/9780197579329.013.44](https://doi.org/10.1093/oxfordhb/9780197579329.013.44) find
917. Anton Korinek. Scenario Planning for an A(G)I Future. IMF Finance & Development Magazine, 60 (4):30–33, December 2023. URL https://www.imf.org/en/Publications/fandd/issues/2023/12/Scenario-Planning-for-an-AGI-future-Anton-korinek. ↗
918. Stephanie A Bell. AI and Job Quality: Insights from Frontline Workers. Available at SSRN 4337611, 2022. find
919. Antonio A. Casilli and Julian Posada. The Platformisation of Labor and Society. In Mark Graham and William H. Dutton, editors, Society and the Internet: How Networks of Information and Communication Are Changing Our Lives. Oxford University Press, Oxford, vol. 2 edition, 2019. find
920. Nanjala Nyabola. ChatGPT and the sweatshops powering the digital age. Al Jazeera, 2023. https://www.aljazeera.com/opinions/2023/1/23/sweatshops-are-making-our-digital-age-work. Accessed on: January 8, 2024. ↗
921. Stephanie A Bell and Anton Korinek. AI’s Economic Peril. Journal of Democracy, 34(4):151–161, 2023. find
922. Gallup. How to improve employee engagement in the workplace, 2022. find
923. Valerio Capraro, Austin Lentsch, Daron Acemoglu, Selin Akgun, Aisel Akhmedova, Ennio Bilancini, Jean-François Bonnefon, Pablo Brañas-Garza, Luigi Butera, Karen M Douglas, et al. The impact of generative artificial intelligence on socioeconomic inequalities and policy making. arXiv preprint arXiv:240… — [arXiv:2401.05377](https://arxiv.org/abs/2401.05377) find
924. Loukas Karabarbounis and Brent Neiman. The global decline of the labor share. The Quarterly journal of economics, 129(1):61–103, 2014. find
925. Anton Korinek and Joseph E Stiglitz. Steering technological progress, 2020. find
926. Katya Klinova and Anton Korinek. AI and shared prosperity. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 645–651, 2021. find
927. Erik Brynjolfsson, Danielle Li, and Lindsey R Raymond. Generative AI at work. Technical report, National Bureau of Economic Research, 2023. find
928. Jon Kleinberg and Manish Raghavan. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences, 118(22):e2018340118, 2021. find
929. Meena Jagadeesan, Michael I Jordan, Jacob Steinhardt, and Nika Haghtalab. Improved Bayes Risk Can Yield Reduced Social Welfare Under Competition. arXiv preprint arXiv:2306.14670, 2023. find
930. Anton Korinek and Jai Vipra. Concentrating Intelligence: Scaling Laws and Market Structure in Generative AI. Prepared for Economic Policy, 39, 2024. find
931. Anthony Barrett, Jessica Newman, and Brandie Nonnecke. Policy Brief on AI Risk Management Standards for General-Purpose AI Systems (GPAIS) and Foundation Models, 2023. https://cltc.berkeley.edu/publication/policy-brief-on-ai-risk-management-standards-for-general -purpose-ai-systems-gpais-and-founda… ↗
932. Irene Solaiman. The gradient of generative AI release: Methods and considerations. In Proceedings of the 2023 ACM conference on fairness, accountability, and transparency, pages 111–122, 2023. find
933. Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of AI on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590, 2023. — [arXiv:2302.06590](https://arxiv.org/abs/2302.06590) find
934. Jan Van Dijk. The digital divide. John Wiley & Sons, 2020. find
935. Andrew McAfee, Daniel Rock, and Erik Brynjolfsson. How to Capitalize on Generative AI. Harvard Business Review, Nov-Dec 2023. URL https://hbr.org/2023/11/how-to-capitalize-on-gener ative-ai. ↗
936. Ethan R. Mollick and Lilach Mollick. Using AI to Implement Effective Teaching Strategies in Classrooms: Five Strategies, Including Prompts. The Wharton School Research Paper, Mar 2023. doi: 10.2139/ssrn.4391243. URL https://ssrn.com/abstract=4391243. — [doi:10.2139/ssrn.4391243](https://doi.org/10.2139/ssrn.4391243) ↗
937. Sal Khan. Harnessing GPT-4 so that all students benefit. A nonprofit approach for equal access. Khan Academy, 2023. find
938. L Yan, L Sha, L Zhao, Y Li, R Martinez-Maldonado, and G Chen. Practical and ethical challenges of large language models in education: A systematic literature review. arXiv preprint arXiv:2303.13379, 2023. find
939. Shakked Noy and Whitney Zhang. Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654):187–192, 2023. doi: 10.1126/science.adh2586. — [doi:10.1126/science.adh2586](https://doi.org/10.1126/science.adh2586) find
940. Fabrizio Dell’Acqua, Edward McFowland, Ethan R Mollick, Hila Lifshitz-Assaf, Katherine Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, and Karim R Lakhani. Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and qu… find
941. Scott Shane and Daisuke Wakabayashi. ‘The Business of War’: Google Employees Protest Work for the Pentagon. The New York Times, 2018. https://www.nytimes.com/2018/04/04/technology/ google-letter-ceo-pentagon-project.html. Accessed on: January 8, 2024. ↗
942. Kristalina Georgieva. AI will transform the global economy. Let’s make sure it benefits humanity. 2024. URL https://www.imf.org/en/Blogs/Articles/2024/01/14/ai-will-transform-the-globa l-economy-lets-make-sure-it-benefits-humanity. ↗
943. Anton Korinek, Martin Schindler, and Joseph Stiglitz. Technological Progress, Artificial Intelligence, and Inclusive Growth. Technical report, International Monetary Fund, 2022. find
944. Similarweb. chatgpt.com, 2024. https://www.similarweb.com/website/chatgpt.com/. Accessed on: 30 January, 2024. ↗
945. Unesco. Transforming education from within: current trends in the status and development of teachers; World Teachers’ Day 2022, 2022. https://unesdoc.unesco.org/ark:/48223/pf0000383002. Accessed on: January 30, 2024. ↗
946. Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. Do multilingual language models think better in English? arXiv preprint arXiv:2308.01223, 2023.
947. Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. The language barrier: Dissecting safety challenges of LLMs in multilingual contexts. arXiv preprint arXiv:2401.13136, 2024.
948. Lora Aroyo, Alex Taylor, Mark Diaz, Christopher Homan, Alicia Parrish, Gregory Serapio-García, Vinodkumar Prabhakaran, and Ding Wang. DICES dataset: Diversity in conversational ai evaluation for safety. Advances in Neural Information Processing Systems, 36, 2024. find
949. Carolin Holtermann, Paul Röttger, Timm Dill, and Anne Lauscher. Evaluating the elementary multilingual capabilities of large language models with MultiQ. arXiv preprint arXiv:2403.03814, 2024. find
950. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R Mortensen, Noah A Smith, and Yulia Tsvetkov. Do all languages cost the same? Tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
951. Aleksandar Petrov, Emanuele La Malfa, Philip HS Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages. Neural Information Processing Systems (NeurIPS), 2023b.
952. Council of Europe. Artificial Intelligence - Work in Progress, 2023. https://www.coe.int/en/web/a rtificial-intelligence/work-in-progress\#01EN. Accessed on: January 30, 2024. ↗
953. Paul Voigt and Axel Von dem Bussche. The EU general data protection regulation (GDPR). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 2017. find
954. Council of the European Union. Proposal for a Regulation of the European Parliament and of the Council on Artificial Intelligence (Artificial Intelligence Act), 2024. https://data.consilium.eur opa.eu/doc/document/ST-5662-2024-INIT/en/pdf. ↗
955. Jane Finlayson-Brown and Susana Ng. China brings into force Regulations on the Administration of Deep Synthesis of Internet Technology, 2023. https://www.allenovery.com/en-gb/global/blogs/data-hub/china-brings-into-force-regulations-on-the-administration-of-deep-synth esis-of-internet-technology-ad… ↗
956. Eric Goldman. An introduction to the california consumer privacy act (ccpa). Santa Clara Univ. Legal Studies Research Paper, 2020. find
957. OECD. Recommendation of the Council on Artificial Intelligence. OECD/LEGAL/0449, 2019. find
958. AI-HLEG, High-Level Expert Group on Artificial Intelligence. Ethics Guidelines for Trustworthy AI. Technical report, European Commission, 2019. https://www.aepd.es/sites/default/files/20 19-12/ai-ethics-guidelines.pdf. ↗
959. Unesco. Recommendation on the Ethics of Artificial Intelligence, 2021. https://unesdoc.unesco.org/ark:/48223/pf0000381137. ↗
960. National Cyber Security Center. Guidelines for Secure AI System Development, 2019. find
961. Staff PAI. Partnership on AI, 2017. https://partnershiponai.org/. Accessed on: January 8, 2024. ↗
962. Anthropic. Anthropic’s Responsible Scaling Policy: Version 1.0, 2023e. https://www-cdn.anthropic.com/1adf000c8f675958c2ee23805d91aaade1cd4613/responsible-scaling-policy.pdf. ↗
963. Google AI. Google AI Principles, 2018. URL https://ai.google/responsibility/principles/. ↗
964. ISO, International Organization for Standardization. ISO/IEC 38505-1:2017 – Information technology – Governance of IT – Governance of data, 2017. find
965. IEEE, Institute of Electrical and Electronics Engineers. IEEE Standard for Enterprise Architecture Description (EAD) Version 2, 2018. https://standards.ieee.org/wp-content/uploads/import/ documents/other/ead_v2.pdf. ↗
966. NIST, National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0), 2023. find
967. Tim Bradshaw, Madhumita Murgia, George Hammond, and Camilla Hodgson. How Microsoft’s Multibillion-dollar Alliance with OpenAI Really Works. Financial Times, 2023. https://www.ft.com/content/458b162d-c97a-4464-8afc-72d65afb28ed. ↗
968. RAIL Team. Responsible AI Licenses, 2024. https://www.licenses.ai/ai-licenses. ↗
969. Justin B. Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew M. Young, and Baobao Zhang. The Oxford Handbook of AI Governance. Oxford University Press, 2022a. doi: 10.1093/oxfordhb/9780197579329.001.0001. — [doi:10.1093/oxfordhb/9780197579329.001.0001](https://doi.org/10.1093/oxfordhb/9780197579329.001.0001) find
970. Michael Veale, Kira Matus, and Robert Gorwa. AI and Global Governance: Modalities, Rationales, Tensions. Annual Review of Law and Social Science, 19, 2023. find
971. Allan Dafoe. AI governance: a research agenda. Governance of AI Program, Future of Humanity Institute, University of Oxford: Oxford, UK, 1442:1443, 2018. find
972. Markus Anderljung and Alexis Carlier. Some AI Governance Research Ideas, 2021. https://docs.g oogle.com/document/d/13LJhP3ksrcEBKxYFG5GkJaC2UoxHKUYAHCRdRlpePEc/edit. Accessed on: January 30, 2024. ↗
973. Yonadav Shavit, Sandhini Agarwal, Miles Brundage, Steven Adler, Cullen O’Keefe, Rosie Campbell, Teddy Lee, Pamela Mishkin, Tyna Eloundou, Alan Hickey, et al. Practices for Governing Agentic AI Systems, 2023. find
974. Nathan Barnard and Erin Robertson. AI Governance and Strategy: A List of Research Agendas and Work That Could Be Done. Less Wrong, 2024. https://www.lesswrong.com/posts/Zn73PkYWGK YjLiBAf/. ↗
975. Inioluwa Deborah Raji. The Anatomy of AI Audits: Form, Process, and Consequences. The Oxford Handbook of AI Governance, 2021. URL https://doi.org/10.1093/oxfordhb/9780197579329.0 13.28. — [doi:10.1093/oxfordhb/9780197579329.0](https://doi.org/10.1093/oxfordhb/9780197579329.0) ↗
976. Sayash Kapoor, Rishi Bommasani, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Peter Cihon, Aspen Hopkins, Kevin Bankston, Stella Biderman, Miranda Bogen, et al. On the Societal Impact of Open Foundation Models. arXiv, 2024. find
977. Atoosa Kasirzadeh. Two Types of AI Existential Risk: Decisive and Accumulative. arXiv preprint arXiv:2401.07836, 2024. — [arXiv:2401.07836](https://arxiv.org/abs/2401.07836) find
978. Future of Life Institute. Pause Giant AI Experiments: An Open Letter, 2023. https://futureoflife.org/open-letter/pause-giant-ai-experiments/. Accessed on: January 30, 2024. ↗
979. Nora Belrose. AI Pause Will Likely Backfire (Guest Post), 2023. https://bounded-regret.ghost.io/ai-pause-will-likely-backfire-by-nora/. Accessed on: January 30, 2024. ↗
980. Sasha Luccioni. The Call to Halt ‘Dangerous’ AI Research Ignores a Simple Truth. Wired, 2023. https://www.wired.com/story/the-call-to-halt-dangerous-ai-research-ignores-a-simpl e-truth/. ↗
981. Gary E Marchant. The growing gap between emerging technologies and the law. Springer, 2011. find
982. Andrea Jelinek and Wojciech Wiewiórowski. Open letter on EDPB budget proposal for 2023, 2022. https://edpb.europa.eu/our-work-tools/our-documents/letters/open-letter-edpb-budge t-proposal-2023_en. Accessed on: January 11, 2024. ↗
983. Andrew Tutt. An FDA for algorithms. Admin. L. Rev., 69:83, 2017. find
984. Jess Weatherbed. The EU still needs to get its AI Act together. The Verge, Jun 2023. URL https://www.theverge.com/2023/6/29/23777239/eu-ai-act-artificial-intelligence-regulatio ns-europe. ↗
985. Adam Au. China vs. US Approaches to AI Governance. The Diplomat, 2023. https://thediplomat.com/2023/10/china-vs-us-approaches-to-ai-governance/. Accessed on: 1 February, 2024. ↗
986. Carlos Ignacio Gutierrez Gaviria. The role of artificial intelligence in pushing the boundaries of US regulation: A systematic review. Santa Clara High Tech. LJ, 38:123, 2022. find
987. Ansh Bhatnagar and Devyani Gajjar. Policy Implications of Artificial Intelligence (AI). Research briefing, UK Parliament, 2024. https://post.parliament.uk/research-briefings/post-pn-0 708/. Accessed on: February 8, 2024. ↗
988. Alex Engler. A comprehensive and distributed approach to AI regulation. Brookings, 2023. find
989. Gillian K Hadfield and Jack Clark. Regulatory Markets: The Future of AI Governance. arXiv preprint arXiv:2304.04914, 2023. — [arXiv:2304.04914](https://arxiv.org/abs/2304.04914) find
990. Center for AI Safety, Aidan O’Gara, Corin Katzke, and Dan Hendrycks. AI Safety Newsletter \#32: Measuring and Reducing Hazardous Knowledge in LLMs. AI Safety Newsletter, 2024. https://newsletter.safe.ai/p/ai-safety-newsletter-32-measuring. ↗
991. Sasha Costanza-Chock, Inioluwa Deborah Raji, and Joy Buolamwini. Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1571–1583, 2022. find
992. Jyoti Mann. OpenAI recruiters are trying to lure Google AI employees with $10 million pay packets, report says. Business Insider, 2023. https://www.businessinsider.com/openai-recruiters-l uring-google-ai-employees-10-million-compensation-package-2023-11. ↗
993. Amanda Askell, Miles Brundage, and Gillian Hadfield. The Role of Cooperation in Responsible AI Development, jul 2019. URL http://arxiv.org/abs/1907.04534. arXiv:1907.04534 [cs]. — [arXiv:1907.04534](https://arxiv.org/abs/1907.04534) ↗
994. Henry Blodget. Mark Zuckerberg On Innovation. Business Insider, 2009. https://www.businessin sider.com/mark-zuckerberg-innovation-2009-10?r=US&IR=T. Accessed on: January 30, 2024. ↗
995. Stuart Armstrong, Nick Bostrom, and Carl Shulman. Racing to the precipice: a model of artificial intelligence development. AI & society, 31:201–206, 2016. find
996. Markus Anderljung and Anton Korinek. Frontier AI Regulation: Safeguards Amid Rapid Progress. Lawfare, January 2024. URL https://www.lawfaremedia.org/article/frontier-ai-regulatio n-safeguards-amid-rapid-progress. ↗
997. Samir Saran and Shashank Mattoo. Big Tech vs. Red Tech: The Diminishing of Democracy in the Digital Age. Observer Research Foundation, February, 15, 2022. find
998. Cat Zakrzewski. Tech Companies Spent Almost $70 million Lobbying Washington In 2021 As Congress Sought To Rein In Their Power, 2022. https://www.washingtonpost.com/technology/2022/01/ 21/tech-lobbying-in-washington/. Accessed on: December 3, 2023. ↗
999. Juho Lindman, Jukka Makinen, and Eero Kasanen. Big Tech’s power, political corporate social responsibility and regulation. Journal of Information Technology, page 02683962221113596, 2023. find
1000. Nani Reventlow. How Artificial Intelligence Impacts Marginalised Groups, 2021. find
1001. Anthropic. The Long-Term Benefit Trust, 2023f. https://www.anthropic.com/news/the-long-ter m-benefit-trust. ↗
1002. Kevin Roose. The Chaos at OpenAI, Explained. The New York Times, 2023. https://www.nytimes.com/2023/11/21/briefing/open-ai-sam-altman-microsoft.html. Accessed on: March 10, 2024. ↗
1003. Robert Reich. The frantic battle over OpenAI shows that money triumphs in the end. The Guardian, 2023. https://www.theguardian.com/commentisfree/2023/nov/28/artificial-intelligenc e-openai-non-profit-money. Accessed on: March 10, 2024. ↗
1004. Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. Co-writing with opinionated language models affects users’ views. In Proceedings of the 2023 CHI conference on human factors in computing systems, 2023. find
1005. Lawrence Lessig. Code and Other Laws of Cyberspace, Version 2.0. Basic Books, 2006. URL https://lessig.org/product/codev2. ↗
1006. Javier Sánchez-Monedero, Lina Dencik, and Lilian Edwards. What does it mean to ‘solve’ the problem of discrimination in hiring?: social, technical and legal perspectives from the UK on automated hiring systems. In FAT* ’20: Proceedings of the 2020 Conference on Fairness, Accountability, and Transpar… find
1007. Acm, January 2020. doi: 10.1145/3351095.3372849. URL https://doi.org/10.1145/3351095.3372849. — [doi:10.1145/3351095.3372849](https://doi.org/10.1145/3351095.3372849) ↗
1008. Derek O’Callaghan, Derek Greene, Maura Conway, Joe Carthy, and Pádraig Cunningham. Down the (white) rabbit hole: The extreme right and online recommender systems. Social Science Computer Review, 33(4):459–478, 2015. find
1009. Mohamed Abdalla and Moustafa Abdalla. The grey hoodie project: Big tobacco, big tech, and the threat on academic integrity. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 287–297, 2021. find
1010. Inioluwa Deborah Raji, SASHA COSTANZA Chock, and J Buolamwini. Change from the outside: Towards credible third-party audits of ai systems. Missing links in AI governance, page 5, 2023. find
1011. Lewis Ho, Joslyn Barnhart, Robert Trager, Yoshua Bengio, Miles Brundage, Allison Carnegie, Rumman Chowdhury, Allan Dafoe, Gillian Hadfield, Margaret Levi, et al. International institutions for advanced AI. arXiv preprint arXiv:2307.04699, 2023. — [arXiv:2307.04699](https://arxiv.org/abs/2307.04699) find
1012. Robert Trager, Ben Harack, Anka Reuel, Allison Carnegie, Lennart Heim, Lewis Ho, Sarah Kreps, Ranjit Lall, Owen Larter, Seán Ó hÉigeartaigh, et al. International Governance of Civilian AI: A Jurisdictional Certification Approach. arXiv preprint arXiv:2308.15514, 2023. — [arXiv:2308.15514](https://arxiv.org/abs/2308.15514) find
1013. Matthijs Maas and José Jaime Villalobos. International AI Institutions: a literature review of models, examples, and proposals. Technical Report AI Foundations Report \#1, Legal Priorities Project, 2023. URL https://www.legalpriorities.org/research/international-ai-institutions. ↗
1014. Governments of Participating Countries. The Bletchley Declaration by Countries Attending the AI Safety Summit. AI Safety Summit 2023, November 2023. URL https://www.gov.uk/governmen t/publications/ai-safety-summit-2023-the-bletchley-declaration/the-bletchley-decla ration-by-countries-attending-the-a… ↗
1015. Charlotte Siegmann and Markus Anderljung. The Brussels effect and artificial intelligence: How EU regulation will impact the global AI market. arXiv preprint arXiv:2208.12645, 2022. — [arXiv:2208.12645](https://arxiv.org/abs/2208.12645) find
1016. Daniel Solove and Paul Schwartz. ALI Data Privacy: Overview and Black Letter Text. UCLA Law Review, 68:1252, 2022. find
1017. Nathalie A Smuha. From a “race to AI” to a “race to AI regulation”: regulatory competition for artificial intelligence. Law, Innovation and Technology, 13(1):57–84, 2021. doi: 10.1080/17579961.2 021.1898300. URL https://doi.org/10.1080/17579961.2021.1898300. — [doi:10.1080/17579961.2](https://doi.org/10.1080/17579961.2) ↗
1018. Ministry of Foreign Affairs, People’s Republic of China. Global AI Governance Initiative, 2023. https://www.mfa.gov.cn/eng/wjdt_665385/2649_665393/202310/t20231020_11164834.html. ↗
1019. Sam Meacham. A Race to Extinction: How Great Power Competition Is Making Artificial Intelligence Existentially Dangerous, 2023. https://hir.harvard.edu/a-race-to-extinction-how-great -power-competition-is-making-artificial-intelligence-existentially-dangerous/. Accessed on: 1 February, 2024. ↗
1020. FAR AI. Scientists Call For International Cooperation on AI Red Lines, 2024. https://far.ai/pos t/2024-03-idais-beijing/. ↗
1021. Zoe Stanley-Lockman and Lena Trabucco. NATO’s role in responsible AI governance in military affairs. In The Oxford Handbook of AI Governance. Oxford University Press, 2022. find
1022. Markus Anderljung and Julian Hazell. Protecting Society from AI Misuse: When are Restrictions on Capabilities Warranted? arXiv preprint arXiv:2303.09377, 2023. — [arXiv:2303.09377](https://arxiv.org/abs/2303.09377) find
1023. Richard Blumenthal and Josh Hawley. Bipartisan Framework for U.S. AI Act, 2023. https://www.blumenthal.senate.gov/imo/media/doc/09072023bipartisanaiframework.pdf. Accessed on: 1 February, 2024. ↗
1024. Wex. Joint and Several Liability, 2023. https://www.law.cornell.edu/wex/joint_and_several_li ability. Accessed on: 9 March, 2024. ↗
1025. Miriam Buiten, Alexandre De Streel, and Martin Peitz. The law and economics of AI liability. Computer Law & Security Review, 48:105794, 2023. find
1026. Timotheus Kampik, Adnane Mansour, Olivier Boissier, Sabrina Kirrane, Julian Padget, Terry R Payne, Munindar P Singh, Valentina Tamma, and Antoine Zimmermann. Governance of autonomous agents on the web: challenges and opportunities. ACM Transactions on Internet Technology, 22(4):1–31, 2022. find
1027. Alan Chan, Carson Ezell, Max Kaufmann, Kevin Wei, Lewis Hammond, Herbie Bradley, Emma Bluemke, Nitarshan Rajkumar, David Krueger, Noam Kolt, et al. Visibility into AI Agents. arXiv preprint arXiv:2401.13138, 2024. — [arXiv:2401.13138](https://arxiv.org/abs/2401.13138) find
1028. Justin B. Bullock, Hsini Huang, and Kyoung-Cheol (Casey) Kim. Machine Intelligence, Bureaucracy, and Human Control. Perspectives on Public Management and Governance, 5(2):187–196, June 2022b. doi: 10.1093/ppmgov/gvac006. URL https://doi.org/10.1093/ppmgov/gvac006. — [doi:10.1093/ppmgov/gvac006](https://doi.org/10.1093/ppmgov/gvac006) ↗
1029. Chyng Wen Tee and Christopher Hian Ann Ting. Cross-section of mini flash crashes and their detection by a state-space approach. Available at SSRN 3402783, 2019. find
1030. CFTC and SEC. Findings Regarding the Market Events of May 6, 2010: Report of the Staffs of the CFTC and SEC to the Joint Advisory Committee on Emerging Regulatory Issues. Technical report, Cftc, sep 2010. URL https://www.sec.gov/files/marketevents-report.pdf. ↗
1031. Zachary S Levine, Scott A Hale, and Luciano Floridi. The October 2014 United States Treasury bond flash crash and the contributory effect of mini flash crashes. PloS one, 12(11):e0186688, 2017. find
1032. Noam Kolt. Governing ai agents. Available at SSRN, 2024. find
1033. Yacine Jernite, Huu Nguyen, Stella Biderman, Anna Rogers, Maraim Masoud, Valentin Danchev, Samson Tan, Alexandra Sasha Luccioni, Nishant Subramani, Isaac Johnson, et al. Data governance in the age of large-scale data-driven language technology. In Proceedings of the 2022 ACM Conference on Fairness,… find
1034. Tim Hwang. Computational power and the social impact of artificial intelligence. arXiv preprint arXiv:1803.08971, 2018. — [arXiv:1803.08971](https://arxiv.org/abs/1803.08971) find
1035. Girish Sastry, Lennart Heim, Haydn Belfield, Markus Anderljung, Miles Brundage, Julian Hazell, Cullen O’Keefe, Gillian K Hadfield, Richard Ngo, Konstantin Pilz, et al. Computing Power and the Governance of Artificial Intelligence. arXiv preprint arXiv:2402.08797, 2024. — [arXiv:2402.08797](https://arxiv.org/abs/2402.08797) find
1036. Steven Gonzalez Monserrate. The cloud is material: On the environmental impacts of computation and data storage. MIT Schwarzman College of Computing, 2022. find
1037. Philipp Hacker, Andreas Engel, and Marco Mauer. Regulating ChatGPT and other Large Generative AI Models. In FAccT ’23: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 1112–1123. find
1038. Acm, June 2023. doi: 10.1145/3593013.3594067. URL https://doi.org/10.1145/3593013.3594067. — [doi:10.1145/3593013.3594067](https://doi.org/10.1145/3593013.3594067) ↗
1039. U.S. Congress. DEEPFAKES Accountability Act, 2023. Introduced in House on September 20, 2023, https://www.congress.gov/bill/118th-congress/house-bill/5586/text. ↗
1040. UK Parliament. Online Safety Act 2023, 2023. Number 50, http://www.legislation.gov.uk/ukpga/2023/50/whole/enacted. ↗
1041. Eurojust and Europol. Common challenges in combating cybercrime. Technical report, Europol and Eurojust Public Information, 2019. https://www.europol.europa.eu/cms/sites/default/file s/documents/common_challenges_in_combating_cybercrime_2018.pdf. ↗
1042. Chloe Xiang. ‘He Would Still Be Here’: Man Dies by Suicide After Talking with AI Chatbot, Widow Says. Vice, 2023b. https://www.vice.com/en/article/pkadgm/man-dies-by-suicide-after-t alking-with-ai-chatbot-widow-says. ↗
1043. David Gray Widder, Sarah West, and Meredith Whittaker. Open (for business): Big tech, concentrated power, and the political economy of open AI. Concentrated Power, and the Political Economy of Open AI (August 17, 2023), 2023. find
1044. Toby Shevlane. Structured access: an emerging paradigm for safe AI deployment. arXiv preprint arXiv:2201.05159, 2022. — [arXiv:2201.05159](https://arxiv.org/abs/2201.05159) find
1045. Geoffrey Fowler. Snapchat tried to make a safe AI. It chats with me about booze and sex. The Washington Post, 2023. https://www.washingtonpost.com/technology/2023/03/14/snapchat-m yai/. Accessed on: 1 February, 2024. ↗
1046. UK Government. Emerging Processes for Frontier AI Safety, 2023. https://www.gov.uk/governmen t/publications/emerging-processes-for-frontier-ai-safety/emerging-processes-for-f rontier-ai-safety. ↗
1047. Jonas Schuett, Noemi Dreksler, Markus Anderljung, David McCaffary, Lennart Heim, Emma Bluemke, and Ben Garfinkel. Towards best practices in AGI safety and governance: A survey of expert opinion. arXiv preprint arXiv:2305.07153, 2023. — [arXiv:2305.07153](https://arxiv.org/abs/2305.07153) find
1048. Brad Smith. Developing and Deploying AI Responsibly: Elements of an Effective Legislative Framework to Regulate AI, 2023. https://blogs.microsoft.com/on-the-issues/2023/09/12/developin g-and-deploying-ai-responsibly-elements-of-an-effective-legislative-framework-to-r egulate-ai/. Accessed on: Januar… ↗
1049. Adam Thierer. Existential Risks and Global Governance Issues around AI and Robotics. R Street Policy Study, 291, June 2023. find
1050. Jeremy Howard. AI Safety and the Age of Dislightenment, 2023. https://www.fast.ai/posts/2023 -11-07-dislightenment.html. Accessed on: January 8, 2024. ↗
1051. Victor Ojewale, Ryan Steed, Briana Vecchione, Abeba Birhane, and Inioluwa Deborah Raji. Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling. arXiv preprint arXiv:2402.17861, 2024. — [arXiv:2402.17861](https://arxiv.org/abs/2402.17861) find
1052. Yonadav Shavit. What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring. arXiv preprint arXiv:2303.11341, 2023. — [arXiv:2303.11341](https://arxiv.org/abs/2303.11341) find
1053. Seán Ó hÉigeartaigh, Yolanda Lannquist, Alexandru Marcoci, Jaime Sevilla, Mónica Alejandra Ul-loa Ruiz, Yaqub Chaudhary, Tim Schreier, Zach Stein-Perlman, and Jeffrey Ladish. Do Companies’ AI Safety Policies Meet Government Best Practice?, 2023. http://lcfi.ac.uk/news-and-events/ news/2023/oct/31/ai… ↗
1054. OpenAI. OpenAI’s Approach to Frontier Risk, 2023d. https://openai.com/global-affairs/our-a pproach-to-frontier-risk. ↗
1055. Google DeepMind. AI Safety Summit: An Update on Our Approach to Safety and Responsibility, 2023. https://deepmind.google/public-policy/ai-summit-policies/. ↗
1056. Daniel J Solove. The Limitations of Privacy Rights. Notre Dame L. Rev., 98:975, 2022. find
1057. Jennafer Shae Roberts and Laura N Montoya. Decolonisation, Global Data Law, and Indigenous Data Sovereignty. arXiv preprint arXiv:2208.04700, 2022. — [arXiv:2208.04700](https://arxiv.org/abs/2208.04700) find
1058. Katherine Lee, A. Feder Cooper, James Grimmelmann, and Daphne Ippolito. AI and Law: The Next Generation. SSRN, July 2023b. find
1059. Dami Choi, Yonadav Shavit, and David Duvenaud. Tools for Verifying Neural Models’ Training Data. arXiv preprint arXiv:2307.00682, 2023b. — [arXiv:2307.00682](https://arxiv.org/abs/2307.00682) find
1060. Sanjam Garg, Aarushi Goel, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody, Guru-Vamsi Policharla, and Mingyuan Wang. Experimenting with zero-knowledge proofs of training. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 1880–1894, 2023. find
1061. Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. Simfluence: Modeling the influence of individual training examples by simulating training runs. arXiv preprint arXiv:2303.08114, 2023. find
1062. Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and fair use. arXiv preprint arXiv:2303.15715, 2023b. find
1063. Peter Henderson, Tatsunori Hashimoto, and Mark Lemley. Where’s the Liability in harmful AI Speech? Journal of Free Speech Law, 3:589, 2023c. find
1064. David L Schwartz and Max Rogers. Inventorless Inventions? The Constitutional Conundrum of AI-Produced Inventions. February, 3:22–05, 2022. find
1065. Kate Knibbs. Why This Award-Winning Piece of AI Art Can’t Be Copyrighted. Wired, 2023. https://www.wired.com/story/ai-art-copyright-matthew-allen/. Accessed on: January 30, 2024. ↗
1066. Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. find
1067. Will Knight. OpenAI’s CEO Says the Age of Giant AI Models Is Already Over. Wired, 2023. https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already -over/. Accessed on: 1 February, 2024. ↗
1068. Jess Whittlestone and Jack Clark. Why and How Governments Should Monitor AI Development. arXiv preprint arXiv:2108.12427, 2021. — [arXiv:2108.12427](https://arxiv.org/abs/2108.12427) find
1069. Konstantin Pilz, Lennart Heim, and Nicholas Brown. Increased Compute Efficiency and the Diffusion of AI Capabilities. arXiv preprint arXiv:2311.15377, 2023. — [arXiv:2311.15377](https://arxiv.org/abs/2311.15377) find
1070. Miles Brundage, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian Hadfield, et al. Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims, apr 2020. URL http://arxiv.org/abs/2004.07213. arXiv:2004.07213 [cs]. ↗
1071. Mauricio Baker. Nuclear Arms Control Verification and Lessons for AI Treaties. arXiv preprint arXiv:2304.04123, 2023. — [arXiv:2304.04123](https://arxiv.org/abs/2304.04123) find
1072. Lennart Heim, Tim Fist, Janet Egan, Sihao Huang, Stephen Zekany, Robert Trager, Michael A Osborne, and Noa Zilberman. Governing Through the Cloud: The Intermediary Role of Compute Providers in AI Regulation. arXiv preprint arXiv:2403.08501, 2024. — [arXiv:2403.08501](https://arxiv.org/abs/2403.08501) find
1073. Janet Egan and Lennart Heim. Oversight for Frontier AI through a Know-Your-Customer Scheme for Compute Providers. arXiv preprint arXiv:2310.13625, 2023. — [arXiv:2310.13625](https://arxiv.org/abs/2310.13625) find
1074. Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. DiLoCo: Distributed Low-Communication Training of Language Models. arXiv preprint arXiv:2311.08105, 2023. — [arXiv:2311.08105](https://arxiv.org/abs/2311.08105) find
1075. Fraser Mince, Dzung Dinh, Jonas Kgomo, Neil Thompson, and Sara Hooker. The grand illusion: The myth of software portability and implications for ml progress. Advances in Neural Information Processing Systems, 36, 2024. find
1076. Raman Arora, Sanjeev Arora, Joan Bruna, Nadav Cohen, Simon Du, Rong Ge, Suriya Gunasekar, Chi Jin, Jason Lee, Tengyu Ma, et al. Theory of deep learning, 2020. find
1077. Nate Soares and Benja Fallenstein. Aligning superintelligence with human interests: A technical research agenda. Machine Intelligence Research Institute (MIRI) technical report, 8, 2014. find
1078. Fact Sheet: Biden-Harris Administration Secures Voluntary Commitments from Leading Artificial Intelligence Companies to Manage the Risks Posed by AI, 2023. https://www.whitehouse.gov/bri efing-room/statements-releases/2023/07/21/fact-sheet-biden-harris-administration-s ecures-voluntary-commitments-f… ↗
1079. Michelle Nichols. UN Security Council meets for first time on AI risks, July 2023. https://www.re uters.com/technology/un-security-council-meets-first-time-ai-risks-2023-07-18/. Accessed on: January 8, 2023. ↗
1080. Renee Shelby, Shalaleh Rismani, Kathryn Henne, Ajung Moon, Negar Rostamzadeh, Paul Nicholas, YILLA-AKBARI N’MAH, Jess Gallegos, Andrew Smart, and Gurleen Virk. Identifying sociotechnical harms of algorithmic systems: Scoping a taxonomy for harm reduction. arXiv preprint arXiv:2210.05791, 2022. — [arXiv:2210.05791](https://arxiv.org/abs/2210.05791) find
1081. Irene Solaiman, Zeerak Talat, William Agnew, Lama Ahmad, Dylan Baker, Su Lin Blodgett, Hal Daumé III, Jesse Dodge, Ellie Evans, Sara Hooker, et al. Evaluating the Social Impact of Generative AI Systems in Systems and Society. arXiv preprint arXiv:2306.05949, 2023. — [arXiv:2306.05949](https://arxiv.org/abs/2306.05949) find
1082. Andrew Critch and Stuart Russell. TASRA: A Taxonomy and Analysis of Societal-Scale Risks from AI. arXiv preprint arXiv:2306.06924, 2023. — [arXiv:2306.06924](https://arxiv.org/abs/2306.06924) find
1083. Stuart Russell, Daniel Dewey, and Max Tegmark. Research priorities for robust and beneficial artificial intelligence. AI magazine, 36(4):105–114, 2015. find
1084. Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 123–129, 2018. find
1085. Emily Dinan, Gavin Abercrombie, A Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. Anticipating safety issues in e2e conversational ai: Framework and tooling. arXiv preprint arXiv:2107.03451, 2021. — [arXiv:2107.03451](https://arxiv.org/abs/2107.03451) find
1086. Ross Gruetzemacher, Florian E Dorner, Niko Bernaola-Alvarez, Charlie Giattino, and David Manheim. Forecasting AI progress: A research agenda. Technological Forecasting and Social Change, 170:120909, 2021. find
1087. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169, 2023.
1088. Chip Huyen. Open challenges in LLM research, August 2023. https://huyenchip.com/2023/08/16/llm-research-open-challenges.html. Accessed on: January 8, 2023. ↗
1089. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, ..., and Ji-Rong Wen. A Survey of Large Language Models, 2023…
1090. Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. AI alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023b.
1091. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022.
1092. Naomi Saphra, Eve Fleisig, Kyunghyun Cho, and Adam Lopez. First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models. arXiv preprint arXiv:2311.05020, 2023. find
1093. Oana Ignat, Zhijing Jin, Artem Abzaliev, Laura Biester, Santiago Castro, Naihao Deng, Xinyi Gao, Aylin Gunal, Jacky He, Ashkan Kazemi, et al. A PhD Student’s Perspective on Research in NLP in the Era of Very Large Language Models. arXiv preprint arXiv:2305.12544, 2023. — [arXiv:2305.12544](https://arxiv.org/abs/2305.12544) find

Updated 2 November 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/anwar-2024-foundational-challenges-assuring-alignment-safety/