Elamur Research⌕
☰
home / papers / 2021 / Unsolved Problems in ML Safety

Unsolved Problems in ML Safety

Paper · 2021 · Dan Hendrycks, Nicholas Carlini, John Schulman et al.

Summary

This position paper by Hendrycks, Carlini, Schulman and Steinhardt asks which technical problems the ML community should prioritize to make powerful, widely deployed models safe. It is a research agenda, not an empirical study, so it introduces no dataset, model or benchmark score. The authors argue that safety work must start now for three reasons. Design decisions made early are hard to undo. ML systems behave differently from conventional software and can show emergent capabilities. Competitive pressure pushes companies to cut corners.

They split the field into four connected areas:

  • Robustness: withstanding adversaries and rare, extreme events.
  • Monitoring: detecting misuse, anomalies and hidden model functionality.
  • Alignment: making models pursue human values that are hard to specify.
  • Systemic Safety: using ML to reduce wider risks such as cyberattacks.

For each area they explain the motivation and propose concrete directions, including stress-test benchmarks with "wild" distribution shifts and attack models broader than small p-norm perturbations. The paper revises and extends the 2016 agenda of Amodei and Olah et al. It matters because it gives researchers a structured, layered view of where safety effort is most needed.

Key points

  • The four problem areas are Robustness, Monitoring, Alignment and Systemic Safety. They are presented as complementary layers in a "Swiss cheese" model of protection.
  • The authors cite a Department of Defense report by Frola and Miller. It found that about 75% of the most critical safety decisions happen early in a system's development.
  • Real-world failures are used as motivation. The 2010 Flash Crash erased roughly a trillion dollars of stock value within minutes. The Boeing 737 MAX shortcuts led to two crashes and 346 deaths.
  • For adversarial robustness, the paper urges going beyond ℓp-bounded perturbations. Suggested alternatives include perceptible attacks and attacks not specified in advance.
  • The agenda explicitly updates Amodei and Olah et al. (2016). It adds newer issues such as emergent capabilities of large pretrained models and risks from the deployment context.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,463 citations🎓University of California, BerkeleyBerkeley · United States209 works · 244,782 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSafety Research · 626 worksEngineering · 188 worksSafety, Risk, Reliability and Quality · 14 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 587 works

Cite

Cite as Dan Hendrycks, Nicholas Carlini, John Schulman and Jacob Steinhardt. 2021. Unsolved Problems in ML Safety. arXiv:2109.13916.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2109.13916 · arXiv 2109.13916

@misc{hendrycks2021unsolved,
  title = {{Unsolved Problems in ML Safety}},
  author = {Dan Hendrycks, Nicholas Carlini, John Schulman and Jacob Steinhardt},
  year = {2021},
  doi = {10.48550/arxiv.2109.13916},
  eprint = {2109.13916},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2109.13916}
}

References

210 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Signed by approximately 2000 AI researchers. “Asilomar AI Principles”. In: (2017). find
  2. Signed by 30000+ people. “Autonomous Weapons: An Open Letter from AI and Robotics Researchers”. In: (2015). find
  3. Martín Abadi, Andy Chu, I. Goodfellow, H. B. McMahan, Ilya Mironov, Kunal Talwar, and L. Zhang. “Deep Learning with Differential Privacy”. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (2016). find
  4. Zeeshan Ahmad, A. Khan, W. Cheah, J. Abdullah, and Farhan Ahmad. “Network intrusion detection system: A systematic study of machine learning and deep learning approaches”. In: Trans. Emerg. Telecommun. Technol. (2021). find
  5. Dario Amodei, Christopher Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dandelion Mané. “Concrete Problems in AI Safety”. In: ArXiv (2016).
  6. Ross J. Anderson and Roger Needham. “Programming Satan’s Computer”. In: Computer Science Today. 1995. find
  7. Drago Anguelov. Machine Learning for Autonomous Driving. 2019. URL : https://www.youtube.com/watch?v=Q0nGo2-y0xY. ↗
  8. Anish Athalye, Nicholas Carlini, and David A. Wagner. “Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples”. In: ICML. 2018. find
  9. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. “Program Synthesis with Large Language Models”. In: ArXiv (2021).
  10. Eugene Bagdasaryan and Vitaly Shmatikov. “Blind Backdoors in Deep Learning Models”. In: USENIX Security Symposium. 2021. find
  11. Abhijit Bendale and Terrance Boult. “Towards Open Set Deep Networks”. In: CVPR (2016). find
  12. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021). find
  13. Machine Learning at Berkeley. Alien Dreams: An Emerging Art Scene. URL : https://ml.berkeley.edu/blog/posts/clip-art/. ↗
  14. Victor Besnier, Andrei Bursuc, David Picard, and Alexandre Briot. “Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation”. In: ArXiv abs/2108.01634 (2021). — [arXiv:2108.01634](https://arxiv.org/abs/2108.01634) find
  15. Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. “Evasion attacks against machine learning at test time”. In: Joint European conference on machine learning and knowledge discovery in databases. Springer. 2013, pp. 387–402. find
  16. Abeba Birhane, Pratyusha Kalluri, D. Card, William Agnew, Ravit Dotan, and Michelle Bao. “The Values Encoded in Machine Learning Research”. In: ArXiv (2021). find
  17. Julian Bitterwolf, Alexander Meinke, and Matthias Hein. “Certifiably Adversarially Robust Detection of Out-of-Distribution Data”. In: NeurIPS (2020). find
  18. Rishi Bommasani et al. “On the Opportunities and Risks of Foundation Models”. In: ArXiv (2021).
  19. Nick Bostrom. “The Vulnerable World Hypothesis”. In: Global Policy (2019). find
  20. G. Botvin, C. Goldberg, E. M. Botvin, and L. Dusenbury. “Smoking behavior of adolescents exposed to cigarette advertising”. In: Public health reports (1993). find
  21. Wieland Brendel, Jonas Rauber, and Matthias Bethge. “Decision-based adversarial attacks: Reliable attacks against black-box machine learning models”. In: arXiv preprint arXiv:1712.04248 (2017). — [arXiv:1712.04248](https://arxiv.org/abs/1712.04248) find
  22. Philip Brickman and Donald Campbell. “Hedonic relativism and planning the good society”. In: 1971. find
  23. T. Brown, B. Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, G. Krüger, T. Henighan, R. Child, Aditya Ramesh, D. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, E… — Language Models are Few-Shot Learners
  24. Miles Brundage, Shahar Avin, Jack Clark, H. Toner, P. Eckersley, Ben Garfinkel, A. Dafoe, P. Scharre, T. Zeitzoff, Bobby Filar, H. Anderson, Heather Roff, Gregory C. Allen, J. Steinhardt, Carrick Flynn, Seán Ó hÉigeartaigh, S. Beard, Haydn Belfield, Sebastian Farquhar, Clare Lyle, Rebecca Crootof, O… find
  25. Miles Brundage, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian K. Hadfield, Heidy Khlaaf, Jingying Yang, H. Toner, Ruth Fong, Tegan Maharaj, P. W. Koh, Sara Hooker, J. Leung, Andrew Trask, Emma Bluemke, Jonathan Lebensbold, Cullen O’Keefe, Mark Koren, T. Ryffel, J. Rubinovitz,… find
  26. Erik Brynjolfsson and Adam Saunders. “What the GDP Gets Wrong (Why Managers Should Care)”. In: MIT Sloan Management Review (2009). find
  27. Ben Buchanan, John Bansemer, Dakota Cary, Jack Lucas, and Micah Musser. “Automating Cyber Attacks”. In: 2021. find
  28. Ben Buchanan, Andrew Lohn, Micah Musser, and Katerina Sedova. “Truth, Lies, and Automation”. In: 2021. find
  29. Nicholas Carlini and A. Terzis. “Poisoning and Backdooring Contrastive Learning”. In: ArXiv abs/2106.09667 (2021). — [arXiv:2106.09667](https://arxiv.org/abs/2106.09667) find
  30. Nicholas Carlini and David Wagner. “Towards evaluating the robustness of neural networks”. In: 2017 ieee symposium on security and privacy (sp). IEEE. 2017, pp. 39–57. find
  31. Y. Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. “Unlabeled Data Improves Adversarial Robustness”. In: NeurIPS. 2019. find
  32. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. “Emerging Properties in Self-Supervised Vision Transformers”. In: Proceedings of the International Conference on Computer Vision (ICCV). 2021.
  33. Dakota Cary and Daniel Cebul. “Destructive Cyber Operations and Machine Learning”. In: 2020. find
  34. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, J. Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea.… — Evaluating Large Language Models Trained on Code
  35. Steven Chen, Nicholas Carlini, and David A. Wagner. “Stateful Detection of Black-Box Adversarial Attacks”. In: Proceedings of the 1st ACM Workshop on Security and Privacy on Artificial Intelligence (2019). find
  36. Jack Clark and Dario Amodei. “Faulty Reward Functions in the Wild”. In: OpenAI (2016). find
  37. Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and J. Schulman. “Quantifying Generalization in Reinforcement Learning”. In: ICML. 2019. find
  38. Jeremy M. Cohen, Elan Rosenfeld, and J. Z. Kolter. “Certified Adversarial Robustness via Randomized Smoothing”. In: ICML. 2019. find
  39. North American Aerospace Defense Command and U.S. Northern Command Public Affairs. 2021. URL : https://www.af.mil/News/Article-Display/Article/2703548/norad-usnorthcom-lead-3rd-global-information-dominance-experiment/. ↗
  40. Andrew Critch and David Krueger. “AI Research Considerations for Human Existential Safety (ARCHES)”. In: ArXiv (2020). find
  41. Francesco Croce, Maksym Andriushchenko, V. Sehwag, Nicolas Flammarion, M. Chiang, Prateek Mittal, and Matthias Hein. “RobustBench: a standardized adversarial robustness benchmark”. In: ArXiv abs/2010.09670 (2020). — [arXiv:2010.09670](https://arxiv.org/abs/2010.09670) find
  42. Maria Cvach. “Monitor alarm fatigue: an integrative review”. In: Biomedical instrumentation & technology (2012). find
  43. Allan Dafoe. “AI governance: a research agenda”. In: Governance of AI Program, Future of Humanity Institute, University of Oxford: Oxford, UK (2018). find
  44. Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R. McKee, Joel Z. Leibo, Kate Larson, and Thore Graepel. “Open Problems in Cooperative AI”. In: ArXiv (2020). find
  45. Mohamad H. Danesh and Alan Fern. “Out-of-Distribution Dynamics Detection: RL-Relevant Benchmarks and Results”. In: ArXiv abs/2107.04982 (2021). — [arXiv:2107.04982](https://arxiv.org/abs/2107.04982) find
  46. Department of Defense. “Quadrennial Defense Review Report”. In: (2001). find
  47. Laura DeNardis. “A history of internet security”. In: The history of information security. Elsevier, 2007. find
  48. Thomas G. Dietterich. “Robust artificial intelligence and robust human organizations”. In: Frontiers of Computer Science (2018). find
  49. Adrien Ecoffet and Joel Lehman. “Reinforcement Learning Under Moral Uncertainty”. In: ArXiv abs/2006.04734 (2021). — [arXiv:2006.04734](https://arxiv.org/abs/2006.04734) find
  50. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, E. Hovy, Hinrich Schütze, and Yoav Goldberg. “Measuring and Improving Consistency in Pretrained Language Models”. In: ArXiv (2021). find
  51. Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. “A rotation and a translation suffice: Fooling cnns with simple transformations”. In: arXiv (2018). find
  52. Facebook. Bringing People Closer Together. URL : https://about.fb.com/news/2018/01/news-feed-fyi-bringing-people-closer-together/. ↗
  53. Pablo Fajnzylber, Daniel Lederman, and Norman V. Loayza. “Inequality and Violent Crime”. In: The Journal of Law and Economics (2002). find
  54. Wendi Folkert. “Assessment results regarding Organization Designation Authorization (ODA) Unit Member (UM) Independence”. In: Aviation Safety (2021). find
  55. F. R. Frola and C. O. Miller. “System Safety in Aircraft Acquisition”. In: 1984. find
  56. Iason Gabriel. “Artificial Intelligence, Values and Alignment”. In: ArXiv (2020). find
  57. John Gall. “Systemantics: How Systems Work and Especially How They Fail”. In: 1977. find
  58. Sneha Gathani, Madelon Hulsebos, James Gale, P. Haas, and cCaugatay Demiralp. “Augmenting Decision Making via Interactive What-If Analysis”. In: 2021. find
  59. Helmut Geist and Eric Lambin. “What drives tropical deforestation?: a meta-analysis of proximate and underlying causes of deforestation based on subnational case study evidence”. In: 2001. find
  60. Yolanda Gil and Bart Selman. “A 20-Year Community Roadmap for Artificial Intelligence Research in the US”. In: ArXiv abs/1908.02624 (2019). — [arXiv:1908.02624](https://arxiv.org/abs/1908.02624) find
  61. J. Gilmer, Ryan P. Adams, I. Goodfellow, David G. Andersen, and George E. Dahl. “Motivating the Rules of the Game for Adversarial Example Research”. In: ArXiv (2018). find
  62. Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael A. Specter, and Lalana Kagal. “Explaining Explanations: An Overview of Interpretability of Machine Learning”. In: (2018). find
  63. Adam Gleave, Michael Dennis, Neel Kant, Cody Wild, Sergey Levine, and Stuart J. Russell. “Adversarial Policies: Attacking Deep Reinforcement Learning”. In: ICLR (2020). find
  64. Charles Goodhart. “Problems of Monetary Management: The UK Experience”. In: 1984. find
  65. Jeremy Greenwood. The third industrial revolution: Technology, productivity, and income inequality. 435. American Enterprise Institute, 1997. find
  66. Nathan Grinsztajn, Johan Ferret, O. Pietquin, P. Preux, and M. Geist. “There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning”. In: ArXiv abs/2106.04480 (2021). — [arXiv:2106.04480](https://arxiv.org/abs/2106.04480) find
  67. Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. “Badnets: Identifying vulnerabilities in the machine learning model supply chain”. In: arXiv preprint arXiv:1708.06733 (2017).
  68. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. “On Calibration of Modern Neural Networks”. In: ICML (2017). find
  69. Dylan Hadfield-Menell, A. Dragan, P. Abbeel, and Stuart J. Russell. “The Off-Switch Game”. In: IJCA (2017). find
  70. Dylan Hadfield-Menell, Stuart J. Russell, P. Abbeel, and A. Dragan. “Cooperative Inverse Reinforcement Learning”. In: NIPS. 2016. find
  71. Richard Harang and Ethan M. Rudd. SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection. 2020. find
  72. Moritz Hardt, Eric Price, and Nathan Srebro. “Equality of Opportunity in Supervised Learning”. In: NIPS. 2016.
  73. P. J. Heawood. “Map-Colour Theorem”. In: Proceedings of The London Mathematical Society (1949), pp. 161–175. find
  74. James Hedlund. “Risky business: safety regulations, risk compensation, and individual behavior”. In: Injury Prevention (2000). find
  75. Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. “The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization”. In: ICCV (2021). find
  76. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. “Aligning AI With Shared Human Values”. In: ICLR (2021).
  77. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. “Measuring Massive Multitask Language Understanding”. In: ICLR (2021).
  78. Dan Hendrycks and Thomas Dietterich. “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations”. In: Proceedings of the International Conference on Learning Representations (2019).
  79. Dan Hendrycks and Kevin Gimpel. “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks”. In: ICLR (2017). find
  80. Dan Hendrycks, Kimin Lee, and Mantas Mazeika. “Using Pre-Training Can Improve Model Robustness and Uncertainty”. In: ICML. 2019. find
  81. Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. “Deep Anomaly Detection with Outlier Exposure”. In: ICLR (2019). find
  82. Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and D. Song. “Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty”. In: NeurIPS. 2019. find
  83. Dan Hendrycks, Mantas Mazeika, Andy Zou, Sahil Patel, Christine Zhu, Jesus Navarro, Dawn Song, Bo Li, and Jacob Steinhardt. “What Would Jiminy Cricket Do? Towards Agents That Behave Morally”. In: NeurIPS (2021). find
  84. Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. “AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty”. In: ICLR (2020). find
  85. Dan Hendrycks, Kevin Zhao, Steven Basart, J. Steinhardt, and D. Song. “Natural Adversarial Examples”. In: CVPR (2021). find
  86. Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song, and Jacob Steinhardt. PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures. 2021. find
  87. T. Henighan, J. Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, A. Ramesh, Nick Ryder, Daniel M. Ziegler, J. Schulman, Dario Amodei, and Sam McCandlish. “Scaling Laws for Autoregressiv… — Scaling Laws for Autoregressive Generative Modeling
  88. J. Hestness, Sharan Narang, Newsha Ardalani, G. Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Y. Yang, and Yanqi Zhou. “Deep Learning Scaling is Predictable, Empirically”. In: ArXiv (2017).
  89. Grant Ho, Asaf Cidon, Lior Gavish, Marco Schweighauser, V. Paxson, S. Savage, G. Voelker, and David A. Wagner. “Detecting and Characterizing Lateral Phishing at Scale”. In: USENIX Security Symposium. 2019. find
  90. Sanghyun Hong, Nicholas Carlini, and A. Kurakin. “Handcrafted Backdoors in Deep Neural Networks”. In: ArXiv (2021). find
  91. Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. “Risks from Learned Optimization in Advanced Machine Learning Systems”. In: ArXiv (2019). find
  92. David Hume. A Treatise of Human Nature. 1739. find
  93. Geoffrey Irving, Paul Christiano, and Dario Amodei. “AI safety via debate”. In: ArXiv (2018).
  94. Michael C Jensen and William H Meckling. “Theory of the firm: Managerial behavior, agency costs and ownership structure”. In: Journal of financial economics 3.4 (1976), pp. 305–360. find
  95. Woojeong Jin, Suji Kim, Rahul Khanna, Dong-Ho Lee, Fred Morstatter, A. Galstyan, and Xiang Ren. “ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data”. In: ACL/IJCNLP. 2021. find
  96. Daniel Kahneman and Angus Deaton. “High income improves evaluation of life but not emotional well-being”. In: Proceedings of the National Academy of Sciences (2010). find
  97. Daniel Kang, Yi Sun, Dan Hendrycks, Tom B. Brown, and J. Steinhardt. “Testing Robustness Against Unforeseen Adversaries”. In: ArXiv (2019). find
  98. Kiran Karra, C. Ashcraft, and Neil Fendley. “The TrojAI Software Framework: An OpenSource tool for Embedding Trojans into Deep Learning Models”. In: ArXiv (2020). find
  99. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. “Alignment of Language Agents”. In: ArXiv (2021).
  100. A. Kirilenko, Mehrdad Samadi, A. Kyle, and Tugkan Tuzun. “The Flash Crash: The Impact of High Frequency Trading on an Electronic Market”. In: 2011. find
  101. Jack Koch, L. Langosco, J. Pfau, James Le, and Lee Sharkey. “Objective Robustness in Deep Reinforcement Learning”. In: ArXiv (2021). find
  102. P. W. Koh, Shiori Sagawa, H. Marklund, Sang Michael Xie, Marvin Zhang, A. Balsubramani, Wei hua Hu, Michihiro Yasunaga, Richard L. Phillips, Sara Beery, J. Leskovec, A. Kundaje, E. Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. “WILDS: A Benchmark of in-the-Wild Distribution Shifts”. In: ICM… find
  103. Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, and S. Legg. “Avoiding Side Effects By Considering Future Tasks”. In: NeurIPS (2020). find
  104. Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, N. Keskar, Shafiq R. Joty, R. Socher, and Nazneen Rajani. “GeDi: Generative Discriminator Guided Sequence Generation”. In: ArXiv (2020).
  105. Ethan Kross, Philippe Verduyn, Emre Demiralp, Jiyoung Park, David Seungjae Lee, Natalie Lin, Holly Shablack, John Jonides, and Oscar Ybarra. “Facebook use predicts declines in subjective well-being in young adults”. In: PloS one (). find
  106. David Krueger, Tegan Maharaj, and J. Leike. “Hidden Incentives for Auto-Induced Distributional Shift”. In: ArXiv abs/2009.09153 (2020). — [arXiv:2009.09153](https://arxiv.org/abs/2009.09153) find
  107. Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. “Accurate uncertainties for deep learning using calibrated regression”. In: ICML (2018). arXiv: 1807.00263. find
  108. Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. “Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration”. In: NeurIPS. 2019. find
  109. Ananya Kumar, Percy Liang, and Tengyu Ma. “Verified Uncertainty Calibration”. In: NeurIPS. 2019. find
  110. Patrick Ky. “Boeing 737 MAX Return to Service Report”. In: (2021). find
  111. Marie-Anne Lachaux, Baptiste Rozière, L. Chanussot, and Guillaume Lample. “Unsupervised Translation of Programming Languages”. In: ArXiv (2020).
  112. Cassidy Laidlaw, Sahil Singla, and S. Feizi. “Perceptual Adversarial Robustness: Defense Against Unseen Threat Models”. In: ICLR (2021). find
  113. Balaji Lakshminarayanan, A. Pritzel, and C. Blundell. “Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles”. In: NIPS. 2017. find
  114. Terran Lane and Carla E Brodley. “An application of machine learning to anomaly detection”. In: Proceedings of the 20th National Information Systems Security Conference. Vol. 377. Baltimore, USA. 1997, pp. 366–380. find
  115. Ralph Langner. “Stuxnet: Dissecting a Cyberwarfare Weapon”. In: IEEE Security & Privacy (2011). find
  116. Katarzyna de Lazari-Radek and Peter Singer. “The Point of View of the Universe: Sidgwick and Contemporary Ethics”. In: 2014. find
  117. Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. “Certified robustness to adversarial examples with differential privacy”. In: 2019 IEEE Symposium on Security and Privacy (SP). IEEE. 2019, pp. 656–672. find
  118. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. “Hallucinations in neural machine translation”. In: (2018). find
  119. J. Lehman, J. Clune, D. Misevic, C. Adami, L. Altenberg, Julie Beaulieu, P. Bentley, Samuel Bernard, G. Beslon, David M. Bryson, P. Chrabaszcz, Nick Cheney, Antoine Cully, S. Doncieux, F. Dyer, Kai Olav Ellefsen, R. Feldt, Stephan Fischer, S. Forrest, Antoine Frénoy, Christian Gagné, L. K. L. Goff,… find
  120. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. “Scalable agent alignment via reward modeling: a research direction”. In: ArXiv (2018).
  121. Nancy Leveson. “Engineering a Safer World: Systems Thinking Applied to Safety”. In: 2012. find
  122. Beishui Liao, Marija Slavkovik, and Leendert van der Torre. “Building Jiminy Cricket: An Architecture for Moral Agreements Among Stakeholders”. In: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (2019). find
  123. Stephanie Lin, Jacob Hilton, and Owain Evans. “TruthfulQA: Measuring How Models Mimic Human Falsehoods”. In: arXiv (2021).
  124. Rachel Luo, Aadyot Bhatnagar, Huan Wang, Caiming Xiong, Silvio Savarese, Yu Bai, Shengjia Zhao, and Stefano Ermon. “Localized Calibration: Metrics and Recalibration”. In: arXiv (2021). find
  125. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. “Towards Deep Learning Models Resistant to Adversarial Attacks”. In: ICLR (2018).
  126. Benoit Mandelbrot and Richard L. Hudson. “The Misbehavior of Markets: A Fractal View of Risk, Ruin, and Reward”. In: 2004. find
  127. David Manheim and Scott Garrabrant. “Categorizing Variants of Goodhart’s Law”. In: ArXiv (2018). find
  128. Filip Maric and Sana Stojanovic-Durdevic. “Formalizing IMO Problems and Solutions in Isabelle/HOL”. In: ThEdu@IJCAR. 2020. find
  129. Microsoft. URL : https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/. ↗
  130. Michael Mitzenmacher. “A Brief History of Generative Models for Power Law and Lognormal Distributions”. In: Internet Mathematics (2003). find
  131. Chaithanya Kumar Mummadi, Robin Hutmacher, K. Rambach, Evgeny Levinkov, T. Brox, and J. H. Metzen. “Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformation”. In: ArXiv (2021). find
  132. Toby Newberry and Toby Ord. “The Parliamentary Approach to Moral Uncertainty”. In: 2021. find
  133. Khanh Nguyen and Brendan T. O’Connor. “Posterior calibration and exploratory analysis for natural language processing models”. In: EMNLP. 2015.
  134. NSA. URL : https://ghidra-sre.org/. ↗
  135. Martha Nussbaum. “CAPABILITIES AS FUNDAMENTAL ENTITLEMENTS: SEN AND SOCIAL JUSTICE”. In: Feminist Economics 9 (2003), pp. 33 –59. find
  136. Rain Ottis. “Analysis of the 2007 Cyber Attacks Against Estonia from the Information Warfare Perspective”. In: 2008. find
  137. Yaniv Ovadia, E. Fertig, J. Ren, Zachary Nado, D. Sculley, S. Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift”. In: NeurIPS. 2019. find
  138. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. “An Empirical Cybersecurity Evaluation of GitHub Copilot’s Code Contributions”. In: ArXiv (2021). find
  139. Denis Peskov, Benny Cheng, Ahmed Elgohary, Joe Barrow, Cristian Danescu-Niculescu-Mizil, and Jordan L. Boyd-Graber. “It Takes Two to Lie: One to Lie, and One to Listen”. In: ACL. 2020. find
  140. Fabio Pierazzi, Feargus Pendlebury, Jacopo Cortellazzi, and Lorenzo Cavallaro. “Intriguing properties of adversarial ml attacks in the problem space”. In: 2020 IEEE Symposium on Security and Privacy (SP). IEEE. 2020, pp. 1332–1349. find
  141. Richard A. Posner. “Utilitarianism, Economics, and Legal Theory”. In: The Journal of Legal Studies (1979). find
  142. Omid Poursaeed, Tianxing Jiang, Harry Yang, Serge Belongie, and Ser-Nam Lim. “Robustness and Generalization via Generative Adversarial Training”. In: 2021. find
  143. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets”. In: ICLR MATH-AI Workshop. 2021.
  144. “Principle 1: Preoccupation with Failure”. In: Managing the Unexpected. John Wiley & Sons, Ltd, 2015. Chap. 3, pp. 45–61. ISBN : 9781119175834. URL : https://onlinelibrary.wiley.com/doi/abs/10.1002/9781119175834.ch03. — [doi:10.1002/9781119175834.ch03](https://doi.org/10.1002/9781119175834.ch03) ↗
  145. Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. “Learning Transferable Visual Models From Natural Language Supervision”. In: ICML. 2021.
  146. Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. “Certified Defenses against Adversarial Examples”. In: ICLR (2018). find
  147. Pranav Rajpurkar, Jeremy A. Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, T. Duan, D. Ding, Aarti Bagul, C. Langlotz, K. Shpanskaya, M. Lungren, and A. Ng. “CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning”. In: ArXiv (2017). find
  148. Theodore D. Raphael. “Integrative Complexity Theory and Forecasting International Crises: Berlin 1946-1962”. In: The Journal of Conflict Resolution (1982). find
  149. John Rawls. A Theory of Justice. Harvard University Press, 1999. find
  150. Alex Ray, Joshua Achiam, and Dario Amodei. “Benchmarking Safe Exploration in Deep Reinforcement Learning”. In: 2019. find
  151. Sylvestre-Alvise Rebuffi, Sven Gowal, D. A. Calian, Florian Stimberg, Olivia Wiles, and Timothy A. Mann. “Fixing Data Augmentation to Improve Adversarial Robustness”. In: ArXiv abs/2103.01946 (2021). — [arXiv:2103.01946](https://arxiv.org/abs/2103.01946) find
  152. V. Ridgway. “Dysfunctional Consequences of Performance Measurements”. In: Administrative Science Quarterly (1956). find
  153. Stuart Russell, Anthony Aguirre, Emilia Javorsky, and Max Tegmark. “Lethal Autonomous Weapons Exist; They Must Be Banned”. In: (2021). find
  154. Stuart J. Russell, Daniel Dewey, and Max Tegmark. “Research Priorities for Robust and Beneficial Artificial Intelligence”. In: AI Magazine (2015). find
  155. William Saunders, Girish Sastry, Andreas Stuhlmüller, and Owain Evans. “Trial without Error: Towards Safe Reinforcement Learning via Human Intervention”. In: AAMAS. 2018. find
  156. Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. “You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion”. In: USENIX (2021). find
  157. David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. “Hidden technical debt in machine learning systems”. In: Advances in neural information processing systems 28 (2015), pp. 2503– 2511. find
  158. A. Shafahi, W. R. Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, T. Dumitras, and T. Goldstein. “Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks”. In: NeurIPS. 2018. find
  159. Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander, P. Abbeel, and A. Dragan. “Preferences Implicit in the State of the World”. In: ICLR (2019). find
  160. Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. “Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition”. In: Proceedings of the 2016 acm sigsac conference on computer and communications security. 2016, pp. 1528–1540. find
  161. E. C. Shin, D. Song, and R. Moazzezi. “Recognizing Functions in Binaries with Neural Networks”. In: USENIX Security Symposium. 2015. find
  162. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. “Membership inference attacks against machine learning models”. In: 2017 IEEE Symposium on Security and Privacy (SP). IEEE. 2017, pp. 3–18.
  163. M. D.Amran Siddiqui, Alan Fern, Thomas G. Dietterich, and Weng Keen Wong. “Sequential feature explanations for anomaly detection”. In: ACM Transactions on Knowledge Discovery from Data (2019). find
  164. Henry Sidgwick. The Methods of Ethics. 1907. find
  165. Robin Sommer and Vern Paxson. “Outside the closed world: On using machine learning for network intrusion detection”. In: 2010 IEEE symposium on security and privacy. IEEE. 2010, pp. 305–316. find
  166. D. H. Stamatis. “Failure mode and effect analysis : FMEA from theory to execution”. In: ASQC Quality Press (1996). find
  167. Keith E. Stanovich, Richard F. West, and Maggie E. Toplak. “The Rationality Quotient: Toward a Test of Rational Thinking”. In: 2016. find
  168. Nick Carr Steve Miller Evan Reese. Shikata Ga Nai Encoder Still Going Strong. URL : https://www.fireeye . com / blog / threat - research / 2019 / 10 / shikata - ga - nai - encoder-still-going-strong.html. ↗
  169. Marilyn Strathern. “‘Improving ratings’: audit in the British University system”. In: European Review (1997). find
  170. Jonathan Stray. “Aligning AI Optimization to Community Well-Being”. In: International Journal of Community Well-Being (2020). find
  171. Jonathan Stray, Ivan Vendrov, Jeremy Nixon, Steven Adler, and Dylan Hadfield-Menell. “What are you optimizing for? Aligning Recommender Systems with Human Values”. In: ArXiv abs/2107.10939 (2021). — [arXiv:2107.10939](https://arxiv.org/abs/2107.10939) find
  172. David Stutz, Matthias Hein, and B. Schiele. “Confidence-Calibrated Adversarial Training: Generalizing to Unseen Attacks”. In: ICML. 2020. find
  173. Octavian Suciu, Scott E. Coull, and Jeffrey Johns. “Exploring Adversarial Examples in Malware Detection”. In: IEEE Security and Privacy Workshops (SPW) (2019). find
  174. RL Sumwalt, B Landsberg, and J Homendy. “Assumptions used in the safety assessment process and the effects of multiple alerts and indications on pilot performance”. In: District of Columbia: National Transportation Safety Board (2019). find
  175. Rebecca Sutton. Chromium-6 in US tap water. Environmental Working Group Washington, DC, 2010. find
  176. Publius Syrus. The Moral Sayings of Publius Syrus, a Roman Slave. L.E. Bernard & Company, 1856. find
  177. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. “Intriguing properties of neural networks”. In: arXiv preprint arXiv:1312.6199 (2013).
  178. Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. “CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances”. In: NeurIPS (2020). find
  179. Jihoon Tack, Sihyun Yu, Jongheon Jeong, Minseong Kim, Sung Ju Hwang, and Jinwoo Shin. “Consistency Regularization for Adversarial Robustness”. In: ArXiv (2021). find
  180. Nassim Taleb. “Antifragile: Things That Gain from Disorder”. In: 2012. find
  181. Nassim Taleb. “Statistical Consequences of Fat Tails: Real World Preasymptotics, Epistemology, and Applications”. In: 2020. find
  182. Nassim Taleb. “The Black Swan: The Impact of the Highly Improbable”. In: 2007. find
  183. Nassim Taleb and Philip Tetlock. “On the Difference between Binary Prediction and True Exposure with Implications for Forecasting Tournaments and Decision Making Research”. In: 2013. find
  184. Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. “Alignment for Advanced Machine Learning Systems”. In: 2016. find
  185. Tesla. Tesla AI Day. 2021. URL : https://www.youtube.com/watch?v=j0z4FweCy4M. ↗
  186. Philip Tetlock and Dan Gardner. “Superforecasting: The Art and Science of Prediction”. In: 2015. find
  187. Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. “Conservative Objective Models for Effective Offline Model-Based Optimization”. In: ICML. 2021. find
  188. Florian Tramèr, Nicholas Carlini, Wieland Brendel, and A. Madry. “On Adaptive Attacks to Adversarial Example Defenses”. In: ArXiv (2020). find
  189. Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Dan Boneh, and Patrick Mcdaniel. “Ensemble Adversarial Training: Attacks and Defenses”. In: ArXiv abs/1705.07204 (2018). — [arXiv:1705.07204](https://arxiv.org/abs/1705.07204) find
  190. A. M. Turner, Neale Ratzlaff, and Prasad Tadepalli. “Avoiding Side Effects in Complex Environments”. In: ArXiv abs/2006.06547 (2020). — [arXiv:2006.06547](https://arxiv.org/abs/2006.06547) find
  191. Alexander Matt Turner, Logan Riggs Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. “Optimal Policies Tend To Seek Power”. In: NeurIPS. 2021. find
  192. Building Seismic Safety Council US et al. “Planning for seismic rehabilitation: societal issues”. In: (1998). find
  193. Carroll L. Wainwright and P. Eckersley. “SafeLife 1.0: Exploring Side Effects in Complex Environments”. In: ArXiv abs/1912.01217 (2020). — [arXiv:1912.01217](https://arxiv.org/abs/1912.01217) find
  194. Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. “Concealed Data Poisoning Attacks on NLP Models”. In: NAACL. 2021. find
  195. Dequan Wang, An Ju, Evan Shelhamer, David A. Wagner, and Trevor Darrell. “Fighting Gradients with Gradients: Dynamic Defenses against Adversarial Attacks”. In: ArXiv abs/2105.08714 (2021). find
  196. Dequan Wang, Evan Shelhamer, Shaoteng Liu, B. Olshausen, and Trevor Darrell. “Tent: Fully Test-Time Adaptation by Entropy Minimization”. In: ICLR. 2021. find
  197. Pei Wang, Yijun Li, Krishna Kumar Singh, Jingwan Lu, and N. Vasconcelos. “IMAGINE: Image Synthesis by Image-Guided Model Inversion”. In: ArXiv abs/2104.05895 (2021). — [arXiv:2104.05895](https://arxiv.org/abs/2104.05895) find
  198. Yue Wang, Esha Sarkar, Wenqing Li, M. Maniatakos, and S. E. Jabari. “Stop-and-Go: Exploring Backdoor Attacks on Deep Reinforcement Learning-based Traffic Congestion Control Systems”. In: arXiv: Cryptography and Security (2020). find
  199. E. G. Williams. “The Possibility of an Ongoing Moral Catastrophe”. In: Ethical Theory and Moral Practice (2015). find
  200. Timothy Wilson and Daniel Gilbert. “Affective Forecasting”. In: Current Directions in Psychological Science (2005). find
  201. Dongxian Wu, Shutao Xia, and Yisen Wang. “Adversarial Weight Perturbation Helps Robust Generalization”. In: NeurIPS (2020). find
  202. Chaowei Xiao, Ruizhi Deng, Bo Li, F. Yu, M. Liu, and D. Song. “Characterizing Adversarial Examples Based on Spatial Consistency Information for Semantic Segmentation”. In: ECCV. 2018. find
  203. Cihang Xie, Mingxing Tan, Boqing Gong, A. Yuille, and Quoc V. Le. “Smooth Adversarial Training”. In: ArXiv abs/2006.14536 (2020). find
  204. Hongxu Yin, Pavlo Molchanov, Zhizhong Li, J. Álvarez, Arun Mallya, Derek Hoiem, N. Jha, and J. Kautz. “Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion”. In: CVPR (2020). find
  205. Sheheryar Zaidi, Arber Zela, T. Elsken, Chris C. Holmes, F. Hutter, and Y. Teh. “Neural Ensemble Search for Uncertainty Estimation and Dataset Shift”. In: 2020. find
  206. Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael I. Jordan. “Theoretically Principled Trade-off between Robustness and Accuracy”. In: ICML. 2019. find
  207. Xinyang Zhang, Zheng Zhang, and Tianying Wang. “Trojaning Language Models for Fun and Profit”. In: ArXiv abs/2008.00312 (2020). — [arXiv:2008.00312](https://arxiv.org/abs/2008.00312) find
  208. Yao Zhu, Jiacheng Ma, Jiacheng Sun, Zewei Chen, Rongxin Jiang, and Zhenguo Li. “Towards Understanding the Generative Capability of Adversarially Robust Classifiers”. In: ArXiv (2021). find
  209. Shoshana Zuboff. “The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power”. In: (2019). find
  210. Remco Zwetsloot, Helen Toner, and Jeffrey Ding. “Beyond the AI arms race: America, China, and the dangers of zero-sum thinking”. In: Foreign Affairs (2018). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hendrycks-2021-unsolved-problems-ml-safety/