Elamur Research⌕
☰
home / papers / 2023 / Model evaluation for extreme risks

Model evaluation for extreme risks

Paper · 2023 · Toby Shevlane, Sebastian Farquhar, Ben Garfinkel et al.

Summary

This position paper, from authors at Google DeepMind, OpenAI, Anthropic, GovAI and academic institutions, argues that general-purpose AI models may develop harmful abilities their creators never targeted. Examples include cyber-offense and manipulation, which could lead to harm on a catastrophic scale. Rather than introducing a dataset or benchmark, the authors propose a conceptual framework that expands evaluation practice in two directions. First, dangerous capability evaluations test what a model is able to do. Second, alignment evaluations test whether the model tends to use those abilities harmfully.

The paper explains how evaluation results should inform four governance areas: decisions about training, decisions about deployment, transparency toward stakeholders, and information security. It lists candidate dangerous capabilities and sets out design criteria for evaluations. It also acknowledges limitations, such as models being trained to pass the tests superficially and harms caused by the testing itself. The main contribution is a theory of change that places such evaluations inside company policy and regulation. This matters because it shaped how frontier labs and governments later approached pre-deployment safety testing.

Key points

  • Scope of "extreme" risk: The authors define it as harm on the order of tens of thousands of deaths or hundreds of billions of dollars in damage, or as severe disruption of social and political order. They cite a 2022 survey in which 36% of AI researchers considered a catastrophe comparable to nuclear war plausible this century.
  • Dangerous capabilities (Table 1): The non-exhaustive list includes cyber-offense, deception, persuasion and manipulation, political strategy, weapons acquisition, long-horizon planning, AI development and situational awareness.
  • Alignment evaluation targets: These include pursuing unintended long-term goals, power-seeking, resisting shutdown, colluding with other AI systems, and how well the model resists misuse by malicious users.
  • Unintended harms of evaluation: One risk is overfitting to evaluations, so the authors suggest keeping some held-out tests private. Another is harm during testing itself, illustrated by ARC's GPT-4 test, in which the model produced deceptive messages for a TaskRabbit worker.
  • Recommendations: Developers should invest in evaluation research, set internal policies, support outside researchers and educate policymakers. Policymakers should track capabilities, fund external evaluation, mandate audits and block deployment of models that pose extreme risks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom256 works · 123,373 citations🎓University of TorontoToronto · Canada110 works · 179,061 citations🎓University of OxfordOxford · United Kingdom122 works · 160,277 citations🎓Université de MontréalMontreal · Canada87 works · 91,597 citations🏢OpenAISan Francisco · United States23 works · 27,874 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada65 works · 14,425 citations🏢AnthropicSan Francisco · United States8 works · 1,908 citations🤝Centre for the Governance of AIOxford · United Kingdom2 works · 302 citations🔬American Anthropological AssociationArlington · United States2 works · 225 citations🏛️Anthropological Survey of IndiaKolkata · India2 works · 225 citations🔬Institute for Anthropological ResearchZagreb · Croatia2 works · 225 citations🔬Laboratoire d'Anthropologie SocialeParis · France2 works · 225 citations

Also on the byline: The Collective Intelligence Project.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSafety Research · 626 worksEngineering · 188 worksSafety, Risk, Reliability and Quality · 14 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul Christiano and Allan Dafoe. 2023. Model evaluation for extreme risks. arXiv:2305.15324.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.15324 · arXiv 2305.15324

@misc{shevlane2023model,
  title = {{Model evaluation for extreme risks}},
  author = {Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul Christiano and Allan Dafoe},
  year = {2023},
  doi = {10.48550/arxiv.2305.15324},
  eprint = {2305.15324},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.15324}
}

References

48 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Strengthening U.S. AI innovation through an ambitious investment in NIST. Technical report, 2023. find
  2. ARC Evals. Update on ARC’s recent eval efforts. https://evals.alignment.org/blog/ 2023-03-18-update-on-recent-evals/ , Mar. 2023. Accessed: 2023-3-20. ↗
  3. E. Bluemke, T. Collins, B. Garfinkel, and A. Trask. Exploring the relevance of data Privacy-Enhancing technologies for AI governance use cases. Mar. 2023. find
  4. M. Brundage, S. Avin, J. Clark, H. Toner, P. Eckersley, B. Garfinkel, A. Dafoe, P. Scharre, T. Zeitzoff, B. Filar, H. Anderson, H. Roff, G. C. Allen, J. Steinhardt, C. Flynn, S. Ó. hÉigeartaigh, S. Beard, H. Belfield, S. Farquhar, C. Lyle, R. Crootof, O. Evans, M. Page, J. Bryson, R. Yampolskiy, and… find
  5. M. Brundage, S. Avin, J. Wang, H. Belfield, G. Krueger, G. Hadfield, H. Khlaaf, J. Yang, H. Toner, R. Fong, T. Maharaj, P. W. Koh, S. Hooker, J. Leung, A. Trask, E. Bluemke, J. Lebensold, C. O’Keefe, M. Koren, T. Ryffel, J. B. Rubinovitz, T. Besiroglu, F. Carugati, J. Clark, P. Eckersley, S. de Haas… find
  6. M. Brundage, K. Mayer, T. Eloundou, S. Agarwal, S. Adler, G. Krueger, J. Leike, and P. Mishkin. Lessons learned on language model safety and misuse. https://openai.com/research/ language-model-safety-and-misuse , Mar. 2022. Accessed: 2023-4-17. ↗
  7. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. Mar. 2023.
  8. C. Burns, H. Ye, D. Klein, and J. Steinhardt. Discovering latent knowledge in language models without supervision. Dec. 2022. find
  9. J. Carlsmith. Is Power-Seeking AI an existential risk? June 2022. find
  10. A. Chan, R. Salganik, A. Markelius, C. Pang, N. Rajkumar, D. Krasheninnikov, L. Langosco, Z. He, Y. Duan, M. Carroll, M. Lin, A. Mayhew, K. Collins, M. Molamohammadi, J. Burden, W. Zhao, S. Rismani, K. Voudouris, U. Bhatt, A. Weller, D. Krueger, and T. Maharaj. Harms from increasingly agentic algori… find
  11. N. Emery-Xu, A. Park, and R. Trager. Uncertainty, information, and risk in international technology races. Unpublished manuscript, 2023. find
  12. T. Everitt, R. Carey, E. Langlois, P. A. Ortega, and S. Legg. Agent incentives: A causal perspective. Feb. 2021. find
  13. D. Ganguli, D. Hernandez, L. Lovitt, N. DasSarma, T. Henighan, A. Jones, N. Joseph, J. Kernion, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, N. Elhage, S. El Showk, S. Fort, Z. Hatfield-Dodds, S. Johnston, S. Kravec, N. Nanda, K. Ndousse, C. Olsson, D. Amodei, D. Amodei, T. Brown, J. K… — Predictability and Surprise in Large Generative Models
  14. L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. Oct. 2022.
  15. A. Glaese, N. McAleese, M. Trębacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, L. Campbell-Gillingham, J. Uesato, P.-S. Huang, R. Comanescu, F. Yang, A. See, S. Dathathri, R. Greig, C. Chen, D. Fritz, J. S. Elias, R. Green, S. Mokrá, N. Fernando, B. Wu, R. Fo… — Improving alignment of dialogue agents via targeted human judgements
  16. D. Hadfield-Menell, A. Dragan, P. Abbeel, and S. Russell. The Off-Switch game. Nov. 2016. find
  17. E. Jones, A. Dragan, A. Raghunathan, and J. Steinhardt. Automatically auditing large language models via discrete optimization. Mar. 2023.
  18. Z. Kenton, R. Kumar, S. Farquhar, J. Richens, M. MacDermott, and T. Everitt. Discovering agents. Aug. 2022. find
  19. H. Khlaaf, P. Mishkin, J. Achiam, G. Krueger, and M. Brundage. A hazard analysis framework for code synthesis large language models. July 2022. find
  20. V. Krakovna and J. Kramar. Power-seeking can be probable and predictive for trained agents. Apr. 2023. find
  21. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. Or… find
  22. I. McKenzie, A. Lyzhov, A. Parrish, A. Prabhu, A. Mueller, N. Kim, S. Bowman, and E. Perez. Inverse scaling prize. https://github.com/inverse-scaling/prize , 2022. Accessed: 2023-4-7. ↗
  23. J. Michael, A. Holtzman, A. Parrish, A. Mueller, A. Wang, A. Chen, D. Madaan, N. Nangia, R. Y. Pang, J. Phang, and S. R. Bowman. What do NLP researchers believe? results of the NLP community metasurvey. Aug. 2022. find
  24. M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru. Model cards for model reporting. Oct. 2018. find
  25. J. Mökander, J. Schuett, H. R. Kirk, and L. Floridi. Auditing large language models: a three-layered approach. Feb. 2023. find
  26. N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability. Jan. 2023.
  27. R. Ngo, L. Chan, and S. Mindermann. The alignment problem from a deep learning perspective. Aug. 2022.
  28. C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. Zoom in: An introduction to circuits. Distill, 5(3), Mar. 2020. find
  29. OpenAI. GPT-4 system card. Mar. 2023a. find
  30. OpenAI. GPT-4 technical report. Mar. 2023b.
  31. T. Ord. Lessons from the development of the atomic bomb. Technical report, Centre for the Governance of AI, Nov. 2022. find
  32. L. Orseau and S. Armstrong. Safely interruptible agents. https://www.deepmind.com/ publications/safely-interruptible-agents , 2016. Accessed: 2023-4-26. ↗
  33. A. Pan, C. J. Shern, A. Zou, N. Li, S. Basart, T. Woodside, J. Ng, H. Zhang, S. Emmons, and D. Hendrycks. Do the rewards justify the means? measuring Trade-Offs between rewards and ethical behavior in the MACHIAVELLI benchmark. Apr. 2023. find
  34. E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. Feb. 2022a.
  35. E. Perez, S. Ringer, K. Lukošiūtė, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, A. Jones, A. Chen, B. Mann, B. Israel, B. Seethor, C. McKinnon, C. Olah, D. Yan, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, G. Khundadze, J. Kernion, J. Landis, J. Kerr, J. M… — Discovering Language Model Behaviors with Model-Written Evaluations
  36. I. D. Raji, A. Smart, R. N. White, M. Mitchell, T. Gebru, B. Hutchinson, J. Smith-Loud, D. Theron, and P. Barnes. Closing the AI accountability gap: Defining an End-to-End framework for internal algorithmic auditing. Jan. 2020. find
  37. I. D. Raji, I. Elizabeth Kumar, A. Horowitz, and A. D. Selbst. The fallacy of AI functionality. June 2022a. find
  38. I. D. Raji, P. Xu, C. Honigsberg, and D. E. Ho. Outsider oversight: Designing a third party audit ecosystem for AI governance. June 2022b. find
  39. S. Rismani, R. Shelby, A. Smart, E. Jatho, J. Kroll, A. Moon, and N. Rostamzadeh. From plane crashes to algorithmic harm: Applicability of safety engineering frameworks for responsible ML. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, number Article 2 in CHI ’23, p… find
  40. R. Shah, V. Varma, R. Kumar, M. Phuong, V. Krakovna, J. Uesato, and Z. Kenton. Goal misgeneralization: Why correct specifications aren’t enough for correct goals. Oct. 2022. find
  41. T. Shevlane. Sharing powerful AI models. https://www.governance.ai/post/ sharing-powerful-ai-models , Jan. 2022a. Accessed: 2023-3-18. ↗
  42. T. Shevlane. Structured access: An emerging paradigm for safe AI deployment. Feb. 2022b. find
  43. A. Turner, L. Smith, R. Shah, A. Critch, and P. Tadepalli. Optimal policies tend to seek power. Adv. Neural Inf. Process. Syst., 34:23063–23074, Dec. 2021. find
  44. J. Wei, N. Kim, Y. Tay, and Q. V. Le. Inverse scaling can become u-shaped. Nov. 2022a. find
  45. J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus. Emergent abilities of large language models. June 2022b.
  46. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-Thought prompting elicits reasoning in large language models. Jan. 2022c.
  47. J. Whittlestone and J. Clark. Why and how governments should monitor AI development. Aug. 2021. find
  48. D. M. Ziegler, S. Nix, L. Chan, T. Bauman, P. Schmidt-Nielsen, T. Lin, A. Scherlis, N. Nabeshima, B. Weinstein-Raun, D. de Haas, B. Shlegeris, and N. Thomas. Adversarial training for High-Stakes reliability. May 2022. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shevlane-2023-model-evaluation-extreme-risks/