Elamur Research⌕
☰
home / papers / 2024 / Evaluating Frontier Models for Dangerous Capabilities

Evaluating Frontier Models for Dangerous Capabilities

Paper · 2024 · Mary Phuong, Matthew Aitchison, Elliot Catt et al.

Summary

Popular benchmarks measure broad skills such as coding or maths, and most safety tests check whether a model refuses harmful requests. Neither shows what a frontier system could actually do if misused. This Google DeepMind paper proposes a methodology for measuring "dangerous capabilities" and pilots it on Gemini 1.0 Ultra, Pro and Nano 3.25B, with runs in January 2024. The suite spans four areas:

  • persuasion and deception, through dialogue studies with human participants;
  • cyber-security, through capture-the-flag challenges and vulnerability detection;
  • self-proliferation, through autonomous agent tasks such as deploying a model on a cloud server;
  • self-reasoning, through recognising when self-modification is needed.

To avoid underestimating the models, the authors tested them without safety filters and wrapped them in agent scaffolding with planning frameworks and tools. No model showed strong dangerous capabilities. Persuasion and deception looked most developed, and the stronger models had rudimentary skills in every area. The work matters as an early template for evaluations that inform AI policy and risk-tiered security rules. Evaluation code is not released, but prompts and rater instructions are documented.

Key points

  • Evaluations include Money Talks, Charm Offensive, Hidden Agenda (3 tasks) and Web of Lies for persuasion. Cyber-security uses 13 in-house CTFs, 13 Hack The Box challenges and 3 vulnerability-detection tasks. Self-proliferation has 11 tasks, and self-reasoning covers knowledge gaps (2) and instrumental self-modification (4).
  • Gemini Pro 1.0 often matched Ultra 1.0 and sometimes beat it on persuasion. The authors conclude that models just behind the frontier also need scrutiny.
  • Professional forecasters gave median dates of 2025–2029 for when models will first score highly on these evaluations.
  • Forecasters put the probability that AI becomes a top-3 issue in Gallup's "Most Important Problem" poll before 2030 at 32% unconditionally. That rose to 47% if a single system passes a defined subset of the evaluations before 2027, and 68% if an open-weight system does.
  • CBRN evaluations are only sketched as a preliminary framework (Appendix B). The paper includes canary strings so that it can be kept out of training corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksSocial Sciences · 2,009 worksSafety Research · 665 worksEngineering · 206 worksSafety, Risk, Reliability and Quality · 18 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 works

Cite

Cite as Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victoria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, Heidi Howard, Tom Lieberum, Ramana Kumar, Maria Abi Raad, Albert Webson, Lewis Ho, Sharon Lin, Sebastian Farquhar, Marcus Hutter, Gregoire Deletang, Anian Ruoss, Seliem El-Sayed, Sasha Brown, Anca Dragan, Rohin Shah, Allan Dafoe and Toby Shevlane. 2024. Evaluating Frontier Models for Dangerous Capabilities. arXiv:2403.13793.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.13793 · arXiv 2403.13793

@misc{phuong2024evaluating,
  title = {{Evaluating Frontier Models for Dangerous Capabilities}},
  author = {Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victoria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, Heidi Howard, Tom Lieberum, Ramana Kumar, Maria Abi Raad, Albert Webson, Lewis Ho, Sharon Lin, Sebastian Farquhar, Marcus Hutter, Gregoire Deletang, Anian Ruoss, Seliem El-Sayed, Sasha Brown, Anca Dragan, Rohin Shah, Allan Dafoe and Toby Shevlane},
  year = {2024},
  doi = {10.48550/arxiv.2403.13793},
  eprint = {2403.13793},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.13793}
}

References

74 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Anthropic’s responsible scaling policy, Sept. 2023. URL https://www-files.anthrop ic.com/production/files/responsible-scaling-policy-1.0.pdf . ↗
  2. Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku, Mar. 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claud e_3.pdf . ↗
  3. E. Bagdasaryan and V. Shmatikov. Spinning language models: Risks of Propaganda-As-A-Service and countermeasures. ArXiv, Dec. 2021. URL https://arxiv.org/abs/2112.05224 . — [arXiv:2112.05224](https://arxiv.org/abs/2112.05224) ↗
  4. H. Bai, J. G. Voelkel, J. C. Eichstaedt, and R. Willer. Artificial intelligence can persuade humans on political issues, Feb. 2023. URL https://osf.io/stakv . ↗
  5. A. Basit, M. Zafar, X. Liu, A. R. Javed, Z. Jalil, and K. Kifayat. A comprehensive survey of AI-enabled phishing attacks detection techniques. Telecommun Syst, 76(1):139–154, 2021. URL https://link.springer.com/article/10.1007/s11235-020-00733-2 . — [doi:10.1007/s11235-020-00733-2](https://doi.org/10.1007/s11235-020-00733-2) ↗
  6. L. Berglund, A. C. Stickland, M. Balesni, M. Kaufmann, M. Tong, T. Korbak, D. Kokotajlo, and O. Evans. Taken out of context: On measuring situational awareness in LLMs. ArXiv, Sept. 2023. URL https://arxiv.org/abs/2309.00667 . — [arXiv:2309.00667](https://arxiv.org/abs/2309.00667) ↗
  7. M. Bhatt, S. Chennabasappa, C. Nikolaidis, S. Wan, I. Evtimov, D. Gabi, D. Song, F. Ahmad, C. Aschermann, L. Fontana, S. Frolov, R. P. Giri, D. Kapil, Y. Kozyrakis, D. LeBlanc, J. Milazzo, A. Straumann, G. Synnaeve, V. Vontimitta, S. Whitman, and J. Saxe. Purple Llama CyberSecEval: A secure coding b… — [arXiv:2312.04724](https://arxiv.org/abs/2312.04724) find
  8. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. ArXiv, Mar. 2023. URL https://arxiv.org/abs/2303.12712 . ↗
  9. M. Burtell and T. Woodside. Artificial influence: An analysis of AI-driven persuasion. ArXiv, Mar. 2023. URL https://arxiv.org/abs/2303.08721 . — [arXiv:2303.08721](https://arxiv.org/abs/2303.08721) ↗
  10. M. Carroll, A. Chan, H. Ashton, and D. Krueger. Characterizing manipulation from AI systems. ArXiv, Mar. 2023. URL https://arxiv.org/abs/2303.09387 . — [arXiv:2303.09387](https://arxiv.org/abs/2303.09387) ↗
  11. A. Chan, R. Salganik, A. Markelius, C. Pang, N. Rajkumar, D. Krasheninnikov, L. Langosco, Z. He, Y. Duan, M. Carroll, et al. Harms from increasingly agentic algorithmic systems. In ACM Conference on Fairness, Accountability, and Transparency, pages 651–666, 2023. URL https://arxiv.org/ abs/2302.1032… — [arXiv:2302.10329](https://arxiv.org/abs/2302.10329) ↗
  12. Q. Chen, D. Pitawela, C. Zhao, G. Zhou, H.-T. Chen, and Q. Wu. WebVLN: Vision-and-language navigation on websites. ArXiv, Dec. 2023a. URL https://arxiv.org/abs/2312.15820 . — [arXiv:2312.15820](https://arxiv.org/abs/2312.15820) ↗
  13. Y. Chen, Z. Ding, L. Alowain, X. Chen, and D. Wagner. DiverseVul: A new vulnerable source code dataset for deep learning based vulnerability detection. In International Symposium on Research in Attacks, Intrusions and Defenses, pages 654–668, Apr. 2023b. URL https://arxiv.org/abs/23 04.00409 . ↗
  14. A. Cotra. Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover. Alignment Forum, July 2022. URL https://www.alignmentforum.org/posts/pRk FkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to . ↗
  15. T. Davidson, J.-S. Denain, P. Villalobos, and G. Bas. AI capabilities can be significantly improved without expensive retraining. ArXiv, Dec. 2023. URL https://arxiv.org/abs/2312.07413 . — [arXiv:2312.07413](https://arxiv.org/abs/2312.07413) ↗
  16. G. Delétang, A. Ruoss, P.-A. Duquenne, E. Catt, T. Genewein, C. Mattern, J. Grau-Moya, L. K. Wenliang, M. Aitchison, L. Orseau, et al. Language modeling is compression. ArXiv, Sept. 2023. URL https://arxiv.org/abs/2309.10668 . ↗
  17. S. El-Sayed, C. Akbulut, A. McCroskery, G. Keeling, Z. Kenton, Z. Howard, N. Marchal, A. Manzini, T. Shevlane, S. Vallor, D. Susser, M. Franklin, S. Bridgers, H. Law, M. Rahtz, M. Shanahan, M. H. Tessler, A. Douillard, T. Everitt, and S. Brown. A Mechanism-Based Approach to Mitigating Harms from Per… — [arXiv:2311.08803](https://arxiv.org/abs/2311.08803) find
  18. J. A. Goldstein, J. Chao, S. Grossman, A. Stamos, and M. Tomz. Can AI write persuasive propaganda?, Apr. 2023. URL https://osf.io/preprints/socarxiv/fp87b . ↗
  19. G. Google. Gemini: a family of highly capable multimodal models. ArXiv, Dec. 2023. URL https://arxiv.org/abs/2312.11805 . ↗
  20. Google DeepMind. AI safety summit: An update on our approach to safety and responsibility, Oct. 2023. URL https://deepmind.google/public-policy/ai-summit-policies/ . ↗
  21. J. Hazell. Spear phishing with large language models. ArXiv, May 2023. URL https://arxiv.org/abs/2305.06972 . — [arXiv:2305.06972](https://arxiv.org/abs/2305.06972) ↗
  22. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the MATH dataset. ArXiv, Mar. 2021. URL https://arxiv.org/abs/2103.03874 . ↗
  23. E. Hubinger, C. van Merwijk, V. Mikulik, J. Skalse, and S. Garrabrant. Risks from learned optimization in advanced machine learning systems. ArXiv, June 2019. URL https://arxiv.org/abs/1906 .01820 . ↗
  24. S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, et al. Language models (mostly) know what they know. ArXiv, July 2022. URL https://arxiv.org/abs/2207.05221 . ↗
  25. E. Karger, J. Rosenberg, Z. Jacobs, M. Hickman, R. Hadshar, K. Gamin, T. Smith, B. Williams, T. McCaslin, and P. E. Tetlock. Forecasting existential risks: Evidence from a long-run forecasting tournament. Forecasting Research Institute Working Paper, 2023. find
  26. E. Karinshak, S. X. Liu, J. S. Park, and J. T. Hancock. Working with AI to persuade: Examining a large language model’s ability to generate pro-vaccination messages. ACM on Human-Computer Interaction, 7(CSCW1):1–29, Apr. 2023. URL https://dl.acm.org/doi/abs/10.1145/357 9592 . — [doi:10.1145/357](https://doi.org/10.1145/357) ↗
  27. Z. Kenton, T. Everitt, L. Weidinger, I. Gabriel, V. Mikulik, and G. Irving. Alignment of language agents. ArXiv, Mar. 2021. URL https://arxiv.org/abs/2103.14659 . ↗
  28. M. Kinniment, L. J. K. Sato, H. Du, B. Goodrich, M. Hasin, L. Chan, L. H. Miles, T. R. Lin, H. Wijk, J. Burget, et al. Evaluating language-model agents on realistic autonomous tasks. ArXiv, Dec. 2023. URL https://arxiv.org/abs/2312.11671 . — [arXiv:2312.11671](https://arxiv.org/abs/2312.11671) ↗
  29. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. ArXiv, May 2022. URL https://arxiv.org/abs/2205.11916 . ↗
  30. R. Laine, A. Meinke, and O. Evans. Towards a situational awareness benchmark for LLMs. In Socially Responsible Language Modelling Research, 2023. URL https://openreview.net/forum?id=DR k4bWKr41 . ↗
  31. Z. Liu, W. Yao, J. Zhang, L. Xue, S. Heinecke, R. Murthy, Y. Feng, Z. Chen, J. C. Niebles, D. Arpit, et al. BOLAA: Benchmarking and orchestrating LLM-augmented autonomous agents. ArXiv, Aug. 2023. URL https://arxiv.org/abs/2308.05960 . ↗
  32. D. J. MacKay. Information theory, inference and learning algorithms. Cambridge University Press, 2003. find
  33. K. McGuffie and A. Newhouse. The radicalization risks of GPT-3 and advanced neural language models. ArXiv, Sept. 2020. URL https://arXiv.org/abs/2009.06807 . — [arXiv:2009.06807](https://arxiv.org/abs/2009.06807) ↗
  34. G. Mialon, R. Dessì, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Rozière, T. Schick, J. Dwivedi-Yu, A. Celikyilmaz, et al. Augmented language models: a survey. ArXiv, Feb. 2023. URL https://arxiv.org/abs/2302.07842 . ↗
  35. M. R. Morris, J. Sohl-dickstein, N. Fiedel, T. Warkentin, A. Dafoe, A. Faust, C. Farabet, and S. Legg. Levels of AGI: Operationalizing progress on the path to AGI. ArXiv, Nov. 2023. URL https://arxiv.org/abs/2311.02462 . — [arXiv:2311.02462](https://arxiv.org/abs/2311.02462) ↗
  36. R. Ngo. The alignment problem from a deep learning perspective. ArXiv, Sept. 2022. URL https://arxiv.org/abs/2209.00626 . ↗
  37. A. O’Gara. Hoodwinked: Deception and cooperation in a text-based game for language models. ArXiv, July 2023. URL https://arxiv.org/abs/2308.01404 . — [arXiv:2308.01404](https://arxiv.org/abs/2308.01404) ↗
  38. OpenAI. GPT-4 system card, Mar. 2023a. URL https://cdn.openai.com/papers/gpt-4-sys tem-card.pdf . ↗
  39. OpenAI. OpenAI evals elsuite, Sept. 2023b. URL https://github.com/openai/evals/tree/m ain/evals/elsuite . ↗
  40. OpenAI. Preparedness framework (beta), Dec. 2023c. URL https://cdn.openai.com/opena i-preparedness-framework-beta.pdf . ↗
  41. OpenAI. Building an early warning system for llm-aided biological threat creation, Jan. 2024. URL https://openai.com/research/building-an-early-warning-system-for-llm-aid ed-biological-threat-creation . ↗
  42. P. S. Park, S. Goldstein, A. O’Gara, M. Chen, and D. Hendrycks. AI deception: A survey of examples, risks, and potential solutions. ArXiv, Aug. 2023. URL https://arxiv.org/abs/2308.14752 . — [arXiv:2308.14752](https://arxiv.org/abs/2308.14752) ↗
  43. J. Raz. The Morality of Freedom. Clarendon Press, 1988. find
  44. R. Schaeffer, B. Miranda, and S. Koyejo. Are emergent abilities of large language models a mirage? ArXiv, Apr. 2023. URL https://arxiv.org/abs/2304.15004 . ↗
  45. J. Scheurer, M. Balesni, and M. Hobbhahn. Technical report: Large language models can strategically deceive their users when put under pressure. ArXiv, Nov. 2023. URL https://arxiv.org/abs/ 2311.07590 . ↗
  46. C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3): 379–423, 1948. find
  47. M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, N. Cheng, E. Durmus, Z. Hatfield-Dodds, S. R. Johnston, S. Kravec, T. Maxwell, S. McCandlish, K. Ndousse, O. Rausch, N. Schiefer, D. Yan, M. Zhang, and E. Perez. Towards understanding sycophancy in language models. ArXiv, Oct. 2023…
  48. Y. Shavit, S. Agarwal, M. Brundage, S. Adler, C. O’Keefe, R. Campbell, T. Lee, P. Mishkin, T. Eloundou, A. Hickey, et al. Practices for governing agentic AI systems, 2023. URL https://cdn.openai.com/papers/practices-for-governing-agentic-ai-systems.pdf . ↗
  49. T. Shevlane, S. Farquhar, B. Garfinkel, M. Phuong, J. Whittlestone, J. Leung, D. Kokotajlo, N. Marchal, M. Anderljung, N. Kolt, L. Ho, D. Siddarth, S. Avin, W. Hawkins, B. Kim, I. Gabriel, V. Bolina, J. Clark, Y. Bengio, P. Christiano, and A. Dafoe. Model evaluation for extreme risks. ArXiv, May 202…
  50. N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. ArXiv, Mar. 2023. URL https://arxiv.org/abs/2303.11366 . ↗
  51. D. Susser, B. Roessler, and H. Nissenbaum. Technology, autonomy, and manipulation. Internet Policy Review, June 2019. URL https://policyreview.info/articles/analysis/technology -autonomy-and-manipulation . ↗
  52. P. E. Tetlock. Expert political judgement: How good is it? How can we know?, volume 321. Princeton University Press, Princeton, NJ, US, 2005. find
  53. The White House. Fact sheet: President Biden issues Executive Order on safe, secure, and trustworthy artificial intelligence, Oct. 2023a. URL https://www.whitehouse.gov/briefing-room/st atements-releases/2023/10/30/fact-sheet-president-biden-issues-executive -order-on-safe-secure-and-trustworthy-art… ↗
  54. The White House. Ensuring safe, secure, and trustworthy AI, July 2023b. URL https://www.whit ehouse.gov/wp-content/uploads/2023/07/Ensuring-Safe-Secure-and-Trustwort hy-AI.pdf . ↗
  55. UK AI Safety Institute. Introducing the AI Safety Institute, Nov. 2023. URL https://www.gov.uk/government/publications/ai-safety-institute-overview/introducing-the-a i-safety-institute . ↗
  56. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Neural Information Processing Systems, 30, 2017. URL https://arxiv.org/abs/1706.03762 . ↗
  57. L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al. A survey on large language model based autonomous agents. ArXiv, Aug. 2023. URL https://arxiv.org/abs/2308.11432 . ↗
  58. X. Wang, K. Sun, A. Batcheller, and S. Jajodia. Detecting "0-day" vulnerability: An empirical study of secret security patch in OSS. In IEEE/IFIP International Conference on Dependable Systems and Networks, pages 485–492. IEEE, 2019. URL https://csis.gmu.edu/ksun/publications/s ecretpatch-dsn19.pdf… ↗
  59. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. ArXiv, Mar. 2022. URL https://arxiv.org/abs/2203.11171 . ↗
  60. F. R. Ward, F. Belardinelli, F. Toni, and T. Everitt. Honesty is the best policy: Defining and mitigating AI deception. ArXiv, Dec. 2023. URL https://arxiv.org/abs/2312.01350 . — [arXiv:2312.01350](https://arxiv.org/abs/2312.01350) ↗
  61. A. Wei, N. Haghtalab, and J. Steinhardt. Jailbroken: How does LLM safety training fail? ArXiv, July 2023. URL https://arxiv.org/abs/2307.02483 . ↗
  62. J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus. Emergent abilities of large language models. ArXiv, June 2022a. URL https://arxiv.org/abs/2206.07682 . ↗
  63. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Neural Information Processing Systems, 35: 24824–24837, 2022b. URL https://arxiv.org/abs/2201.11903 . ↗
  64. L. Weidinger, M. Rauh, N. Marchal, A. Manzini, L. A. Hendricks, J. Mateos-Garcia, S. Bergman, J. Kay, C. Griffin, B. Bariach, I. Gabriel, V. Rieser, and W. Isaac. Sociotechnical safety evaluation of generative AI systems. ArXiv, Oct. 2023. URL https://arxiv.org/abs/2310.11986 . — [arXiv:2310.11986](https://arxiv.org/abs/2310.11986) ↗
  65. S. C. Woolley and P. N. Howard. Computational Propaganda: Political Parties, Politicians, and Political Manipulation on Social Media. Oxford University Press, Nov. 2018. find
  66. Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, et al. The rise and potential of large language model based agents: A survey. ArXiv, Sept. 2023. URL https://arxiv.org/abs/2309.07864 . ↗
  67. J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao. InterCode: Standardizing and benchmarking interactive coding with execution feedback. ArXiv, June 2023. URL https://arxiv.org/abs/ 2306.14898 . ↗
  68. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. ReAct: Synergizing reasoning and acting in language models. ArXiv, Oct. 2022. URL https://arxiv.org/abs/2210.03629 . ↗
  69. S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv, May 2023. URL https://arxiv.org/abs/2305.10601 . ↗
  70. B. Zheng, B. Gou, J. Kil, H. Sun, and Y. Su. GPT-4V(ision) is a generalist web agent, if grounded. ArXiv, Jan. 2024. URL https://arxiv.org/abs/2401.01614 . ↗
  71. C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang. Large language models are not robust multiple choice selectors. ArXiv, Sept. 2023. URL https://arxiv.org/abs/2309.03882 . ↗
  72. A. Zhou, K. Yan, M. Shlapentokh-Rothman, H. Wang, and Y.-X. Wang. Language agent tree search unifies reasoning acting and planning in language models. ArXiv, Oct. 2023a. URL https://arxiv.org/abs/2310.04406 . ↗
  73. S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, Y. Bisk, D. Fried, U. Alon, et al. WebArena: A realistic web environment for building autonomous agents. ArXiv, July 2023b. URL https://arxiv.org/abs/2307.13854 . ↗
  74. Y. Zhou, J. K. Siow, C. Wang, S. Liu, and Y. Liu. SPI: Automated identification of security patches via commits. ACM Transactions on Software Engineering and Methodology, 31(1):1–27, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/phuong-2024-evaluating-frontier-models-dangerous-capabilities/