Elamur Research⌕
☰
home / papers / 2024 / RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts

RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts

Paper · 2024 · Hjalmar Wijk, Tao Lin, Joel Becker et al.

Summary

AI developers and governments want early-warning tests that show whether AI agents could take over frontier AI research and development, but few realistic evaluations exist, and almost none compare agents directly with people. METR addresses this gap with RE-Bench (V1), seven hand-built, open-ended ML research engineering environments. Each gives the participant a starting solution, a scoring function and 1–6 H100 GPUs. Humans and AI agents work under identical conditions. The human baseline comes from 61 ML experts who made 71 eight-hour attempts. Agents built on o1-preview and Claude 3.5 Sonnet were run in two scaffolds and compared using best-of-k sampling across different time budgets.

With short budgets, agents come out ahead. With longer ones, humans pull away, because experts benefit more from extra time. Agents iterate far faster and more cheaply, yet most of their runs make no real improvement. The benchmark therefore offers a calibrated, human-anchored way to track how close agents are to automating AI R&D, which matters for safety policy.

Key points

  • Human baseline is solid: 82% of expert attempts scored above zero, and 24% equalled or beat the strong reference solutions.
  • Results by time budget:
    • At 2 hours per environment, the best agents scored about 4× higher than humans.
    • At 8 hours, humans were narrowly ahead.
    • At 32 total hours, humans reached roughly double the top agent's score.
  • Agent strengths: agents produced new solutions over 10× faster than experts at much lower token cost. Both models found kernel-optimization solutions (including a custom Triton kernel) that beat all 9 human experts.
  • Limitations noted by the authors:
    • The environments have clearer goals and shorter feedback loops than real research.
    • Agent scores are noisy and heavily right-skewed.
    • Visible test scores may allow overfitting, notably in "Finetune GPT-2 for QA".
    • Results in "Scaling Law Experiment" depend partly on luck.
  • Released resources: environments at github.com/METR/ai-rd-tasks and agent trajectories at transcripts.metr.org. Analysis code and anonymized human data were announced as coming soon.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksSocial Sciences · 2,009 worksSafety Research · 665 works

Cite

Cite as Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Holden Karnofsky, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Sato, William Saunders, Maksym Taran, Ben West and Elizabeth Barnes. 2024. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv:2411.15114.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2411.15114 · arXiv 2411.15114

@misc{wijk2024re,
  title = {{RE-Bench: Evaluating frontier AI R\&D capabilities of language model agents against human experts}},
  author = {Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Holden Karnofsky, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Sato, William Saunders, Maksym Taran, Ben West and Elizabeth Barnes},
  year = {2024},
  doi = {10.48550/arxiv.2411.15114},
  eprint = {2411.15114},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.15114}
}

References

58 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chen, M. et al. Evaluating large language models trained on code. ArXiv abs/2107.03374 (2021). URL https://api.semanticscholar.org/CorpusID:235755472. ↗
  2. Li, Y. et al. Competition-level code generation with AlphaCode. Science 378, 1092–1097 (2022). URL https://www.science.org/doi/abs/10.1126/science.abq1158. https://www.science.org/doi/pdf/10.1126/science.abq1158. ↗
  3. Gunasekar, S. et al. Textbooks are all you need (2023). URL https://arxiv.org/abs/2306.11644. 2306.11644. ↗
  4. AI at Meta. The Llama 3 herd of models (2024). URL https://arxiv.org/abs/2407.21783.2407.21783. ↗
  5. Li, X. et al. Self-alignment with instruction backtranslation. In The Twelfth International Conference on Learning Representations (2024). URL https://openreview.net/forum?id=1oijHJBRsT. ↗
  6. Wang, X. et al. OpenDevin: An open platform for AI software developers as generalist agents (2024). URL https://arxiv.org/abs/2407.16741. 2407.16741. ↗
  7. Owen, D. Interviewing AI researchers on automation of AI r&d (2024). URL https://epochai.org/blog/interviewing-ai-researchers-on-automation-of-ai-rnd. Accessed: 2024-09-27. ↗
  8. Erdil, E., Besiroglu, T. & Ho, A. Estimating idea production: A methodological survey. SSRN Electronic Journal (2024). Available at SSRN: https://ssrn.com/abstract=4814445 or http://dx.doi.org/10.2139/ssrn.4814445. — [doi:10.2139/ssrn.4814445](https://doi.org/10.2139/ssrn.4814445) ↗
  9. Erdil, E. & Besiroglu, T. Explosive growth from AI automation: A review of the arguments (2024). URL https://arxiv.org/abs/2309.11690. 2309.11690. — [arXiv:2309.11690](https://arxiv.org/abs/2309.11690) ↗
  10. OpenAI. OpenAI preparedness framework (beta). Tech. Rep., OpenAI (2023). URL https://cdn.openai.com/openai-preparedness-framework-beta.pdf. ↗
  11. Google DeepMind. Frontier safety framework, version 1.0. Tech. Rep., Google DeepMind (2024). URL https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/introducing-the-frontier-safety-framework/fsf-technical-report.pdf. ↗
  12. Anthropic. Anthropic responsible scaling policy. Policy, Anthropic (2024). https://assets.anthropic.com/m/24a47b00f10301cd/original/Anthropic-Responsible-Scaling-Policy-2024-10-15.pdf. ↗
  13. Union, E. Recital 110 of the eu artificial intelligence act (2024). artificialintelligenceact.eu/recital/110/. find
  14. The White House. Memorandum on advancing the United States’ leadership in artificial intelligence; harnessing artificial intelligence to fulfill national security objectives; and fostering the safety, security, and trustworthiness of artificial intelligence (2024). URL https://www.whitehouse.gov/bri… ↗
  15. UK Government and International Partners. The bletchley declaration on AI safety (2023). URL https://www.gov.uk/government/publications/ai-safety-summit-2023-the-bletchley-declaration. Accessed: November 12, 2024. ↗
  16. United Kingdom Department for Science, Innovation & Technology. Frontier AI safety commitments, AI seoul summit 2024. Policy paper, United Kingdom Department for Science, Innovation & Technology (2024). URL https://www.gov.uk/government/publications/frontier-ai-safety-commitments-ai-seoul-summit-20… ↗
  17. OpenAI. Building an early warning system for llm-aided biological threat creation (2024). URL https://openai.com/research/building-an-early-warning-system-for-llm-aided-biological-threat-creation. Accessed: November 12, 2024. ↗
  18. Wan, S. et al. CYBERSECEVAL 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models (2024). URL https://arxiv.org/abs/2408.01605.2408.01605. — [arXiv:2408.01605](https://arxiv.org/abs/2408.01605) ↗
  19. Davidson, T. What a compute-centric framework says about takeoff speeds (2023). URL https://www.openphilanthropy.org/research/what-a-compute-centric-framework-says-about-takeoff-speeds/. Accessed: October 17, 2024. ↗
  20. Benton, J. et al. Sabotage evaluations for frontier models (2024). URL https://arxiv.org/abs/2410.21514. 2410.21514. URL URL https:// ↗
  21. Nevo, S. et al. Securing AI model weights: Preventing theft and misuse of frontier models. Tech. Rep. RR-A2849-1, RAND Corporation (2024). URL https://www.rand.org/pubs/research_reports/RRA2849-1.html. ↗
  22. Jimenez, C. E. et al. SWE-bench: Can language models resolve real-world github issues? In ICLR (2023). URL https://github.com/princeton-nlp/SWE-bench. ↗
  23. Anthropic. Raising the bar on swe-bench verified with claude 3.5 sonnet (2024). URL https://www.anthropic.com/research/swe-bench-sonnet. Accessed: November 12, 2024. ↗
  24. Maslej, N. et al. Artificial intelligence index report 2024 (2024). URL https://arxiv.org/abs/2405.19522. 2405.19522. — [arXiv:2405.19522](https://arxiv.org/abs/2405.19522) ↗
  25. Chan, J. S. et al. MLE-bench: Evaluating machine learning agents on machine learning engineering (2024). URL https://arxiv.org/abs/2410.07095. 2410.07095. ↗
  26. Tian, M. et al. Scicode: A research coding benchmark curated by scientists (2024). URL https://arxiv.org/abs/2407.13168. 2407.13168. ↗
  27. Rein, D. et al. GPQA: A graduate-level google-proof q&a benchmark. ArXiv abs/2311.12022 (2023). URL https://api.semanticscholar.org/CorpusID:265295009. ↗
  28. Mialon, G. et al. GAIA: a benchmark for general AI assistants (2023). URL https://arxiv.org/abs/2311.12983. 2311.12983. ↗
  29. Chowdhury, N. et al. SWE-bench verified: Human-validated subset of swe-bench (2024). URL https://openai.com/index/introducing-swe-bench-verified/. 500 human-verified samples for evaluating AI models on GitHub issues. ↗
  30. Zhou, S. et al. Webarena: A realistic web environment for building autonomous agents (2024). URL https://arxiv.org/abs/2307.13854. 2307.13854. ↗
  31. Huang, Q., Vora, J., Liang, P. & Leskovec, J. MLAgentBench: Evaluating language agents on machine learning experimentation (2024). URL https://arxiv.org/abs/2310.03302.2310.03302. ↗
  32. Jansen, P. et al. DISCOVERYWORLD: A virtual environment for developing and evaluating automated scientific discovery agents (2024). URL https://arxiv.org/abs/2406.06769.2406.06769. ↗
  33. LeGris, S., Vong, W. K., Lake, B. M. & Gureckis, T. M. H-arc: A robust estimate of human performance on the abstraction and reasoning corpus benchmark (2024). URL https://arxiv.org/abs/2409.01374. 2409.01374. ↗
  34. Ouyang, L. et al. Training language models to follow instructions with human feedback (2022). URL https://arxiv.org/abs/2203.02155. 2203.02155. ↗
  35. Bai, Y. et al. Constitutional AI: Harmlessness from AI feedback (2022). URL https://arxiv.org/abs/2212.08073. 2212.08073. ↗
  36. Adler, B. et al. Nemotron-4 340b technical report (2024). URL https://arxiv.org/abs/2406.11704. 2406.11704. ↗
  37. Li, R., Patel, T., Wang, Q. & Du, X. MLR-copilot: Autonomous machine learning research based on large language models agents (2024). URL https://arxiv.org/abs/2408.14033.2408.14033. ↗
  38. Chen, A., Dohan, D. M. & So, D. R. Evoprompting: Language models for code-level neural architecture search (2023). URL https://arxiv.org/abs/2302.14838. 2302.14838. — [arXiv:2302.14838](https://arxiv.org/abs/2302.14838) ↗
  39. Guo, S. et al. DS-agent: Automated data science by empowering large language models with case-based reasoning (2024). URL https://arxiv.org/abs/2402.17453. 2402.17453. ↗
  40. Bogin, B. et al. SUPER: Evaluating agents on setting up and executing tasks from research repositories (2024). URL https://arxiv.org/abs/2409.07440. 2409.07440. ↗
  41. Lu, C. et al. The AI scientist: Towards fully automated open-ended scientific discovery (2024). URL https://arxiv.org/abs/2408.06292. 2408.06292. ↗
  42. Ma, Y. J. et al. Eureka: Human-level reward design via coding large language models (2024). URL https://arxiv.org/abs/2310.12931. 2310.12931. ↗
  43. Faldor, M., Zhang, J., Cully, A. & Clune, J. OMNI-EPIC: Open-endedness via models of human notions of interestingness with environments programmed in code (2024). URL https://arxiv.org/abs/2405.15568. 2405.15568. ↗
  44. Lu, C. et al. Discovering preference optimization algorithms with and for large language models (2024). URL https://arxiv.org/abs/2406.08414. 2406.08414. ↗
  45. Ma, Y. et al. Sciagent: Tool-augmented language models for scientific reasoning (2024). URL https://arxiv.org/abs/2402.11451. 2402.11451. ↗
  46. Bran, A. M. et al. Chemcrow: Augmenting large-language models with chemistry tools (2023). URL https://arxiv.org/abs/2304.05376. 2304.05376. ↗
  47. Zhang, Q. et al. Scientific large language models: A survey on biological & chemical domains (2024). URL https://arxiv.org/abs/2401.14656. 2401.14656. ↗
  48. Tornede, A. et al. AutoML in the age of large language models: Current challenges, future opportunities and risks (2024). URL https://arxiv.org/abs/2306.08107. 2306.08107. ↗
  49. Mirhoseini, A. et al. Chip placement with deep reinforcement learning (2020). URL https://arxiv.org/abs/2004.10746. 2004.10746. ↗
  50. Hobbhahn, M., Heim, L. & Aydos, G. Trends in machine learning hardware (2023). URL https://epochai.org/blog/trends-in-machine-learning-hardware. Accessed: 2024-09-27. ↗
  51. METR. Vivaria: Open-source platform for agent evaluations (2024). URL https://vivaria.metr.org/. Accessed: October 17, 2024. Claude 3.5 Sonnet model card addendum (2024). URL https: ↗
  52. OpenAI. OpenAI o1 system card (2024). URL https://openai.com/index/openai-o1-system-card/. Accessed: October 17, 2024. ↗
  53. weco.ai. AIDE: Data science automation technical report (2024). URL https://www.weco.ai/blog/technical-report. Accessed: October 17, 2024. ↗
  54. METR. Details about METR’s preliminary evaluation of OpenAI o1-preview (2024). URL https://metr.github.io/autonomy-evals-guide/openai-o1-preview-report/. Accessed: October 18, 2024. ↗
  55. Analysis, A. Claude 3.5 Sonnet: Quality, performance & price analysis (2024). URL https://artificialanalysis.ai/models/claude-35-sonnet. Accessed: October 17, 2024. ↗
  56. Hoffmann, J. et al. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22 (Curran Associates Inc., Red Hook, NY, USA, 2024).
  57. Rafailov, R. et al. Direct preference optimization: Your language model is secretly a reward model (2024). URL https://arxiv.org/abs/2305.18290. 2305.18290. ↗
  58. King, D. et al. mosaicml/llm-foundry (2023). URL https://github.com/mosaicml/llm-foundry. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wijk-2024-re-bench-evaluating-frontier/