Elamur Research⌕
☰
home / papers / 2025 / PaperBench: Evaluating AI's Ability to Replicate AI Research

PaperBench: Evaluating AI's Ability to Replicate AI Research

Paper · 2025 · Giulio Starace, Oliver Jaffe, Dane Sherburn et al.

Summary

PaperBench measures whether AI agents can independently reproduce recent machine learning research. Agents receive 20 Spotlight and Oral papers from ICML 2024, covering 12 topic areas. For each paper, an agent must write a complete codebase from scratch, without access to the authors' original code, and include a reproduce.sh entry script. That script is then rerun on a clean Ubuntu 24.04 machine with an A10 GPU.

Grading uses hierarchical rubrics built together with the original paper authors. Each rubric is a weighted tree whose leaves are binary checks of three kinds: Code Development, Execution and Result Match. Because human grading of a single attempt can take many hours, the authors built an LLM-based judge and validated it against human gold labels in a companion benchmark, JudgeEval.

The strongest agent, Claude 3.5 Sonnet (New) with a simple scaffold, averaged only 21.0%. On a small subset, ML PhDs outperformed the models. The benchmark offers a way to track autonomous ML R&D ability, which is relevant to frontier-model safety frameworks.

Key points

  • The benchmark contains 20 ICML 2024 papers with 8,316 individually gradable rubric requirements. Scores propagate up each rubric tree as weighted averages to produce a Replication Score.
  • The best automated judge, o3-mini-high with custom scaffolding, reached an F1 of 0.83 on JudgeEval.
  • On a 3-paper subset, the best of three PhD attempts scored 41.4% after 48 hours, compared with 26.6% for o1.
  • PaperBench Code-Dev is a lighter variant that checks code only and needs no GPU. On it, o1 scored 43.4%, and grading cost fell from about $66 to about $10 per paper.
  • A full o1 IterativeAgent run costs roughly $400 per paper, or about $8,000 for all 20. A monitor disqualified 10 of 646 runs for using blacklisted resources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

Cite

Cite as Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese and Tejal Patwardhan. 2025. PaperBench: Evaluating AI's Ability to Replicate AI Research. arXiv:2504.01848.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2504.01848 · arXiv 2504.01848

@misc{starace2025paperbench,
  title = {{PaperBench: Evaluating AI's Ability to Replicate AI Research}},
  author = {Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese and Tejal Patwardhan},
  year = {2025},
  doi = {10.48550/arxiv.2504.01848},
  eprint = {2504.01848},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.01848}
}

References

43 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Albergo, M. S., Goldstein, M., Boffi, N. M., Ranganath, R., and Vanden-Eijnden, E. Stochastic Interpolants with Data-Dependent Couplings. June 2024. URL https://openreview.net/forum?id=FFILRGD0jG. ↗
  2. However, the capability to autonomously replicate and ex-Anthropic. Responsible Scaling Policy. https://www.anthropic.com/news/announcing-our-up dated-responsible-scaling-policy, 2024. Accessed: 2025-01-29. ↗
  3. Cai, C., Ye, Z., Feng, L., Qi, J., and Liu, F. Sample-specific Masks for Visual Reprogramming-based Prompting. June 2024a. URL https://openreview.net/forum ?id=4sikyurTLX. ↗
  4. Harvey Team. Introducing BigLaw Bench, August 2024. URL https://www.harvey.ai/blog/intro ducing-biglaw-bench. ↗
  5. Huang, Q., Vora, J., Liang, P., and Leskovec, J. MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. In Forty-first International Conference on Machine Learning, 2024.
  6. Jansen, P., Côté, M.-A., Khot, T., Bransom, E., Mishra, B. D., Majumder, B. P., Tafjord, O., and Clark, P. DIS-COVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents. arXiv preprint arXiv:2406.06769, 2024. find
  7. Cai, D., Modi, C., Pillaud-Vivien, L., Margossian, C., Gower, R. M., Blei, D., and Saul, L. K. Batch and match: black-box variational inference with a score-based divergence. June 2024b. URL https://openreview.net/forum?id=bplNmU2ROC. ↗
  8. Jin, X. and Ren, X. What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement. June 2024. URL https://openreview.net/for um?id=bzNwexOPWm. ↗
  9. Chan, J. S., Chowdhury, N., Jaffe, O., Aung, J., Sherburn, D., Mays, E., Starace, G., Liu, K., Maksin, L., Patwardhan, T., et al. MLE-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024.
  10. Jing, L., Huang, Z., Wang, X., Yao, W., Yu, W., Ma, K., Zhang, H., Du, X., and Yu, D. DSBench: How Far Are Data Science Agents to Becoming Data Science Experts? arXiv preprint arXiv:2409.07703, 2024. find
  11. Chen, D., Chen, R., Zhang, S., Liu, Y., Wang, Y., Zhou, H., Zhang, Q., Wan, Y., Zhou, P., and Sun, L. MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark. arXiv preprint arXiv:2402.04788, 2024.
  12. Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. RewardBench: Evaluating Reward Models for Language Modeling, 2024. URL https://arxiv.org/abs/2403.13787. ↗
  13. Cheng, Z., Wu, X., Yu, J., Yang, S., Wang, G., and Xing, X. RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation. Lee, A., Bai, X., Pres, I., Wattenberg, M., Kummerfeld, J. K., and Mihalcea, R. A Mechanistic Understanding of Alignment Algorithms: A Case Study on… — A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity
  14. Chiang, C.-H. and yi Lee, H. Can Large Language Models Be an Alternative to Human Evaluations?, 2023. URL https://arxiv.org/abs/2305.01937. ↗
  15. Niu, S., Miao, C., Chen, G., Wu, P., and Zhao, P. Test-Time Model Adaptation with Only Forward Passes. June 2024. URL https://openreview.net/forum?id= qz1Vx1v9iK. ↗
  16. DeepMind. Specification Gaming: The Flip Side of AI Ingenuity, 2024. URL https://deepmind.google/ discover/blog/specification- gaming-the- flip- side- of- ai- ingenuity/. Accessed: 10 March 2025. ↗
  17. OpenAI. Preparedness Framework, December 2023. find
  18. Frans, K., Park, S., Abbeel, P., and Levine, S. Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings. June 2024. URL https://openreview.net/forum?id=a6wCNfIj8E. ↗
  19. Pan, A., Bhatia, K., and Steinhardt, J. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models, 2022. URL https://arxiv.org/abs/ 2201.03544. ↗
  20. Fu, J., Ng, S.-K., Jiang, Z., and Liu, P. GPTScore: Evaluate as You Desire. arXiv preprint arXiv:2302.04166, 2023. find
  21. Rathore, P., Lei, W., Frangella, Z., Lu, L., and Udell, M. Challenges in Training PINNs: A Loss Landscape Perspective. June 2024. URL https://openreview.net/forum?id=mJGiFr8jLa. ↗
  22. Gloeckler, M., Deistler, M., Weilbach, C. D., Wood, F., and Macke, J. H. All-in-one simulation-based inference. June 2024. URL https://openreview.net/forum ?id=DL79HYCFFq. ↗
  23. Sanchez, G., Spangher, A., Fan, H., Levi, E., and Biderman, S. Stay on Topic with Classifier-Free Guidance. June 2024. URL https://openreview.net/forum ?id=RiM3cl9MdK. ↗
  24. Google DeepMind. Frontier Safety Framework, May 2024. find
  25. Sawada, T., Paleka, D., Havrilla, A., Tadepalli, P., Vidas, P., Kranias, A., Nay, J. J., Gupta, K., and Komatsuzaki, A. ARB: Advanced Reasoning Benchmark for Large Language Models, 2023. URL https://arxiv.org/abs/2307.13692. ↗
  26. J., and Li, X. Self-Taught Evaluators, 2024a. URL https://arxiv.org/abs/2408.02666. ↗
  27. Wang, X., Lin, B., Liu, D., Chen, Y.-C., and Xu, C. Bridging Data Gaps in Diffusion Models with Adversarial Noise-Based Transfer Learning. June 2024b. URL https://openreview.net/forum?id=PpBs2iL0jv. ↗
  28. Schlarmann, C., Singh, N. D., Croce, F., and Hein, M. Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models. June 2024. URL https://openreview.net/forum?id=WLPhywf1si. ↗
  29. Wijk, H., Lin, T., Becker, J., Jawhar, S., Parikh, N., Broadley, T., Chan, L., Chen, M., Clymer, J., Dhyani, J., et al. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114, 2024. find
  30. Sharrock, L., Simons, J., Liu, S., and Beaumont, M. Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Models. June 2024. URL https://openreview.net/forum ?id=8viuf9PdzU. ↗
  31. Wolczyk, M., Cupial, B., Ostaszewski, M., Bortkiewicz, M., Zajac, M., Pascanu, R., Kucinski, L., and Milos, P. Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem. June 2024. URL https://openreview.net/forum?id=53iSXb1m8w. ↗
  32. Shi, J., Gare, G. R., Tian, J., Chai, S., Lin, Z., Vasudevan, A. B., Feng, D., Ferroni, F., and Kong, S. LCA-on-the-Line: Benchmarking Out of Distribution Generalization with Class Taxonomies. June 2024. URL https://op enreview.net/forum?id=HPXRzM9BYZ. find
  33. Xia, X., Liu, J., Zhang, S., Wu, Q., Wei, H., and Liu, T. Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraints. June 2024. URL https://openreview.net/forum?id= yb5xV8LFDq. ↗
  34. Si, C., Yang, D., and Hashimoto, T. Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers. arXiv preprint arXiv:2409.04109, 2024.
  35. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. ReAct: Synergizing Reasoning and Acting in Language Models, March 2023. URL http://arxiv.org/abs/2210.03629. arXiv:2210.03629 [cs]. ↗
  36. Siegel, Z. S., Kapoor, S., Nagdir, N., Stroebl, B., and Narayanan, A. CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark. arXiv preprint arXiv:2409.11363, 2024. find
  37. Zhao, B., Hajishirzi, H., and Cao, Q. APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference. June 2024. URL https://openreview.net/forum?id=sb81Xl50JG. ↗
  38. Singla, J., Agarwal, A., and Pathak, D. SAPG: Split and Aggregate Policy Gradients. June 2024. URL https://openreview.net/forum?id=4dOJAfXhNV. ↗
  39. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36: 46595–46623, 2023.
  40. Sun, H., Zhuang, Y., Wei, W., Zhang, C., and Dai, B. BBox-Adapter: Lightweight Adapting for Black-Box Large Language Models. Zhuge, M., Zhao, C., Ashley, D., Wang, W., Khizbullin, D., Xiong, Y., Liu, Z., Chang, E., Krishnamoorthi, R., Tian, Y., et al. Agent-as-a-Judge: Evaluate Agents with Agents. a… find
  41. UK AI Safety Institute. Inspect, 2025. URL https://inspect.ai-safety-institute.org.uk /agents.html\#sec-basic-agent. van der Weij, T., Hofstätter, F., Jaffe, O., Brown, S. F., and Ward, F. R. AI Sandbagging: Language Models Can Strategically Underperform on Evaluations. arXiv preprint arXiv:2406.07… ↗
  42. Wang, R., Jansen, P., Côté, M.-A., and Ammanabrolu, P. Scienceworld: Is Your Agent Smarter than a 5th Grader? arXiv preprint arXiv:2203.07540, 2022. find
  43. Wang, T., Kulikov, I., Golovneva, O., Yu, P., Yuan, W., Dwivedi-Yu, J., Pang, R. Y., Fazel-Zarandi, M., Weston, A. Future Directions in AI Evaluation find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/starace-2025-paperbench-evaluating-ai-s/