Elamur Research⌕
☰
home / papers / 2024 / CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

Paper · 2024 · Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir et al.

Summary

Many published studies cannot be rerun to obtain their reported numbers, even when the authors share code and data, and checking this by hand takes expert effort. The authors ask whether AI agents can take over this verification work, and they build CORE-Bench to measure it. Starting from 5,090 research capsules on CodeOcean, they kept 90 papers from computer science, social science, and medicine whose Python or R code they confirmed runs locally. They derived three tasks per paper, for 270 in total. Difficulty depends on how much help the agent gets. Easy supplies pre-computed outputs, Medium supplies a Dockerfile, and Hard supplies only the README, so the agent must install dependencies itself. Questions require reading either text output or figures. A parallel evaluation harness runs each task in its own VM and cuts evaluation time from weeks to hours. The baselines are AutoGPT and an adapted CORE-Agent, each run on GPT-4o and GPT-4o-mini. Light task-specific tuning helps substantially, yet the best agent solves only 21% of Hard tasks. The benchmark gives research-assistant agents a realistic target that can be updated over time.

Key points

  • Dataset: CORE-Bench has 270 tasks from 90 CodeOcean papers in three disciplines, with Python and R codebases. It is one of the few agent benchmarks that includes R. A task counts as solved only if every question about the repository is answered correctly.
  • Best results: CORE-Agent with GPT-4o reached about 60% on CORE-Bench-Easy and 21% on CORE-Bench-Hard. AutoGPT, the general-purpose agent, scored notably lower.
  • Text vs. figures: Questions based on text output were much easier than figure-based ones. On Easy, CORE-Agent with GPT-4o answered 87.88% of written questions and 59.26% of vision questions correctly. Python tasks were clearly easier than R tasks.
  • Failure modes:
    • Agents often picked the wrong output file or figure.
    • They got stuck reinstalling dependencies until they hit the cost limit, and raising the limit did not help much.
    • Average cost was $0.54 for successful runs versus $2.59 for failed ones (CORE-Agent with GPT-4o).
  • Motivation: In the 2022 ML reproducibility challenge, only 18 of 28 papers with code and data were fully reproducible. Code and harness are available at github.com/siegelz/core-bench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Princeton UniversityPrinceton · United States137 works · 53,849 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksDecision Sciences · 306 worksInformation Systems and Management · 96 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 works

Cite

Cite as Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl and Arvind Narayanan. 2024. CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark. arXiv:2409.11363.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.11363 · arXiv 2409.11363

@misc{siegel2024core,
  title = {{CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark}},
  author = {Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl and Arvind Narayanan},
  year = {2024},
  doi = {10.48550/arxiv.2409.11363},
  eprint = {2409.11363},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.11363}
}

References

54 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AISI. Inspect, 2024. URL https://inspect.ai-safety-institute.org.uk. ↗
  2. Rose L. Andrew, Arianne Y.K. Albert, Sebastien Renaut, Diana J. Rennison, Dan G. Bock, and Tim Vines. Assessing the reproducibility of discriminant function analyses. PeerJ, 3:e1137, August 2015. ISSN 2167-8359. doi: 10.7717/peerj.1137. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4540019/. — [doi:10.7717/peerj.1137](https://doi.org/10.7717/peerj.1137) ↗
  3. Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. A Systematic Review of Reproducibility Research in Natural Language Processing. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational… find
  4. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles… find
  5. Matteo Brivio and Çağrı Çöltekin. [Re] Exploring the Representation of Word Meanings in Context. ReScience C, 9(2):\#5, July 2023. doi: 10.5281/zenodo.8173658. URL https://zenodo.org/records/8173658. — [doi:10.5281/zenodo.8173658](https://doi.org/10.5281/zenodo.8173658) ↗
  6. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, July 2024. URL http://arxiv.org/abs/2407.21787. arXiv:2407.21787 [cs]. ↗
  7. Jonathan B. Buckheit and David L. Donoho. WaveLab and Reproducible Research. In Anestis Antoniadis and Georges Oppenheim (eds.), Wavelets and Statistics, pp. 55–81. Springer, New York, NY, 1995. ISBN 978-1-4612-2544-7. doi: 10.1007/978-1-4612-2544-7_5. URL https://doi.org/10.1007/978-1-4612-2544-7_… — [doi:10.1007/978-1-4612-2544-7_5](https://doi.org/10.1007/978-1-4612-2544-7_5) ↗
  8. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation,…
  9. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  10. Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Alijubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Kevin Liu, and Aleksander Madry. Introducing SWE-bench Verified, August 2024. URL https://openai.com/index/ introducing-swe-benc… ↗
  11. April Clyburne-Sherin, Xu Fei, and Seth Ariel Green. Computational Reproducibility via Containers in Psychology. Meta-Psychology, 3, November 2019. ISSN 2003-2714. doi: 10.15626/MP.2018.892. URL https://open.lnu.se/index.php/metapsychology/article/view/892. — [doi:10.15626/mp.2018.892](https://doi.org/10.15626/mp.2018.892) ↗
  12. Christian Collberg and Todd A. Proebsting. Repeatability in computer systems research. Communications of the ACM, 59(3):62–69, February 2016. ISSN 0001-0782, 1557-7317. doi: 10.1145/2812803. URL https://dl.acm.org/doi/10.1145/2812803. — [doi:10.1145/2812803](https://doi.org/10.1145/2812803) ↗
  13. Paul Gertler, Sebastian Galiani, and Mauricio Romero. How to make replication the norm. Nature, 554(7693): 417–419, February 2018. doi: 10.1038/d41586-018-02108-9. URL https://www.nature.com/articles/ d41586-018-02108-9. Bandiera_abtest: a Cg_type: Comment Publisher: Nature Publishing Group Subject_… — [doi:10.1038/d41586-018-02108-9](https://doi.org/10.1038/d41586-018-02108-9) ↗
  14. Kimberly J. Gilbert, Rose L. Andrew, Dan G. Bock, Michelle T. Franklin, Nolan C. Kane, Jean-Sébastien Moore, Brook T. Moyers, Sébastien Renaut, Diana J. Rennison, Thor Veen, and Timothy H. Vines. Recommendations for utilizing and reporting population genetic analyses: the reproducibility of genetic… — [doi:10.1111/j.1365-294x.2012.05754.x](https://doi.org/10.1111/j.1365-294x.2012.05754.x) find
  15. Tom E. Hardwicke, Maya B. Mathur, Kyle MacDonald, Gustav Nilsonne, George C. Banks, Mallory C. Kidwell, Alicia Hofelich Mohr, Elizabeth Clayton, Erica J. Yoon, Michael Henry Tessler, Richie L. Lenne, Sara Altman, Bria Long, and Michael C. Frank. Data availability, reusability, and analytic reproduci… — [doi:10.1098/rsos.180448](https://doi.org/10.1098/rsos.180448) find
  16. Tom E. Hardwicke, Manuel Bohn, Kyle MacDonald, Emily Hembacher, Michèle B. Nuijten, Benjamin N. Peloquin, Benjamin E. deMayo, Bria Long, Erica J. Yoon, and Michael C. Frank. Analytic reproducibility in articles receiving open data badges at the journal Psychological Science : an observational study.… — [doi:10.1098/rsos.201494](https://doi.org/10.1098/rsos.201494) find
  17. Michael Hassid, Tal Remez, Jonas Gehring, Roy Schwartz, and Yossi Adi. The Larger the Better? Improved LLM Code-Generation via Budget Reallocation, July 2024. URL http://arxiv.org/abs/2404.00725. arXiv:2404.00725 [cs]. — [arXiv:2404.00725](https://arxiv.org/abs/2404.00725) ↗
  18. Yifeng He, Ethan Wang, Yuyang Rong, Zifei Cheng, and Hao Chen. Security of AI Agents, 2024. URL https://arxiv.org/abs/2406.08689. Version Number: 2. — [arXiv:2406.08689](https://arxiv.org/abs/2406.08689) ↗
  19. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Benchmarking Large Language Models As AI Research Agents, October 2023. URL https://arxiv.org/abs/2310.03302v1. ↗
  20. John P. A. Ioannidis, David B. Allison, Catherine A. Ball, Issa Coulibaly, Xiangqin Cui, Aedín C. Culhane, Mario Falchi, Cesare Furlanello, Laurence Game, Giuseppe Jurman, Jon Mangion, Tapan Mehta, Michael Nitzberg, Grier P. Page, Enrico Petretto, and Vera van Noort. Repeatability of published micro… — [doi:10.1038/ng.295](https://doi.org/10.1038/ng.295) find
  21. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, October 2023. URL https://arxiv.org/abs/2310.06770v1. ↗
  22. Sayash Kapoor and Arvind Narayanan. Evaluating LLMs is a minefield, October 2023. URL https://www.aisnakeoil.com/p/evaluating-llms-is-a-minefield. ↗
  23. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That Matter, July 2024. URL http://arxiv.org/abs/2407.01502. arXiv:2407.01502 [cs]. ↗
  24. Markus Konkol, Christian Kray, and Max Pfeiffer. Computational reproducibility in geoscientific papers: Insights from a series of studies with geoscientists and a reproduction study. International Journal of Geographical Information Science, 33(2):408–429, February 2019. ISSN 1365-8816. doi: 10.1080… — [doi:10.1080/13658816](https://doi.org/10.1080/13658816) find
  25. Jimmy Koppel. Everyone’s talking about Sakana’s AI scientist. But no-one’s answering the big question: is its output good?, August 2024. URL https://x.com/jimmykoppel/status/1828077203956850756. ↗
  26. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James M… — Competition-Level Code Generation with AlphaCode
  27. David M. Liu and Matthew J. Salganik. Successes and Struggles with Computational Reproducibility: Lessons from the Fragile Families Challenge. Socius, 5:2378023119849803, January 2019. ISSN 2378-0231. doi: 10.1177/2378023119849803. URL https://doi.org/10.1177/2378023119849803. Publisher: SAGE Public… — [doi:10.1177/2378023119849803](https://doi.org/10.1177/2378023119849803) ↗
  28. Victor Livernoche and Vidya Sujaya. [Re] A Reproduction of Automatic Multi-Label Prompting: Simple and Interpretable Few-Shot Classification. ReScience C, 9(2):\#33, July 2023. doi: 10.5281/zenodo.8173735. URL https://zenodo.org/records/8173735. — [doi:10.5281/zenodo.8173735](https://doi.org/10.5281/zenodo.8173735) ↗
  29. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery, August 2024. URL http://arxiv.org/abs/ 2408.06292. arXiv:2408.06292 [cs]. ↗
  30. Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, F… find
  31. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. DiscoveryBench: Towards Data-Driven Discovery with Large Language Models, July 2024. URL http://arxiv.org/abs/2407.01725.… ↗
  32. B. D. McCullough, Kerry Anne McGeary, and Teresa D. Harrison. Lessons from the JMCB Archive. Journal of Money, Credit and Banking, 38(4):1093–1107, 2006. ISSN 0022-2879. URL https://www.jstor.org/ stable/3838995. Publisher: [Wiley, Ohio State University Press]. ↗
  33. National Academies of Sciences Engineering and Medicine. Reproducibility and Replicability in Science. 2019. doi: 10.17226/25303. URL https://nap.nationalacademies.org/read/25303/chapter/7. — [doi:10.17226/25303](https://doi.org/10.17226/25303) ↗
  34. METR. Vivaria, 2024. URL https://vivaria.metr.org. ↗
  35. Florian Naudet, Charlotte Sakarovitch, Perrine Janiaud, Ioana Cristea, Daniele Fanelli, David Moher, and John P. A. Ioannidis. Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS M… — [doi:10.1136/bmj.k400](https://doi.org/10.1136/bmj.k400) find
  36. Pepijn Obels, Daniël Lakens, Nicholas A. Coles, Jaroslav Gottfried, and Seth A. Green. Analysis of Open Data and Computational Reproducibility in Registered Reports in Psychology. Advances in Methods and Practices in Psychological Science, 3(2):229–237, June 2020. ISSN 2515-2459. doi: 10.1177/251524… — [doi:10.1177/2515245920918872](https://doi.org/10.1177/2515245920918872) find
  37. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d’Alche Buc, Emily Fox, and Hugo Larochelle. Improving Reproducibility in Machine Learning Research(A Report from the NeurIPS 2019 Reproducibility Program). Journal of Machine Learning Research, 22… find
  38. Ori Press, Andreas Hochlehnert, Ameya Prabhu, Vishaal Udandarao, Ofir Press, and Matthias Bethge. CiteME: Can Language Models Accurately Cite Scientific Claims?, July 2024. URL http://arxiv.org/ abs/2407.12861. arXiv:2407.12861 [cs]. ↗
  39. Christophe Pérignon, Olivier Akmansoy, Christophe Hurlin, Anna Dreber, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Michael Kirchler, Albert J. Menkveld, Michael Razen, and Utz Weitzel. Computational Reproducibility in Finance: Evidence from 1,000 Tests, February 2024. URL https://papers.s… ↗
  40. Edward Raff. A Step Toward Quantifying Independently Reproducible Machine Learning Research, September 2019. URL http://arxiv.org/abs/1909.06674. arXiv:1909.06674 [cs, stat]. ↗
  41. Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. AI and the Everything in the Whole Wide World Benchmark, 2021. URL https://arxiv.org/abs/2111.15366. Version Number: 1. ↗
  42. Sheeba Samuel and Daniel Mietchen. Computational reproducibility of Jupyter notebooks from biomedical publications. GigaScience, 13:giad113, January 2024. ISSN 2047-217X. doi: 10.1093/gigascience/giad113. URL https://doi.org/10.1093/gigascience/giad113. — [doi:10.1093/gigascience/giad113](https://doi.org/10.1093/gigascience/giad113) ↗
  43. Significant Gravitas. AutoGPT, September 2024. URL https://github.com/Significant-Gravitas/ AutoGPT. original-date: 2023-03-16T09:21:07Z. ↗
  44. Koustuv Sinha, Maurits Bleeker, Samarth Bhargav, Jessica Zosa Forde, Sharath Chandra Raparthy, Jesse Dodge, Joelle Pineau, and Robert Stojnic. ML Reproducibility Challenge 2022. ReScience C, 9(2):\#46, July 2023. doi: 10.5281/zenodo.8200058. URL https://zenodo.org/records/8200058. — [doi:10.5281/zenodo.8200058](https://doi.org/10.5281/zenodo.8200058) ↗
  45. Jeffrey R. Spence and David J. Stanley. Prediction Interval: What to Expect When You’re Expecting . . . A Replication. PLOS ONE, 11(9):e0162874, September 2016. ISSN 1932-6203. doi: 10.1371/journal.pone. 0162874. URL https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0162874. Publishe… — [doi:10.1371/journal.pone](https://doi.org/10.1371/journal.pone) ↗
  46. Daniel Stockemer, Sebastian Koehler, and Tobias Lentz. Data Access, Transparency, and Replication: New Insights from the Political Behavior Literature. PS: Political Science & Politics, 51 (4):799–803, October 2018. ISSN 1049-0965, 1537-5935. doi: 10.1017/S1049096518000926. URL https://www.cambridge… — [doi:10.1017/s1049096518000926](https://doi.org/10.1017/s1049096518000926) ↗
  47. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yuf… — SciCode: A Research Coding Benchmark Curated by Scientists
  48. Ana Trisovic, Matthew K. Lau, Thomas Pasquier, and Mercè Crosas. A large-scale study on research code quality and execution. Scientific Data, 9(1):60, February 2022. ISSN 2052-4463. doi: 10.1038/ s41597-022-01143-6. URL https://www.nature.com/articles/s41597-022-01143-6. Publisher: Nature Publishing… ↗
  49. Benjamin D. K. Wood, Rui Müller, and Annette N. Brown. Push button replication: Is impact evaluation evidence for international development verifiable? PLOS ONE, 13(12):e0209416, December 2018. ISSN 1932-6203. doi: 10.1371/journal.pone.0209416. URL https://journals.plos.org/plosone/article? id=10.13… — [doi:10.1371/journal.pone.0209416](https://doi.org/10.1371/journal.pone.0209416) ↗
  50. Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. ChartBench: A Benchmark for Complex Visual Reasoning in Charts, June 2024. URL http://arxiv.org/abs/2312.15915. arXiv:2312.15915 [cs]. — [arXiv:2312.15915](https://arxiv.org/abs/2312.15915) ↗
  51. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-AGENT: Agent-Computer Interfaces Enable Automated Software Engineering. 2024. URL https://swe-agent.com/paper.pdf. ↗
  52. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. $\\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, June 2024. URL http://arxiv.org/abs/2406.12045. arXiv:2406.12045 [cs]. ↗
  53. Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The Shift from Models to Compound AI Systems, February 2024. URL http://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/. ↗
  54. Yaolun Zhang, Yinxu Pan, Yudong Wang, and Jie Cai. PyBench: Evaluating LLM Agent on various real-world coding tasks, August 2024. URL http://arxiv.org/abs/2407.16732. arXiv:2407.16732 [cs]. — [arXiv:2407.16732](https://arxiv.org/abs/2407.16732) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/siegel-2024-core-bench-fostering-credibility/