Elamur Research⌕
☰
home / papers / 2025 / ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code

ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code

Paper · 2025 · Tianyu Hua; Harper Hua; Violet Xiang; Benjamin Klieger; Sang T. Truong; Weixin Liang; Fan-Yun Sun; and Nick Haber

Summary

Can LLMs turn genuinely new ML ideas, ones published after their training data was gathered, into working code? ResearchCodeBench tests this with an execution-based benchmark built from 20 recent machine learning papers. The papers come from top venues such as NeurIPS, ICLR and CVPR, plus arXiv, and cover generative modeling, vision, theory and reinforcement learning.

In each paper's official repository, the authors tag the code behind its central contribution and mask it at several levels of granularity, from whole functions to single lines. A model must fill the gap using the paper text, relevant context files and a short hint. The completion is re-inserted and judged by equivalence tests against the reference implementation, supplemented by hand-written unit tests. These tests were developed with paper authors or domain experts.

Across more than 30 proprietary and open models, the strongest passes under 40% of the code by a line-weighted score. Most failures are logic errors rather than syntax problems, which points to scientific reasoning as the main bottleneck. The framework is open, inexpensive to run and built for community expansion.

Key points

  • The benchmark contains 212 fill-in-the-blank coding challenges drawn from 20 ML papers from 2024–2025. Tasks are nested hierarchically so that inner snippets are never harder than the snippets that contain them.
  • The headline metric is the scaled pass rate, in which each snippet counts in proportion to how many executable lines of code it contains. A plain per-snippet pass rate is also reported.
  • Top results: Gemini-2.5-Pro-Preview 37.3%, O3 (High) 32.3%, O4-mini (High) 30.8%.
  • Failures were classified with GPT-4O-Mini. Functional (semantic) errors make up 58.6%, while name, type and syntax errors are each roughly 8–9%.
  • Unlike PaperBench, the benchmark isolates the core implementation step and relies on deterministic tests instead of LLM judges. Stated limitations include ML-only scope, manually written tests and no human baseline. Code and data are at researchcodebench.github.io.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States567 works · 335,190 citations🏢OpenAISan Francisco · United States25 works · 27,899 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

Cite

Cite as Tianyu Hua; Harper Hua; Violet Xiang; Benjamin Klieger; Sang T. Truong; Weixin Liang; Fan-Yun Sun; and Nick Haber. 2025. ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code. arXiv:2506.02314.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2506.02314 · arXiv 2506.02314

@misc{sun2025researchcodebench,
  title = {{ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code}},
  author = {Tianyu Hua; Harper Hua; Violet Xiang; Benjamin Klieger; Sang T. Truong; Weixin Liang; Fan-Yun Sun; and Nick Haber},
  year = {2025},
  doi = {10.48550/arxiv.2506.02314},
  eprint = {2506.02314},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.02314}
}

References

43 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Silas Alberti, Kenan Hasanaliyev, Manav Shah, and Stefano Ermon. Data unlearning in diffusion models. In International Conference on Learning Representations (ICLR), 2025. find
  2. Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, and Tushar Khot. Super: Evaluating agents on setting up and executing tasks from research repositories. arXiv preprint arXiv:2409.07440, 2024. find
  3. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024.
  4. Hansheng Chen, Kai Zhang, Hao Tan, Zexiang Xu, Fujun Luan, Leonidas Guibas, Gordon Wetzstein, and Sai Bi. Gaussian mixture flow matching models. arXiv preprint arXiv:2504.05304, 2025a. URL https://arxiv.org/abs/2504.05304. — [arXiv:2504.05304](https://arxiv.org/abs/2504.05304) ↗
  5. Lesi Chen, Chengchang Liu, and Jingzhao Zhang. Second-order min-max optimization with lazy hessians. In International Conference on Learning Representations, 2025b. find
  6. Aaron Defazio, Xingyu Yang, Ahmed Khaled, Konstantin Mishchenko, Harsh Mehta, and Ashok Cutkosky. The road less scheduled. Advances in Neural Information Processing Systems, 37: 9974–10007, 2024. find
  7. Juan Agustin Duque, Milad Aghajohari, Tim Cooijmans, Razvan Ciuca, Tianyu Zhang, Gauthier Gidel, and Aaron Courville. Advantage alignment algorithms. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=QFO1asgas2. ↗
  8. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020.
  9. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist. arXiv preprint arXiv:2502.18864, February 2025. URL https://arxiv.org/abs/2502.18864. ↗
  10. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning, 2024.
  11. Hugging Face. Hugging face papers, 2025. URL https://huggingface.co/papers. Accessed: 2025-05-08. ↗
  12. Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. R2e: Turning any github repository into a programming agent environment. In Forty-first International Conference on Machine Learning, 2024. find
  13. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. ↗
  14. Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang. Agentreview: Exploring peer review dynamics with llm agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1208–1226, 2024.
  15. Tommie Kerssies, Niccolò Cavagnero, Alexander Hermans, Narges Norouzi, Giuseppe Averta, Bastian Leibe, Gijs Dubbelman, and Daan de Geus. Your vit is secretly an image segmentation model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. URL https://a… — [arXiv:2503.19108](https://arxiv.org/abs/2503.19108) find
  16. Hyunwoo Lee, Hayoung Choi, and Hyunju Kim. Robust weight initialization for tanh neural networks with fixed point analysis. In International Conference on Learning Representations (ICLR), 2025. find
  17. Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers. arXiv preprint arXiv:2504.10483, 2025. URL https://arxiv.org/abs/2504.10483. ↗
  18. Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. Preference leakage: A contamination problem in llm-as-a-judge. arXiv preprint arXiv:2502.01534, 2025a. — [arXiv:2502.01534](https://arxiv.org/abs/2502.01534) find
  19. Tianhong Li, Qinyi Sun, Lijie Fan, and Kaiming He. Fractal generative models. arXiv preprint arXiv:2502.17437, 2025b. URL https://arxiv.org/abs/2502.17437. ↗
  20. Weixin Liang, Eric Zhang, et al. Mapping the increasing use of llms in scientific papers. In Proceedings of the Conference on Language Modeling (COLM), 2024a.
  21. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8):AIoa2400196, 2024b.
  22. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.
  23. Mistral. Mistral ocr, March 2025. URL https://mistral.ai/news/mistral-ocr. Accessed: 2025-04-27. ↗
  24. Matthieu Moy. latexpand: Flatten latex files by replacing input and include. https://gitlab.com/latexpand/latexpand/, 2023. Accessed: 2025-04-27. ↗
  25. Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. Ml… — MLGym: A New Framework and Benchmark for Advancing AI Research Agents
  26. Minh Nguyen, Andrew Baker, Clement Neo, Allen Roush, Andreas Kirsch, and Ravid Shwartz-Ziv. Turning up the heat: Min-p sampling for creative and coherent llm outputs. In International Conference on Learning Representations, 2025. find
  27. Jianning Pei, Han Hu, and Shuyang Gu. Optimal stepsize for diffusion sampling. arXiv preprint arXiv:2503.21774, 2025. — [arXiv:2503.21774](https://arxiv.org/abs/2503.21774) find
  28. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, M. Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. ArXiv, abs/2009.10297, 2020. URL https://api.semanticscholar.org/CorpusID:221836101. ↗
  29. Giuseppe Russo Latona, Manoel Horta Ribeiro, Tim R. Davidson, Veniamin Veselovsky, and Robert West. The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates. arXiv preprint arXiv:2405.02150, 2024.
  30. Simon Schug, Seijin Kobayashi, Yassir Akram, João Sacramento, and Razvan Pascanu. Attention as a hypernetwork. In International Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2406.05816. — [arXiv:2406.05816](https://arxiv.org/abs/2406.05816) ↗
  31. Juntong Shi, Minkai Xu, Harper Hua, Hengrui Zhang, Stefano Ermon, and Jure Leskovec. Tabdiff: a mixed-type diffusion model for tabular data generation. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=swvURjrt8z. ↗
  32. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=M23dTGWCZy. ↗
  33. Zachary S Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. arXiv preprint arXiv:2409.11363, 2024.
  34. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai’s ability to replicate ai research. In International Conference on Machin… — PaperBench: Evaluating AI's Ability to Replicate AI Research
  35. Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yuf… — SciCode: A Research Coding Benchmark Curated by Scientists
  36. Weixi Tong and Tianyi Zhang. CodeJudge: Evaluating code generation with large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20032–20051, Miami, Florida, USA, November 2024.… find
  37. Eric Wallace et al. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. find
  38. Yutong Wang, Pengliang Ji, Chaoqun Yang, Kaixin Li, Ming Hu, Jiaoyang Li, and Guillaume Sartoretti. Mcts-judge: Test-time scaling in llm-as-a-judge for code correctness evaluation. arXiv preprint arXiv:2502.12468, 2025. — [arXiv:2502.12468](https://arxiv.org/abs/2502.12468) find
  39. Xin Wen, Bingchen Zhao, Ismail Elezi, Jiankang Deng, and Xiaojuan Qi. ” principal components” enable a new language of images. arXiv preprint arXiv:2503.08685, 2025. — [arXiv:2503.08685](https://arxiv.org/abs/2503.08685) find
  40. Dehong Xu, Ruiqi Gao, Wen-Hao Zhang, Xue-Xin Wei, and Ying Nian Wu. On conformal isometry of grid cells: Learning distance-preserving position embedding. In International Conference on Learning Representations, 2025. find
  41. Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. In International Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2410.05258. — [arXiv:2410.05258](https://arxiv.org/abs/2410.05258) ↗
  42. Ziming Zhang, Fangzhou Lin, Haotian Liu, Jose Morales, Haichong Zhang, Kazunori Yamada, Vijaya B Kolachalama, and Venkatesh Saligrama. Gps: A probabilistic distributional similarity with gumbel priors for set-to-set matching. In The Thirteenth International Conference on Learning Representations, 20… find
  43. Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. URL https://arxiv.org/abs/2503.10622. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hua-2025-researchcodebench-benchmarking-llms-implementing/