Elamur Research
☰
home / papers / 2024 / MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2410.07095

Summary

MLE-bench addresses the lack of benchmarks that test whether AI agents can carry out machine learning engineering autonomously from start to finish, rather than solving isolated coding problems. The OpenAI authors hand-picked 75 Kaggle competitions covering NLP, computer vision, signal processing, tabular data and other domains. Each task bundles the competition description, a dataset (often with a newly built train-test split when the original test labels were never released), local grading code and a snapshot of the Private leaderboard. Agents earn Kaggle-style medals by comparing their submission with how humans performed. The main metric is the share of attempts that reach at least bronze. Several frontier models were tested inside open-source scaffolds (AIDE, MLAB, OpenHands). The strongest setup, o1-preview with AIDE, reached a medal in 16.9% of competitions. Results improved with more attempts and longer runtime. Agents handled problems solvable with standard recipes but had trouble debugging and recovering from errors. The benchmark matters for tracking progress toward autonomous ML R&D, a capability that frontier-lab safety frameworks monitor.

Key points

  • Selection pipeline: 5673 finished competitions in Meta Kaggle were narrowed to 586 by removing Community Competitions, then manually screened down to 75. A separate set of 7 competitions serves as a development split.
  • The 75 tasks are labeled by complexity: 22 Low (30%), 38 Medium (50%) and 15 High (20%). Combined prize money is $1,948,016.
  • Medal rates for o1-preview + AIDE:
    • 16.9% at pass@1, rising to 34.1% at pass@8.
    • 34.3% on Low-complexity tasks.
  • GPT-4o gained from extra time, going from 8.7% with 24 hours per competition to 11.8% with 100 hours.
  • Rules require predictions to come from a trained model and forbid looking up online solutions. The authors also studied pre-training contamination, used tools to detect plagiarism, and released the code at github.com/openai/mle-bench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

123 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 117 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.07095 · arXiv 2410.07095

@misc{chan2024mle,
  title = {{MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering}},
  author = {Chan, Jun Shern and Chowdhury, Neil and Jaffe, Oliver and Aung, James and Sherburn, Dane and Mays, Evan and Starace, Giulio and Liu, Kevin and Maksin, Leon and Patwardhan, Tejal and Weng, Lilian and Mądry, Aleksander},
  year = {2024},
  doi = {10.48550/arxiv.2410.07095},
  eprint = {2410.07095},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.07095}
}

References

31 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Anthropic’s Responsible Scaling Policy, Version 1.0, September 2023. find
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program Synthesis with Large Language Models, August 2021. URL http://arxiv.org/abs/2108.07732. arXiv:2108.07732 [cs]. ↗ our card
  3. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models, March 2023. URL http: //arxiv.org/abs/2202.07646. arXiv:2202.07646 [cs]. find
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… find our card
  5. Rhiju Das, H Wayment-Steele, Do Soon Kim, Christian Choe, Bojan Tunguz, Walter Reade, and Maggie Demkin. Openvaccine: Covid-19 mrna vaccine degradation prediction, 2020. URL https://kaggle.com/competitions/stanford-covid-vaccine. ↗
  6. Jasper Dekoninck, Mark Niklas Müller, and Martin Vechev. ConStat: Performance-Based Contamination Detection in Large Language Models, May 2024. URL http://arxiv.org/abs/ 2405.16281. arXiv:2405.16281 [cs]. — [arXiv:2405.16281](https://arxiv.org/abs/2405.16281) ↗
  7. Thomas Dohmke. GitHub Copilot Workspace: Welcome to the Copilot-native developer environment, April 2024. URL https://github.blog/news-insights/product-news/ github-copilot-workspace/. factory.ai. ↗
  8. Code Droid Technical Report, June 2024. URL https://www.factory.ai/ news/code-droid-technical-report. ↗
  9. Luca Gioacchini, Giuseppe Siracusano, Davide Sanvito, Kiril Gashteovski, David Friede, Roberto Bifulco, and Carolin Lawrence. AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents, April 2024. URL http://arxiv.org/abs/2404. 06411. arXiv:2404.06411 [cs]. ↗
  10. Google DeepMind. Frontier Safety Framework, May 2024. find
  11. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring Coding Challenge Competence With APPS, November 2021. URL http://arxiv.org/abs/2105. 09938. arXiv:2105.09938 [cs]. ↗ our card
  12. Dong Huang, Jie M. Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agent-Coder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation, May 2024a. URL http://arxiv.org/abs/2312.13010. arXiv:2312.13010 [cs]. ↗
  13. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. In Forty-first International Conference on Machine Learning, June 2024b. URL https://openreview.net/forum?id= 1Fs1LvjYQW. ↗ our card
  14. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code, June 2024. URL http://arxiv.org/abs/ 2403.07974. arXiv:2403.07974 [cs]. ↗ our card
  15. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, April 2024. URL http://arxiv.org/abs/2310.06770. arXiv:2310.06770 [cs]. ↗ our card
  16. Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. DSBench: How Far Are Data Science Agents to Becoming Data Science Experts?, September 2024. URL http://arxiv.org/abs/2409.07703. arXiv:2409.07703 [cs]. ↗
  17. Kaggle. Kaggle Progression System \| Kaggle, 2024. URL https://www.kaggle.com/ progression. ↗
  18. Eirini Kalliamvakou. Research: quantifying GitHub Copilot’s impact on developer productivity and happiness, September 2022. URL https://github.blog/news-insights/research/ research-quantifying-github-copilots-impact-on-developer-productivity-and-happines Sayash Kapoor, Benedikt Stroebl, Zachary S. S… ↗
  19. AI Agents That Matter, July 2024. URL http://arxiv.org/abs/2407.01502. arXiv:2407.01502 [cs]. ↗
  20. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien De Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James M… find our card
  21. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as Age… find our card
  22. Alex Lourenco, Brent Seales, Christy Chapman, Daniel Havir, Ian Janicki, JP Posma, Nat Friedman, Ryan Holbrook, Seth P., Stephen Parsons, and Will Cukierski. Vesuvius challenge - ink detection, 2023. URL https://kaggle.com/competitions/ vesuvius-challenge-ink-detection. ↗
  23. Rien Maertens, Maarten Van Neyghem, Maxiem Geldhof, Charlotte Van Petegem, Niko Strijbol, Peter Dawyndt, and Bart Mesuere. Discovering and exploring cases of educational source code plagiarism with Dolos, 2024. URL https://github.com/dodona-edu/dolos. Publication Title: SoftwareX original-date: 2019… ↗
  24. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for General AI Assistants, November 2023. URL http: //arxiv.org/abs/2311.12983. arXiv:2311.12983 [cs]. find our card
  25. OpenAI. Preparedness Framework, December 2023. find
  26. Dominik Schmidt, Zhengyao Jiang, and Yuxiang Wu. Introducing Weco AIDE, April 2024. URL https://www.weco.ai/blog/technical-report. ↗
  27. Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gers… find
  28. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, a… find
  29. Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The shift from models to compound ai systems, 2024. URL http://bair.berkeley.edu/blog/2024/ 02/18/compound-ai-systems/. ↗
  30. Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. AutoCodeRover: Autonomous Program Improvement, July 2024. URL http://arxiv.org/abs/2404. 05427. arXiv:2404.05427 [cs]. — [arXiv:2404.05427](https://arxiv.org/abs/2404.05427) ↗
  31. Mingkai Zheng, Xiu Su, Shan You, Fei Wang, Chen Qian, Chang Xu, and Samuel Albanie. Can GPT-4 Perform Neural Architecture Search?, August 2023. URL http://arxiv.org/abs/ 2304.10970. arXiv:2304.10970 [cs]. ↗