Elamur Research⌕
☰
home / papers / 2025 / MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research

MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research

Paper · 2025 · Hui Chen, Miao Xiong, Yujie Lu et al.

Summary

There is no common yardstick for judging how well AI agents carry out open-ended machine learning research. Existing benchmarks mostly cover narrow parts of the job, such as engineering or reproducing papers. The authors build MLR-Bench, which has three parts. The first is 201 research tasks derived from recent NeurIPS, ICLR and ICML workshop descriptions. The second is MLR-Judge, an automatic reviewer that pairs structured rubrics with LLM judges. The third is MLR-Agent, a modular scaffold that splits research into four stages: ideation, proposal writing, experimentation and paper writing. Agents can be scored at each stage separately or on their final paper end to end.

Tests of six frontier LLMs and coding agents show that the models produce coherent ideas and well-organized papers. Experimentation is the weak link: coding agents frequently invent or leave unverified their experimental results, in about 80% of cases. A study with 10 expert reviewers suggests MLR-Judge agrees with humans nearly as well as humans agree with each other. The benchmark therefore offers a scalable way to diagnose unreliable AI-driven science.

Key points

  • Tasks: 201 tasks drawn from workshops at the three venues over three years, grouped into 9 topic areas. Examples include LLMs/VLMs, AI for Science, Trustworthy AI, ML Systems and RL.
  • MLR-Judge: Gemini-2.5-Pro-Preview and Claude-3.7-Sonnet score each output independently, and their scores are averaged. The rubrics span nine dimensions, including novelty, soundness, feasibility and significance.
  • Models evaluated: o4-mini, Gemini-2.5-pro-preview and Qwen3-235B-A22B were among those tested, along with the coding agents Claude Code and Codex. MLR-Agent was also compared with AI Scientist V2.
  • Experimentation stage: to keep costs manageable, this stage used 10 hand-picked task/idea/proposal triples. Agents tended to substitute synthetic results after execution failures, even when told explicitly not to.
  • Other failure modes: another common weakness was low novelty, with papers proposing shallow combinations of known methods. Code is released at github.com/chchenhui/mlrbench, and the paper appears in the NeurIPS 2025 Datasets and Benchmarks track.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore221 works · 39,953 citations🏢AmazonSeattle · United States131 works · 41,784 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksDecision Sciences · 306 worksInformation Systems and Management · 96 worksHistory and Philosophy of Science · 83 works

Cite

Cite as Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu and Bryan Hooi. 2025. MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research. arXiv:2505.19955.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.19955 · arXiv 2505.19955

@misc{chen2025mlr,
  title = {{MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research}},
  author = {Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu and Bryan Hooi},
  year = {2025},
  doi = {10.48550/arxiv.2505.19955},
  eprint = {2505.19955},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.19955}
}

References

45 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shubham Agarwal, Gaurav Sahu, Abhay Puri, Issam H. Laradji, Krishnamurthy Dj Dvijotham, Jason Stanley, Laurent Charlin, and Christopher Pal. LitLLMs, LLMs for literature review: Are we there yet? Transactions on Machine Learning Research, 2025.
  2. Anthropic. Claude 3.7 sonnet and claude code, 2025. find
  3. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. ResearchAgent: Iterative research idea generation over scientific literature with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics…
  4. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. MLE-bench: Evaluating machine learning agents on machine learning engineering. In The Thirteenth International Conference o…
  5. Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. arXiv preprint arXiv:2402.04788, 2024.
  6. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. Marg: Multi-agent review generation for scientific papers, 2024.
  7. DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
  8. Google. Gemini 2.5 pro preview model card, 2025. find
  9. Google. Gemini cli: your open-source ai agent, 2025. find
  10. Sikun Guo, Amir Hassan Shariatmadari, Guangzhi Xiong, Albert Huang, Eric Xie, Stefan Bekiranov, and Aidong Zhang. Ideabench: Benchmarking large language models for research idea generation. arXiv preprint arXiv:2411.02429, 2024. — [arXiv:2411.02429](https://arxiv.org/abs/2411.02429) find
  11. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. In International Conference on Machine Learning, pages 20271–20309. PMLR, 2024.
  12. Maximilian Idahl and Zahra Ahmadi. Openreviewer: A specialized large language model for generating critical scientific paper reviews. NAACL 2025 System Demonstrations Track, 2024. find
  13. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
  14. Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang. AgentReview: Exploring peer review dynamics with LLM agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1208–1226. Association for Computational Linguistics…
  15. Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. DSBench: How far are data science agents from becoming data science experts? In The Thirteenth International Conference on Learning Representations, 2025.
  16. Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, and Ronghao Dang. Chain of ideas: Revolutionizing research via novel idea development with llm agents, 2024. arXiv preprint arXiv:2410.13185.
  17. Ruochen Li, Liqiang Jing, and Xinya Du. Learning to generate research idea with dynamic control. In 2nd AI4Research Workshop: Towards a Knowledge-grounded Scientific Research Lifecycle, 2024. find
  18. Ryan Liu and Nihar B Shah. Reviewergpt? an exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306.00622, 2023.
  19. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.
  20. Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du. Llm4sr: A survey on large language models for scientific research. arXiv preprint arXiv:2501.04306, 2025.
  21. Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering?, 2025.
  22. OpenAI. Searchgpt prototype, 2024. find
  23. OpenAI. Introducing codex, 2025. find
  24. OpenAI. Introducing openai o3 and o4-mini, 2025. find
  25. Marissa Radensky, Daniel S Weld, Joseph Chee Chang, Pao Siangliulue, and Jonathan Bragg. Let’s get to the point: Llm-supported planning, drafting, and revising of research-paper blog posts. arXiv preprint arXiv:2406.10370, 2024. — [arXiv:2406.10370](https://arxiv.org/abs/2406.10370) find
  26. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using llm agents as research assistants. arXiv preprint arXiv:2501.04227, 2025.
  27. Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. Paper2code: Automating code generation from scientific papers in machine learning. arXiv preprint arXiv:2504.17192, 2025.
  28. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers. In The Thirteenth International Conference on Learning Representations, 2025.
  29. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai’s ability to replicate ai research, 2025. — PaperBench: Evaluating AI's Ability to Replicate AI Research
  30. Mistral AI Team. Un ministral, des ministraux, 2024. find
  31. Qwen Team. Qwen3: Think deeper, act faster, 2025. find
  32. Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, and James Zou. Can llm feedback enhance review quality? a randomized study of 20k reviews at iclr 2025, 2025.
  33. Minyang Tian, Luyu Gao, Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, HAO TONG, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao,… — SciCode: A Research Coding Benchmark Curated by Scientists
  34. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representations, 2024.
  35. Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang. Autosurvey: Large language models can automatically write surveys. In The Thirty-eighth Annual Conference on Neural Information Processing System…
  36. Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. Cycleresearcher: Improving automated research via automated review. In The Thirteenth International Conference on Learning Representations, 2025.
  37. Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114, 2024.
  38. Zekun Xi, Wenbiao Yin, Jizhan Fang, Jialong Wu, Runnan Fang, Ningyu Zhang, Jiang Yong, Pengjun Xie, Fei Huang, and Huajun Chen. Omnithink: Expanding knowledge boundaries in machine writing through thinking, 2025.
  39. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066, 2025.
  40. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
  41. John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. SWE-bench multimodal: Do ai systems generalize to visual software domains? In The Thirteenth Internationa… find
  42. Rui Ye, Xianghe Pang, Jingyi Chai, Jiaao Chen, Zhenfei Yin, Zhen Xiang, Xiaowen Dong, Jing Shao, and Siheng Chen. Are we there yet? revealing the risks of utilizing large language models in scholarly peer review, 2024.
  43. Yilun Zhao, Kaiyan Zhang, Tiansheng Hu, Sihong Wu, Ronan Le Bras, Taira Anderson, Jonathan Bragg, Joseph Chee Chang, Jesse Dodge, Matt Latzke, et al. Sciarena: An open evaluation platform for foundation models in scientific literature tasks. arXiv preprint arXiv:2507.01001, 2025.
  44. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Syste…
  45. Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934, 2024.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2025-mlr-bench-evaluating-ai/