Elamur Research
☰
home / papers / 2023 / MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation

MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2310.03302

Summary

MLAgentBench asks whether agents built on large language models can run machine learning experiments on their own: design an approach, write and execute code, read the results, and iterate. The authors built a benchmark of 13 tasks. Each task comes with a goal description, starter files (data and usually a baseline script), and an automatic evaluator that scores the agent's final submission. The tasks range from classic problems such as CIFAR-10 to recent Kaggle competitions, research problems like BabyLM, and code-speedup challenges. Agents work in a sandboxed workspace using file and execution actions. They count as successful if they beat the baseline metric by at least 10%. The authors also designed a ReAct-style agent that, at each step, writes a reflection, a plan, a fact check and its reasoning before choosing an action. They tested it with several LLM backends. Claude v3 Opus performed best, but results depend heavily on how new the task is, with near-total failure on recent challenges. This shows that autonomous ML research remains unreliable, mainly because of weak long-horizon planning and hallucination.

Key points

  • Task suite: 13 tasks in five groups:
    • canonical: CIFAR-10, imdb, ogbn-arxiv;
    • classic Kaggle: house-price, spaceship-titanic;
    • recent Kaggle: parkinsons-disease, fathomnet, feedback, identify-contrails;
    • research: CLRS, BabyLM;
    • code speedup: llama-inference, vectorization.
  • Action space: the environment offers primitive actions (list, read, write, copy and execute files, undo edits) and compound LM-driven actions (Understand File, Edit Script, Edit Script Segment).
  • Models compared: GPT-4, GPT-4-turbo, Claude v1.0, v2.1 and v3 Opus, Gemini-Pro, and Mixtral.
    • Claude v3 Opus reached a 37.5% average success rate.
    • The best agents succeeded in 100% of 8 runs on house-price but only 0–25% on recent Kaggle tasks and BabyLM.
  • Efficiency: GPT-4-turbo used about 51% fewer tokens than the average agent. A full benchmark pass with it took roughly 6M tokens, about $60, at around 26% average success.
  • Availability: code is released at github.com/snap-stanford/MLAgentBench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

128 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 121 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.03302 · arXiv 2310.03302

@misc{huang2023mlagentbench,
  title = {{MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation}},
  author = {Huang, Qian and Vora, Jian and Liang, Percy and Leskovec, Jure},
  year = {2023},
  doi = {10.48550/arxiv.2310.03302},
  eprint = {2310.03302},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.03302}
}

References

40 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anna Montoya, D. House prices - advanced regression techniques, 2016. URL https://kaggle.com/ competitions/house-prices-advanced-regression-techniques. ↗
  2. Anthropic. Introducing claude, 2023. https://www.anthropic.com/index/ introducing-claude. ↗
  3. URL Berens, P., Cranmer, K., Lawrence, N. D., von Luxburg, U., and Montgomery, J. Ai for science: An emerging agenda. ArXiv, abs/2303.04217, 2023. — [arXiv:2303.04217](https://arxiv.org/abs/2303.04217) find
  4. Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax. ↗
  5. Chollet, F. et al. Keras, 2015. URL https:// github.com/fchollet/keras. find
  6. Elsken, T., Metzen, J. H., and Hutter, F. Neural architecture search: a survey. J. Mach. Learn. Res., 20(1):1997–2017, jan 2019. ISSN 1532-4435. find
  7. Franklin, A., Maggie, Benner, M., Rambis, N., Baffour, P., Holbrook, R., Crossley, S., and ulrichboser. Feedback prize - english language learning, 2022. URL https: //kaggle.com/competitions/feedback-prize-english-language-learning. find
  8. H., Burget, J., Ho, A., Barnes, E., and Christiano, P. F. Evaluating language-model agents on realistic autonomous tasks. ArXiv, abs/2312.11671, 2023. URL https://api.semanticscholar.org/ CorpusID:260472392. — [arXiv:2312.11671](https://arxiv.org/abs/2312.11671) ↗
  9. He, X., Zhao, K., and Chu, X. Automl: A survey of the state-of-the-art. Knowledge-Based Systems, 212:106622, 2021. ISSN 0950-7051. doi: https://doi.org/10.1016/j.knosys.2020.106622. URL https://www.sciencedirect.com/science/ article/pii/S0950705120307516. — [doi:10.1016/j.knosys.2020.106622](https://doi.org/10.1016/j.knosys.2020.106622) ↗
  10. Kirsch, L., Dane, S., Adam, S., and Dardov, V. Amp®-parkinson’s disease progression prediction, 2023. URL https://kaggle.com/competitions/ amp-parkinsons-disease-progression-prediction. ↗
  11. Kitano, H. Nobel turing challenge: creating the engine for scientific discovery. NPJ Systems Biology and Applications, 7, 2021. find
  12. Howard, A., Chow, A., and Holbrook, R. Spaceship titanic, 2022. URL https://kaggle.com/ competitions/spaceship-titanic. ↗
  13. Kramer, S., Cerrato, M., Dzeroski, S., and King, R. D. Automated scientific discovery: From equation discovery to autonomous discovery systems. ArXiv, abs/2305.02251, 2023. find
  14. Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovec, J. Open graph benchmark: Datasets for machine learning on graphs. ArXiv, abs/2005.00687, 2020. find
  15. Krizhevsky, A. Learning multiple layers of features from tiny images. 2009. find
  16. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de Las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., La… find
  17. Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., Zhang, S., Deng, X., Zeng, A., Du, Z., Zhang, C., Shen, S., Zhang, T., Su, Y., Sun, H., Huang, M., Dong, Y., and Tang, J. Agentbench: Evaluating LLMs as agents. In The Twelfth International Conference on Learn… find our card
  18. Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150, Portland, Oregon, USA, June 2011. Association… find our card
  19. Jumper, J. M., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zı́dek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Ballard, A., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D. A., C… find our card
  20. Nakano, R., Hilton, J., Balaji, S., Wu, J., Long, O., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. Webgpt: Browser-assisted question-answering with human feedback. ArXiv, abs/2112.09332,… find our card
  21. King, R. D., Whelan, K. E., Jones, F. M., Reiser, P. G. K., Bryant, C. H., Muggleton, S. H., Kell, D. B., and Oliver, S. G. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature, 427:247– 252, 2004. find
  22. King, R. D., Rowland, J. J., Oliver, S. G., Young, M., Aubrey, W., Byrne, E., Liakata, M., Markham, M., Pir, P., Soldatova, L. N., Sparkes, A., Whelan, K. E., and Clare, A. The automation of science. Science, 324:85 – 89, 2009. find
  23. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. find our card
  24. Park, J. S., O’Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Kinniment, M., Sato, L. J. K., Du, H., Goodrich, B., Hasin, M., Chan, L., Mi… find
  25. URL https://api.semanticscholar.org/ CorpusID:249210177. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L. J., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models. ArXiv, abs/2305.16291, 2023a. ↗
  26. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-perfor… find our card
  27. Wang, Q., Downey, D., Ji, H., and Hope, T. Scimon: Scientific inspiration machines optimized for novelty. arXiv preprint arXiv:2305.14259, 2023b. find
  28. Warstadt, A., Choshen, L., Mueller, A., Williams, A., Wilcox, E. G., and Zhuang, C. Call for papers - the babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus. ArXiv, abs/2301.11796, 2023. find
  29. Sarna, A., Elkin, C., inversion, Ng, J., Maggie, and Reade, W. Google research - identify contrails to reduce global warming, 2023. URL https://kaggle.com/competitions/ google-research-identify-contrails-reduce-global-warming. ↗
  30. Woodward, B., eor123, GenevievePatterson, and Carlsen, L. Fathomnet 2023, 2023. URL https://kaggle.com/competitions/ fathomnet-out-of-sample-detection. ↗
  31. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. find our card
  32. Schick, T., Dwivedi-Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/ forum?id… ↗ our card
  33. Zhang, L., Zhang, Y., Ren, K., Li, D., and Yang, Y. Mlcopilot: Unleashing the power of large language models in solving machine learning tasks. ArXiv, abs/2304.14979, 2023a. URL https: //api.semanticscholar.org/CorpusID: 258418182. find
  34. Schwaller, P., Gaudin, T., Lanyi, D., Bekas, C., and Laino, T. “found in translation”: predicting outcomes of complex organic chemistry reactions using neural sequence-to-sequence models† †electronic supplementary information (esi) available: Time-split test set and example predictions, together wit… find
  35. Zhang, M., Qamar, M., Kang, T., Jung, Y., Zhang, C., Bae, S.-H., and Zhang, C. A survey on graph diffusion models: Generative ai in science for molecule, protein and material. ArXiv, abs/2304.01565, 2023b. — [arXiv:2304.01565](https://arxiv.org/abs/2304.01565) find
  36. Zhang, S., Gong, C., Wu, L., Liu, X., and Zhou, M. Automl-gpt: Automatic machine learning with gpt. ArXiv, abs/2305.02499, 2023c. URL https: //api.semanticscholar.org/CorpusID: 258480269. find
  37. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openreview.net/forum?id=vAElhFcKW6. find our card
  38. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., and Neubig, G. Webarena: A realistic web environment for building autonomous agents. ArXiv, abs/2307.13854, 2023. URL https://api.semanticscholar.org/ CorpusID:260164780. ↗ our card
  39. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971, 2023. find
  40. Velivckovi’c, P., Badia, A. P., Budden, D., Pascanu, R., Banino, A., Dashevskiy, M., Hadsell, R., and Blundell, C. The clrs algorithmic reasoning benchmark. In International Conference on Machine Learning, 2022. find