Elamur Research⌕
☰
home / papers / 2025 / AIDE: AI-Driven Exploration in the Space of Code

AIDE: AI-Driven Exploration in the Space of Code

Paper · 2025 · Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth et al.

Summary

Developing strong machine learning models depends heavily on repetitive trial-and-error. Classic automation methods such as AutoML, NAS and hyperparameter search also require a hand-designed configuration space. This paper presents AIDE, an LLM-based agent from Weco AI that treats ML engineering as the optimization of code itself. ReAct-style agents keep appending to an ever-growing interaction history. AIDE instead stores every candidate script as a node in a solution tree and scores each one with a stateless evaluator, such as a validation metric. A fixed heuristic policy then decides whether to write a fresh draft, repair a failing script, or apply one atomic change to the current best solution. A summarization step condenses earlier attempts, and a small data preview is added to prompts, so context stays compact. AIDE was tested on the authors' Kaggle tabular tasks, OpenAI's MLE-Bench and METR's RE-Bench. It reached state-of-the-art results and sometimes outperformed human experts under time limits. This shows that structured search over code can turn extra compute into better ML solutions. The implementation is open source.

Key points

  • The core loop has four components: a search policy π (draft / debug / improve), a coding operator f, a summarization operator Σ(T), and an evaluator h that scores each script independently. Code is available at github.com/WecoAI/aideml.
  • On MLE-Bench, AIDE earned roughly twice as many medals as a later competing agent when both used GPT-4o, and the gap grew further with o1-preview.
  • On the MLE-Bench Lite subset with o1-preview, adding AIDE improved results as follows (all p < 0.01):
    • valid submissions rose from 63.6% to 92.4%;
    • above-median-human scores rose from 13.6% to 59.1%;
    • gold medals rose from 6.1% to 21.2%;
    • any medal rose from 7.6% to 36.4%.
  • On RE-Bench, METR compared AIDE + o1-preview with top human researchers across 7 AI R&D tasks:
    • AIDE scored higher within a six-hour budget.
    • In Optimize a Kernel, it found a custom Triton solution faster than any of nine experts working up to 64 hours.
  • Limitations: the greedy policy can get stuck in local optima, so humans eventually caught up. AIDE also struggled with large codebases and multi-step changes, for example repeating local patches in Agent for Rust CodeContests.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 works

Cite

Cite as Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko and Yuxiang Wu. 2025. AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.13138 · arXiv 2502.13138

@misc{jiang2025aide,
  title = {{AIDE: AI-Driven Exploration in the Space of Code}},
  author = {Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko and Yuxiang Wu},
  year = {2025},
  doi = {10.48550/arxiv.2502.13138},
  eprint = {2502.13138},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.13138}
}

References

13 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Claude 3.5 sonnet model card addendum. Technical report, Anthropic, 2024. URL https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/ Model_Card_Claude_3_Addendum.pdf. ↗
  2. J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. M adry. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. arXiv preprint arXiv:2410.07095, 2024.
  3. T. Elsken, J. H. Metzen, and F. Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1–21, 2019. find
  4. S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and Efficient Hyperparameter Optimization at Scale. In Proc. of ICML, volume 35, 2018. find
  5. M. Feurer, A. Klein, K. Eggensperger, J. T. Springenberg, M. Blum, and F. Hutter. Efficient and Robust Automated Machine Learning. In Proc. of NeurIPS, volume 28, 2015. find
  6. M. Feurer, K. Eggensperger, E. Bergman, F. Pfisterer, B. Bischl, and F. Hutter. Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning. arXiv preprint arXiv:2007.04074, 2020. — [arXiv:2007.04074](https://arxiv.org/abs/2007.04074) find
  7. Google. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
  8. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning, 2024.
  9. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 20…
  10. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. ↗
  11. H. Jin, Q. Song, and X. Hu. Auto-Keras: An Efficient Neural Architecture Search System. In Proc. of ACM SIGKDD (KDD), 2019. find
  12. H. Jin, F. Chollet, Q. Song, and X. Hu. AutoKeras: An AutoML Library for Deep Learning. Journal of Machine Learning Research, 24(6):1–7, 2023. find
  13. Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2):99–134, 1998. E. LeDell and S. Poirier. H2O AutoML: Scalable Automatic Machine Learning. In Proc. of the AutoML Conference, pages 1–8,… — Competition-Level Code Generation with AlphaCode

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jiang-2025-aide-ai-driven-exploration/