📄 Towards Execution-Grounded Automated AI Research

Summary

LLMs can propose research ideas that sound convincing but fail when tested. This paper asks two questions: can idea testing be automated at scale, and can models improve from the measured results? The authors built an automated executor. An Implementer turns natural-language ideas into code diffs, a Scheduler assigns GPU resources, and Workers run the jobs and log the metrics. They tested it on two open-ended environments: speeding up nanoGPT pre-training and improving GRPO post-training for math reasoning. Most ideas from frontier LLMs could be implemented and run automatically.

Using execution results to drive evolutionary search worked well with few samples. Within ten epochs it beat both baselines, and on post-training it slightly edged out the best human solution. RL from execution reward had a different outcome. It raised the ideator's average score but not its best score, because the model collapsed onto a few simple ideas. The work shows that execution-grounded automated research is feasible and identifies its current limits.

Key points

• Environments:
  • nanoGPT: train a 124M GPT-2 on FineWeb to 3.28 validation loss on 8 H100s, measured as wall-clock time.
  • GRPO: finetune Qwen2.5-Math-1.5B on MATH, measured as validation accuracy.
  • Both include safeguards against reward hacking, such as frozen evaluation code.
• Execution rates: over 90% of ideas ran successfully in the pre-training environment with Claude-4.5-Sonnet and Claude-4.5-Opus.
• Evolutionary search results:
  • Post-training accuracy reached 69.4% against the 48.0% GRPO baseline, compared with 68.8% for the best human expert.
  • Pre-training time dropped to 19.7 min from the 35.9 min nanoGPT baseline, compared with 2.1 min for the best human expert.
  • Search beat best-of-N under the same sampling budget.
• Scaling: only Claude-4.5-Opus showed a clear scaling trend; Claude-4.5-Sonnet and GPT-5 plateaued early.
• RL with Qwen3-30B: mean reward rose but max reward did not, and thinking length and idea diversity collapsed. Environments and trajectories are released in the GitHub repository NoviScl/Automated-AI-Researcher.