📄 Training Software Engineering Agents and Verifiers with SWE-Gym

Summary

Open-weight software engineering (SWE) agents have lagged behind systems built on proprietary models, partly because no environment existed for training them on realistic repository-level work. The authors introduce SWE-Gym, which provides 2,438 real GitHub-issue tasks from 11 Python repositories. Each task includes an executable runtime with dependencies already installed, unit tests, and a natural-language problem statement. The repositories are disjoint from SWE-Bench to avoid contamination. The tests provide reward signals. The authors use these to fine-tune Qwen-2.5 coder models inside two scaffolds: OpenHands, by learning from successful sampled trajectories, and MoatlessTools, through self-improvement via rejection sampling. They also train verifiers, which are outcome-supervised reward models that rank candidate trajectories at inference time. Fine-tuning improves resolution rates by up to 19% absolute. Combining the agents with verifiers reaches 32.0% on SWE-Bench Verified and 26.0% on SWE-Bench Lite, a new best among open-weight agents. Performance keeps rising with more training trajectories and more rollouts, which suggests that sampling compute, not dataset size, is the current limit. The environment, models and trajectories are released publicly.

Key points

• Released resources:
  • SWE-Gym: 2,438 tasks.
  • SWE-Gym Lite: 230 easier, self-contained tasks for quick prototyping.
  • SWE-Gym Raw: 64,689 issues from 358 repositories, without executable environments.
• OpenHands result: A 32B Qwen-2.5 coder fine-tuned on only 491 trajectories improves to 15.3% on SWE-Bench Lite (+12.3) and 20.6% on SWE-Bench Verified (+13.6).
• MoatlessTools result: Self-improvement through rejection sampling fine-tuning reaches up to 19.7% on SWE-Bench Lite.
• Verifier scaling: Selecting the best of several rollouts gives roughly log-linear gains. In the MoatlessTools setting with k = 8, a 32B verifier lifts a 32B policy from 19.7% to 26.3% on Lite, while a 7B verifier stops improving after k = 4.
• Data scaling: Experiments that vary the number of trajectories, unique instances and repositories show no saturation, indicating that task and repository diversity are not yet bottlenecks.