Elamur Research⌕
☰
home / papers / 2025 / Evaluating Agent-based Program Repair at Google

Evaluating Agent-based Program Repair at Google

Paper · 2025 · Pat Rondon, Renyao Wei, Jose Cambronero et al.

Summary

LLM-based repair agents have mostly been built and tested on open-source GitHub Python bugs, especially SWE-Bench. It is unclear whether they work inside a large company. The authors build GITS-Eval, a benchmark of 178 fixed bugs from Google's internal issue tracker (GITS). Developers filed 78 of them, and automated tools filed 100: sanitizers (SAN) and a test-order-dependency analyzer (TOD). A multi-phase filtering funnel keeps only bugs with small patches, no multimedia, and reliable, non-flaky tests that expose each bug. It also drops fixes that need "magic constants" the agent could not infer. A separate sample of 2,000 GITS bugs differs from SWE-Bench in language mix, patch size and spread, and how often code terms appear in reports. As a baseline, the authors implement Passerine, a SWE-Agent-style agent adapted to Google's monorepo and tooling. It runs Gemini 1.5 Pro with 20 trajectories per bug. Passerine produces test-passing patches for most machine-reported bugs and about a quarter of human-reported ones. Manual review finds fewer patches equivalent to the real fix. A simple agent already fixes real enterprise bugs, but SWE-Bench scores may not transfer.

Key points

  • GITS-Eval has 178 bugs: 78 human-reported, 50 SAN and 50 TOD. The fix patches date from June 26 to August 30, 2024, and filtering began with about 80k fixed bugs. Patches are limited to under 150 lines.
  • Plausible-patch rates (pass bug tests) are 73% for machine-reported bugs (68% TOD, 78% SAN) and 25.6% for human-reported bugs.
  • After manual annotation, at least one patch matched the ground-truth fix semantically for 43% of machine-reported bugs (24% TOD, 62% SAN) and 17.9% of human-reported ones.
  • Passerine changes its command patterns depending on bug type and benefits from detailed bug reports. This suggests bug reports could be designed with agents in mind.
  • Analyzing the agent's runs shows room for efficiency gains, for example by cutting off degenerate trajectories early. The authors make no claims about performance on SWE-Bench or SWE-Bench-Lite.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksInformation Systems · 1,888 worksSoftware · 150 worksComputer Networks and Communications · 115 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 worksCode & programming languages · 194 works

Cite

Cite as Pat Rondon, Renyao Wei, Jose Cambronero, Jurgen Cito, Aaron Sun, Siddhant Sanyam, Michele Tufano and Satish Chandra. 2025. Evaluating Agent-based Program Repair at Google. arXiv:2501.07531.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.07531 · arXiv 2501.07531

@misc{rondon2025evaluating,
  title = {{Evaluating Agent-based Program Repair at Google}},
  author = {Pat Rondon, Renyao Wei, Jose Cambronero, Jurgen Cito, Aaron Sun, Siddhant Sanyam, Michele Tufano and Satish Chandra},
  year = {2025},
  doi = {10.48550/arxiv.2501.07531},
  eprint = {2501.07531},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.07531}
}

References

21 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Avail…
  2. J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,” arXiv preprint arXiv:2405.15793, 2024.
  3. Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1592–1604. find
  4. I. Bouzenia, P. Devanbu, and M. Pradel, “Repairagent: An autonomous, llm-based agent for program repair,” arXiv preprint arXiv:2403.17134, 2024. — [arXiv:2403.17134](https://arxiv.org/abs/2403.17134) find
  5. D. Chen, S. Lin, M. Zeng, D. Zan, J.-G. Wang, A. Cheshkov, J. Sun, H. Yu, G. Dong, A. Aliev et al., “Coder: Issue resolving with multi-agent and task graphs,” arXiv preprint arXiv:2406.01304, 2024.
  6. M. Tufano, A. Agarwal, J. Jang, R. Z. Moghaddam, and N. Sundaresan, “Autodev: Automated ai-driven development,” arXiv preprint arXiv:2403.08299, 2024. — [arXiv:2403.08299](https://arxiv.org/abs/2403.08299) find
  7. X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh et al., “Opendevin: An open platform for ai software developers as generalist agents,” arXiv preprint arXiv:2407.16741, 2024.
  8. H. Ruan, Y. Zhang, and A. Roychoudhury, “Specrover: Code intent extraction via llms,” arXiv preprint arXiv:2408.02232, 2024. — [arXiv:2408.02232](https://arxiv.org/abs/2408.02232) find
  9. Y. Liu, P. Gao, X. Wang, C. Peng, and Z. Zhang, “Marscode agent: Ai-native automated bug fixing,” arXiv preprint arXiv:2409.00899, 2024. — [arXiv:2409.00899](https://arxiv.org/abs/2409.00899) find
  10. R. Potvin and J. Levenberg, “Why google stores billions of lines of code in a single repository,” Communications of the ACM, vol. 59, no. 7, pp. 78–87, 2016. find
  11. Bazel, “Distributed Builds,” https://bazel.build/basics/distributed-builds, 2024, [Online; accessed October 2024]. ↗
  12. C. Sadowski, K. T. Stolee, and S. Elbaum, “How developers search for code: a case study,” in Proceedings of the 2015 10th joint meeting on foundations of software engineering, 2015, pp. 191–201. find
  13. S. Saha et al., “Harnessing evolution for multi-hunk program repair,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 13–24. find
  14. V. Sobreira, T. Durieux, F. Madeiral, M. Monperrus, and M. de Almeida Maia, “Dissection of a bug dataset: Anatomy of 395 patches from defects4j,” in 2018 IEEE 25th international conference on software analysis, evolution and reengineering (SANER). IEEE, 2018, pp. 130–140. find
  15. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. [Online]. Available: https://op…
  16. X. Wang, Y. Chen, L. Yuan, Y. Zhang, Y. Li, H. Peng, and H. Ji, “Executable code actions elicit better llm agents,” in Proceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2025.
  17. A. Greenberg, “Building a distributed build system at google scale,” https://gotocon.com/dl/goto-chicago-2016/slides/AysyluGreenberg BuildingADistributedBuildSystemAtGoogleScale.pdf, 2016, [Online; accessed October 2024]. ↗
  18. K. Wang, D. Rall, G. Tener, V. Gullapalli, X. Huang, and A. Gad, “Smart build targets batching service at Google,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 2021, pp. 160–169. find
  19. Z. Qi, F. Long, S. Achour, and M. Rinard, “An analysis of patch plausibility and correctness for generate-and-validate patch generation systems,” in Proceedings of the 2015 International Symposium on Software Testing and Analysis, 2015, pp. 24–36. find
  20. Y. Xiong, X. Liu, M. Zeng, L. Zhang, and G. Huang, “Identifying patch correctness in test-based program repair,” in Proceedings of the 40th international conference on software engineering, 2018, pp. 789–799. find
  21. B. Yang and J. Yang, “Exploring the differences between plausible and correct patches at fine-grained level,” in 2020 IEEE 2nd International Workshop on Intelligent Bug Fixing (IBF). IEEE, 2020, pp. 1–8. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rondon-2025-evaluating-agent-based-program-repair/