Elamur Research⌕
☰
home / papers / 2024 / SWE-Bench+: Enhanced Coding Benchmark for LLMs

SWE-Bench+: Enhanced Coding Benchmark for LLMs

Paper · 2024 · Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer et al.

Summary

SWE-bench, a collection of 2,294 real GitHub issues drawn from 12 popular Python repositories, is widely used to judge how well LLM-based agents repair software. However, nobody had checked how reliable it is. The authors studied SWE-Agent + GPT-4, the leaderboard leader at the time. They kept the 251 of its patches whose logs confirmed that every test passed. Three authors then independently compared each one with the developer-written gold patch, also reviewing issue texts, hints, logs and agent trajectories.

Almost two thirds of these apparent successes proved questionable. Either the fix was already spelled out in the issue discussion, or the tests were too lax to reject wrong, partial or misplaced edits. The same flaws appear in SWE-bench Lite and SWE-bench Verified, and nearly all issues predate model training cutoffs.

To fix this, the authors built SWE-Bench+ from issues opened after the newest model's cutoff, excluding any whose reports reveal the answer. On the new set, most agents score far lower, which suggests that earlier leaderboard figures overstate real issue-solving ability.

Key points

  • Of 286 instances listed as resolved by SWE-Agent + GPT-4, 251 had logs showing all tests passed. Of these, 82 (32.67%) involved solution leakage and 78 (31.08%) passed only because of weak tests (incorrect, incomplete, or applied to other files/functions).
  • After removing suspicious fixes, the resolution rate of SWE-Agent + GPT-4 on SWE-bench Full falls from 12.47% to 3.97%. Over 94% of issues were created before LLM knowledge cutoffs.
  • SWE-Bench+ gathers issues created between 2023-11-01 and 2024-08-22, after the cutoffs of GPT-3.5, GPT-4 and GPT-4o. It contains no solution-leak cases and has the lowest share of weak tests among SWE-bench variants.
  • Resolution rates on SWE-Bench+:
    • SWE-Agent + GPT-4: 0.55%
    • SWE-RAG + GPT-4: 0.73% (1.31% on the original leaderboard)
    • AutoCodeRover + GPT-4o: 3.83% (18.83% on the original leaderboard)
    • SWE-RAG + GPT-3.5: 0.55%, which is slightly above its original 0.17%
  • The dataset is released on Zenodo (record 13,879,453). The authors also recommend reporting cost alongside accuracy; for example, AutoCodeRover costs about $12.61 per fixed issue.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of YorkYork · United Kingdom19 works · 6,587 citations🎓York UniversityToronto · Canada18 works · 3,378 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksInformation Systems · 1,888 worksSoftware · 150 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksEvaluation & benchmarks · 1,394 works

Cite

Cite as Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin and Song Wang. 2024. SWE-Bench+: Enhanced Coding Benchmark for LLMs. arXiv:2410.06992.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.06992 · arXiv 2410.06992

@misc{aleithan2024swe,
  title = {{SWE-Bench+: Enhanced Coding Benchmark for LLMs}},
  author = {Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin and Song Wang},
  year = {2024},
  doi = {10.48550/arxiv.2410.06992},
  eprint = {2410.06992},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.06992}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kamel Alrashedy and Ahmed Binjahlan. Language models are better bug detector through code-pair classification, 2024. URL https://arxiv.org/abs/2311.07957. — [arXiv:2311.07957](https://arxiv.org/abs/2311.07957) ↗
  2. Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, Jie Wang, Xiao Cheng, Guangtai Liang, Yuchi Ma, Pan Bian, Tao Xie, and Qianxiang Wang. Coder: Issue resolving with multi-agent and task graphs, 2024. URL https://arxiv.org/a… ↗
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  4. David de Fitero-Dominguez, Eva Garcia-Lopez, Antonio Garcia-Cabot, and Jose-Javier Martinez-Herraiz. Enhanced automated code vulnerability repair using large language models. Engineering Applications of Artificial Intelligence, 138:109291, 2024. ISSN 0952-1976. doi: https://doi.org/10.1016/j.engappa… — [doi:10.1016/j.engappai.2024.109291](https://doi.org/10.1016/j.engappai.2024.109291) ↗
  5. Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models, 2023a. URL https://arxiv.org/abs/2212.14834. — [arXiv:2212.14834](https://arxiv.org/abs/2212.14834) ↗
  6. Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt, 2023b. URL https://arxiv.org/abs/2304.02014. — [arXiv:2304.02014](https://arxiv.org/abs/2304.02014) ↗
  7. Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Evaluating large language models in class-level code generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24, New York, N… — [doi:10.1145/3597503.3639219](https://doi.org/10.1145/3597503.3639219) find
  8. Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. Large language models for software engineering: Survey and open problems, 2023. URL https://arxiv.org/abs/2310.03533. — [arXiv:2310.03533](https://arxiv.org/abs/2310.03533) ↗
  9. Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Nguyen, and Omer Tripp. A deep dive into large language models for automated bug localization and repair. Proc. ACM Softw. Eng., 1(FSE), July 2024. doi: 10.1145/3660773. URL https://doi.org/10.1145/3660773. — [doi:10.1145/3660773](https://doi.org/10.1145/3660773) ↗
  10. Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic literature review, 2024. URL https://arxiv.org/abs/2308.10620. — [arXiv:2308.10620](https://arxiv.org/abs/2308.10620) ↗
  11. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation, 2024. URL https://arxiv.org/abs/2406.00515. ↗
  12. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL https://arxiv.org/abs/2310.06770. ↗
  13. Xue Li and Till Döhmen. Towards efficient data wrangling with llms using code generation. In Proceedings of the Eighth Workshop on Data Management for End-to-End Machine Learning, DEEM ’24, pp. 62–66, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706110. doi: 10.1145/3650… — [doi:10.1145/3650203.3663334](https://doi.org/10.1145/3650203.3663334) find
  14. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James M… — Competition-Level Code Generation with AlphaCode
  15. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024.
  16. Hanbin Luo, Jianxin Wu, Jiajing Liu, and Maxwell Fordjour Antwi-Afari. Large language model-based code generation for the control of construction assembly robots: A hierarchical generation approach. Developments in the Built Environment, 19:100488, 2024. ISSN 2666-1659. doi: https://doi.org/10.1016/… — [doi:10.1016/j.dibe.2024.100488](https://doi.org/10.1016/j.dibe.2024.100488) find
  17. Ricardo La Rosa, Corey Hulse, and Bangdi Liu. Can github issues be solved with tree of thoughts?, 2024. URL https://arxiv.org/abs/2405.13057. — [arXiv:2405.13057](https://arxiv.org/abs/2405.13057) ↗
  18. Lin Shi, Fangwen Mu, Xiao Chen, Song Wang, Junjie Wang, Ye Yang, Ge Li, Xin Xia, and Qing Wang. Are we building on the rock? on the importance of data preprocessing for code summarization. pp. 107–119, 2022. find
  19. Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. Magis: Llm-based multi-agent framework for github issue resolution, 2024. URL https://arxiv.org/abs/2403.17927. — [arXiv:2403.17927](https://arxiv.org/abs/2403.17927) ↗
  20. Chunqiu Steven Xia and Lingming Zhang. Revisiting automated program repair via https://arxiv.org/abs/2207.08281. Less training, more repairing please: zero-shot learning, 2022. URL Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated program repair in the era of large pre-trained language… ↗
  21. IEEE Press, 2023. ISBN 9781665457019. doi: 10.1109/ICSE48619.2023.00129. URL https://doi.org/10.1109/ICSE48619.2023.00129. — [doi:10.1109/icse48619.2023.00129](https://doi.org/10.1109/icse48619.2023.00129) ↗
  22. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents, 2024. URL https://arxiv.org/abs/2407.01489. ↗
  23. Boyang Yang, Haoye Tian, Weiguo Pian, Haoran Yu, Haitao Wang, Jacques Klein, Tegawendé F. Bissyandé, and Shunfu Jin. Cref: An llm-based conversational software repair framework for programming tutors. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, IS… — [doi:10.1145/3650212.3680328](https://doi.org/10.1145/3650212.3680328) find
  24. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024b. URL https://arxiv.org/abs/2405.15793. ↗
  25. Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, and Jian-Guang Lou. Large language models meet nl2code: A survey, 2023. URL https://arxiv.org/abs/2212.09420. ↗
  26. Daoguang Zan, Zhirong Huang, Ailun Yu, Shaoxin Lin, Yifan Shi, Wei Liu, Dong Chen, Zongshuai Qi, Hao Yu, Lei Yu, Dezhi Ran, Muhan Zeng, Bo Shen, Pan Bian, Guangtai Liang, Bei Guan, Pengjie Huang, Tao Xie, Yongji Wang, and Qianxiang Wang. Swe-bench-java: A github issue resolving benchmark for java, 2… — [arXiv:2408.14354](https://arxiv.org/abs/2408.14354) find
  27. Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh Murthy, Tian Lan, Lei Li, Renze Lou, Jiacheng Xu, Bo Pang, Yingbo Zhou, Shelby Heinecke, Silvio Savarese, Huan Wang, and Caiming Xiong. Diversity empowers intelligence: Integrating expertise of software engineering agents, 2024a. UR… — [arXiv:2408.07060](https://arxiv.org/abs/2408.07060) find
  28. Quanjun Zhang, Tongke Zhang, Juan Zhai, Chunrong Fang, Bowen Yu, Weisong Sun, and Zhenyu Chen. A critical review of large language model on software engineering: An example from chatgpt and automated program repair, 2024b. URL https://arxiv.org/abs/2310.08879. — [arXiv:2310.08879](https://arxiv.org/abs/2310.08879) ↗
  29. Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement, 2024c. URL https://arxiv.org/abs/2404.05427. — [arXiv:2404.05427](https://arxiv.org/abs/2404.05427) ↗
  30. Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen. A survey of large language models for code: Evolution, benchmarking, and future trends, 2024. URL https://arxiv.org/abs/2311.10372. — [arXiv:2311.10372](https://arxiv.org/abs/2311.10372) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aleithan-2024-swe-bench-enhanced-coding/