Elamur Research⌕
☰
home / papers / 2025 / The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason

The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason

Paper · 2025 · Shanchao Liang, Spandan Garg and Roshanak Zilouchian Moghaddam

Summary

The paper asks whether the high scores of recent LLMs on SWE-Bench Verified, a popular benchmark of real GitHub issue resolution, show real coding skill or recall of public repositories seen during training. The authors design diagnostic probes that deliberately withhold the information needed to solve a task honestly:

  • File-path identification: guess the buggy file from the issue text and repository name only.
  • Function reproduction: regenerate the patched functions after they are removed from the file.
  • Prefix completion: continue a solution verbatim from its opening code.

They run ten models from Anthropic and OpenAI on several comparable benchmarks and treat score gaps between similar tasks as a sign of memorization, since true skill should transfer evenly. Models locate files and reproduce fixed code far better on SWE-Bench data than on external or newer tasks. This suggests contamination inflates reported progress. The authors argue the field needs time-controlled, cross-repository, contamination-resistant evaluation, and their method requires no access to training data or model internals.

Key points

  • With no repository access, OpenAI o3 picks the correct buggy file for up to 76% of SWE-Bench Verified issues (500 instances from 12 Python repos). On repositories outside SWE-Bench, accuracy reaches at most 53%, with drops of up to 47 percentage points.
  • In function reproduction, consecutive 5-gram overlap with the ground truth reaches up to 35% on SWE-Bench Verified and Full SWE-Bench. On other benchmarks it stays at or below 18%.
  • The comparison sets are Full SWE-Bench, SWE-Bench Extra, SWE-Bench C#, RefactorBench, and outside-repo tasks. Models tested include the Claude 3.5–4 family, GPT-4o, GPT-4.1, o3-mini and o3.
  • Model outputs resemble the patched code more than the buggy code on SWE-Bench Verified and SWE-Bench C#. On SWE-Bench Extra, whose tasks are newer than the models, this tendency vanishes or reverses.
  • The authors identify two memorization types: instance-specific memorization and repository-bias memorization. They also report a filtered accuracy that excludes issues explicitly mentioning file paths.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States404 works · 163,863 citations🎓Purdue University West LafayetteWest Lafayette · United States28 works · 2,141 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 worksSoftware · 150 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Shanchao Liang, Spandan Garg and Roshanak Zilouchian Moghaddam. 2025. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason. arXiv:2506.12286.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2506.12286 · arXiv 2506.12286

@misc{liang2025swe,
  title = {{The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason}},
  author = {Shanchao Liang, Spandan Garg and Roshanak Zilouchian Moghaddam},
  year = {2025},
  doi = {10.48550/arxiv.2506.12286},
  eprint = {2506.12286},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.12286}
}

References

22 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Introducing Claude 4. Anthropic PBC. https://www.anthropic.com/news/claude-4 Blog post. ↗
  2. Samuel R. Bowman and George E. Dahl. 2021. What Will it Take to Fix Benchmarking in Natural Language Understanding? arXiv:2104.02145 [cs.CL] https://arxiv.org/abs/2104.02145 ↗
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  4. Wentao Chen, Lizhe Zhang, Li Zhong, Letian Peng, Zilong Wang, and Jingbo Shang. 2025. Memorize or Generalize? Evaluating LLM Code Generation with Evolved Questions. arXiv:2503.02296 [cs.AI] https://arxiv.org/abs/2503.02296 — [arXiv:2503.02296](https://arxiv.org/abs/2503.02296) ↗
  5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Cla… — PaLM: Scaling Language Modeling with Pathways
  6. Dhruv Gautam, Spandan Garg, Jinu Jang, Neel Sundaresan, and Roshanak Zilouchian Moghaddam. 2025. RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code. arXiv:2503.07832 [cs.AI] https://arxiv.org/abs/2503.07832 — [arXiv:2503.07832](https://arxiv.org/abs/2503.07832) ↗
  7. Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Nguyen, and Omer Tripp. 2024. A Deep Dive into Large Language Models for Automated Bug Localization and Repair. 1, FSE, Article 66 (July 2024), 23 pages. doi:10.1145/3660773 — [doi:10.1145/3660773](https://doi.org/10.1145/3660773) find
  8. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66 ↗
  9. Yucheng Li. 2023. Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation. arXiv:2309.10677 [cs.CL] https://arxiv.org/abs/2309.10677 — [arXiv:2309.10677](https://arxiv.org/abs/2309.10677) ↗
  10. OpenAI. 2024. Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/. https://openai.com/index/introducing-swe-bench-verified/ Published August 13, 2024; updated February 24, 2025. ↗
  11. OpenAI. 2025. Models · OpenAI API. OpenAI. https://platform.openai.com/docs/models API documentation page. ↗
  12. OpenAI. 2025. O3 model. OpenAI. https://platform.openai.com/docs/models/o3 Model documentation. ↗
  13. Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. 2024. Training Software Engineering Agents and Verifiers with SWE-Gym. arXiv:2412.21139 [cs.SE] https://arxiv.org/abs/2412.21139 ↗
  14. Daniel Ramos, Claudia Mamede, Kush Jain, Paulo Canelas, Catarina Gamboa, and Claire Le Goues. 2025. Are Large Language Models Memorizing Bug Benchmarks? arXiv:2411.13323 [cs.SE] https://arxiv.org/abs/2411.13323 — [arXiv:2411.13323](https://arxiv.org/abs/2411.13323) ↗
  15. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Al… — Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
  16. Florian Tambon, Amin Nikanjam, Cyrine Zid, Foutse Khomh, and Giuliano Antoniol. 2025. TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models. arXiv:2407.21227 [cs.SE] https://arxiv.org/abs/2407.21227 — [arXiv:2407.21227](https://arxiv.org/abs/2407.21227) ↗
  17. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:23…
  18. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, a… — OpenHands: An Open Platform for AI Software Developers as Generalist Agents
  19. Chunqiu Steven Xia, Yinlin Deng, and Lingming Zhang. 2024. Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM. arXiv preprint (2024). find
  20. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv:2405.15793 [cs.SE]
  21. Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. Don’t Make Your LLM an Evaluation Benchmark Cheater. arXiv:2311.01964 [cs.CL] https://arxiv.org/abs/2311.01964 ↗
  22. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2024. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. arXiv preprint arXiv:2406.15877 (2024).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liang-2025-swe-bench-illusion-state/