Elamur Research⌕
☰
home / papers / 2026 / SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?

SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?

Paper · 2026 · Yuxuan Sun, Yuze Zhao, Yufeng Wang et al.

Summary

Training and evaluating LLMs for software engineering depends on test suites, both for synthesizing repair trajectories and for providing reward signals in reinforcement learning. Tests written by LLMs are often shallow and cannot separate correct patches from subtly wrong ones. Existing mutation-based benchmarks mostly use rule-based or few-shot mutants, which are easy to catch, so they tend to overrate test quality.

SWE-Mutation addresses this by measuring how well model-written tests catch realistic faulty variants of the reference fix. The mutants come from an agentic, language-agnostic pipeline with four modules: Locate, Mutation, Judge and Self-Play. Claude Sonnet 4 serves as the backbone. The pipeline was applied to SWE-bench Verified and SWE-bench-Multilingual, and the benchmark covers two tasks: test generation and test repair.

Seven LLMs were evaluated under Mini-Swe-Agent and Claude Code, and all of them struggled. Results were weaker still outside Python. Compared with conventional mutants, the agentic ones are considerably harder to detect and separate models more clearly. This makes the benchmark a stricter testbed for test-writing ability.

Key points

  • Benchmark size: 2,636 mutants built from 800 instances, with 3–5 mutants per instance.
    • Python part: 1,664 mutants over 500 SWE-bench Verified instances from 11 GitHub repositories.
    • Multilingual part: 972 mutants over 300 instances in 9 languages.
  • Metrics: Pass@1, Verified Reproduction Rate (VRR), and the new Relative Detection Rate (RDR), which is the relative share of mutants killed by the tests.
  • Headline result: DeepSeek-V3.1 achieves only 10.20% VRR and 36.15% RDR. Compared with conventional mutation methods, the agentic mutants cut average RDR from 71.04% to 39.81%.
  • Generator-bias check: The Claude generator was swapped for DeepSeek-V3.1 and Qwen3-Coder.
    • RDR shifted by at most 1.5 pp.
    • Model rankings stayed stable, with Spearman ρ of 0.96 and 0.93.
  • Common failure causes:
    • weak grasp of global repository structure;
    • errors when working across files and generating long test code;
    • misreading the issue;
    • mismatched environment or package versions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓University of Science and Technology of ChinaHefei · China108 works · 112,698 citations🎓Beihang UniversityBeijing · China65 works · 8,452 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 worksSoftware · 92 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksCode & programming languages · 129 works

Cite

Cite as Yuxuan Sun, Yuze Zhao, Yufeng Wang, Yao Du, Zhiyuan Ma, Jinbo Wang, Mengdi Zhang, Kai Zhang and Zhenya Huang. 2026. SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1976

@inproceedings{sun2026swe,
  title = {{SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?}},
  author = {Yuxuan Sun, Yuze Zhao, Yufeng Wang, Yao Du, Zhiyuan Ma, Jinbo Wang, Mengdi Zhang, Kai Zhang and Zhenya Huang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1976},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1976}
}

References

45 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Ai research and products that put safety at the frontier. find
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  3. Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering, 41(5):507–525. find
  4. Cristian Cadar and Koushik Sen. 2013. Symbolic execution for software testing: three decades later. Communications of the ACM, 56(2):82–90. find
  5. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2023. MultiPL-E: a scalable and polyglot approach to benchmarking neural code generat… find
  6. Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 33(8):1–79. find
  7. Ali Reza Ibrahimzada, Yang Chen, Ryan Rong, and Reyhaneh Jabbarvand. 2025. Challenging bug prediction and repair models with synthetic bugs. In Proceedings of the 25th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM ’25). IEEE. find
  8. Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. 2025. Testgeneval: A real world unit test generation and test completion benchmark. In The Thirteenth International Conference on Learning Representations. find
  9. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.
  10. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. CodeT: Code generation with generated tests. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6462–6477. Association for Computa… find
  11. René Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. 2014a. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 654–665. find
  12. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  13. René Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. 2014b. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE ’14), pages 654–665. A… find
  14. Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. PIT: A practical mutation testing tool for Java. In Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA ’16), pages 449–452. ACM. find
  15. DeepSeek-AI. 2025. Deepseek-v3.1 release. find
  16. Renzo Degiovanni and Mike Papadakis. 2022. µBERT: Mutation testing using pre-trained language models. In Proceedings of the 15th IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW ’22), pages 160–169. IEEE. find
  17. Yanmin Dong, Zhenya Huang, Zheng Zhang, Guanhao Zhao, Likang Wu, Hongke Zhao, Binbin Jin, and Qi Liu. 2025. Enhancing code search intent with René Just, Franz Schweiggert, and Gregory M. Kapfhammer. 2011. MAJOR: an efficient and extensible tool for mutation analysis in a java compiler. In Proceeding… find
  18. Kimi Team, Yifan Bai, and 1 others. 2025. Kimi k2: Open agentic intelligence. Preprint, arXiv:2507.20534.
  19. James C King. 1976. Symbolic execution and program testing. Communications of the ACM, 19(7):385– 394. find
  20. Hung Le, Yue Wang, Akhilesh D Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pages 21314–21328.
  21. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, volume 36.
  22. Niels Mündler, Mark Niklas Müller, Jingxuan He, and Martin Vechev. 2024. Swt-bench: Testing and validating real-world bug-fixes with code agents. In Advances in Neural Information Processing Systems, volume 37. find
  23. OpenAI. 2024. Introducing swe-bench verified. find
  24. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925.
  25. Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation testing advances: An analysis and survey. Advances in Computers, 112:275–378. find
  26. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  27. Frank Tip, Jonathan Bell, and Max Schäfer. 2025. Llmorpheus: Mutation testing using large language models. IEEE Transactions on Software Engineering, 51(6):1365–1381. find
  28. John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. 2025a. Swe-smith: Scaling data for software engineering agents. Preprint, arXiv:2504.21798. find
  29. Zheyuan Yang, Zexi Kuang, Xue Xia, and Yilun Zhao. 2025b. Can LLMs generate high-quality test cases for algorithm problems? TestCase-Eval: A systematic evaluation of fault coverage and exposure. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Sho… find
  30. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
  31. Junhao Yu, Yan Zhuang, Yuxuan Sun, Weibo Gao, Qi Liu, Mingyue Cheng, Zhenya Huang, and Enhong Chen. 2025. TestAgent: An adaptive and intelligent expert for human assessment. In Findings of the Association for Computational Linguistics: ACL 2025, pages 724–747, Vienna, Austria. Association for Comput… find
  32. Zhiqiang Yuan, Yifan Yan, Wenhan Liu, Zhen Chen, Ge Li, and 1 others. 2023. No more manual tests? evaluating and improving chatgpt for unit test generation. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. find
  33. Michele Tufano, Jason Kimko, Shiya Wang, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, and Denys Poshyvanyk. 2020. DeepMutation: A neural mutation tool. In Proceedings of the 42nd International Conference on Software Engineering: Companion Proceedings (ICSE ’20), pages 73–76. ACM. find
  34. Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. 2025. Multi-swe-bench: A multilingual benchmark for issue resolving… — [arXiv:2504.02605](https://arxiv.org/abs/2504.02605) find
  35. Guancheng Wang, Qinghua Xu, Lionel C. Briand, and Kui Liu. 2025a. Mutation-guided unit test generation with a large language model. In Proceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering (ASE ’25). IEEE/ACM. find
  36. Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023. Self-edit: Fault-aware code editor for code generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 769–787. Association for Computational Linguistics. find
  37. Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. 2025b. Testeval: Benchmarking large language models for test case generation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3547–3562. Association… find
  38. Quanjun Zhang, Ye Shang, Chunrong Fang, Siqi Gu, Jianyi Zhou, and Zhenyu Chen. 2025. Testbench: Evaluating class-level test case generation capability of large language models. In Proceedings of the 47th International Conference on Software Engineering (ICSE ’25). IEEE/ACM. find
  39. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  40. Yuze Zhao, Zhenya Huang, Yixiao Ma, Rui Li, Kai Zhang, Hao Jiang, Qi Liu, Linbo Zhu, and Yu Su. 2024. RePair: Automated program repair with process-based feedback. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16415–16429, Bangkok, Thailand. Association for Computatio… find
  41. Yuze Zhao, Zhenya Huang, Kai Zhang, Weibo Gao, Qi Liu, Xukai Liu, Fangzhou Yao, and Enhong Chen. 2025a. Semantic-aligned code summarization: Bridging the gap between code and natural language through data flow analysis. IEEE Transactions on Neural Networks and Learning Systems, 36(10):19240–19254. find
  42. Yuze Zhao, Tianyun Ji, Wenjun Feng, Zhenya Huang, Qi Liu, Zhiding Liu, Yixiao Ma, Kai Zhang, and Enhong Chen. 2025b. Unveiling the magic of code reasoning through hypothesis decomposition and amendment. In The Thirteenth International Conference on Learning Representations. find
  43. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, and 1 others. 2023. CodeGeeX: A pre-trained model for code generation with multilingual benchmarking on HumanEval-X. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery an…
  44. Zhipu AI. 2025. Introducing glm-4.6. find
  45. A \\\# Original (Safe default): def fetch_url(url, timeout=30): return requests.get(url, timeout=timeout) \\\# Mutant (Dangerous default - works in fast tests, hangs in prod): def fetch_url(url, timeout=None): \# specific timeout requirement is removed return requests.get(url, timeout=timeout) , → Listin… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-swe-mutation-can-llms-generate/