Elamur Research⌕
☰
home / papers / 2026 / When “Correct” Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?

When “Correct” Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?

Paper · 2026 · Yibo Peng, James Song, Lei Li et al.

Summary

Code agents are usually judged by whether their patches resolve an issue and pass the test suite. This paper argues that passing tests does not mean a patch is secure. The authors define a threat class called functionally correct yet vulnerable (FCV) patches: fixes that pass every test but still contain a CWE-classified weakness.

They first show that such patches occur even without any attacker, using Mini-SWE-Agent on SWE-bench. They then propose FCV-Attack, which appends a developer-style suggestion targeting a specific CWE to a GitHub issue description. The attack needs only black-box access and a single query. They test it on 12 combinations of four LLMs and three agent scaffolds, and every combination proves susceptible.

Controlled-trajectory experiments suggest the harm comes from the injected text contaminating the model's internal state. Agents that retrieve the right files and reason correctly still emit vulnerable code. This implies that monitoring agent behaviour and checking functional correctness are not enough, and that security-aware defenses are needed.

Key points

  • Benign baseline: among test-passing patches from Mini-SWE-Agent, the share containing security weaknesses was 6.0% for Qwen3-Coder, 5.0% for Kimi-K2-Instruct, 4.5% for GPT-5 mini and 4.3% for Claude Sonnet 4. The paper calls this metric the "FCV Rate."
  • Setup: four models (Qwen3-Coder-480B-A35B-Instruct, Kimi-K2-Instruct, GPT-5-mini, Claude-Sonnet-4) were combined with three scaffolds (Mini-SWE-Agent, SWE-Agent, OpenHands) on SWE-Bench. The targeted CWEs were CWE-538, CWE-79, CWE-89 and CWE-94.
  • Attack results: information exposure (CWE-538) was the easiest weakness to induce. Its attack success rate reached 40.7% on GPT-5 mini + OpenHands and 55.6% on Claude Sonnet 4 + OpenHands, with a maximum of 56.3% across all settings.
  • Defense test: adding a generic security reminder to the system prompt only modestly lowered ASR. For CWE-538, Kimi-K2 dropped from 54.2% to 43.3% and Qwen3-Coder from 19.0% to 17.9%. Both stayed well above their clean baselines of 0.8% and 9.5%.
  • Code and limitations: code is released at github.com/Infini-AI-Lab/FCV. The authors note that the study covers only four CWEs. They also acknowledge that the internal-state explanation rests on observed agent behaviour rather than analysis of the model's representations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓University of MichiganAnn Arbor · United States70 works · 13,693 citations🎓Colorado State UniversityFort Collins · United States3 works · 33 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 worksSoftware · 92 works

Cite

Cite as Yibo Peng, James Song, Lei Li, Xinyu Yang, Mihai Christodorescu, Ravi Mangal, Corina S. Pasareanu, Haizhong Zheng and Beidi Chen. 2026. When “Correct” Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.707

@inproceedings{peng2026when,
  title = {{When “Correct” Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?}},
  author = {Yibo Peng, James Song, Lei Li, Xinyu Yang, Mihai Christodorescu, Ravi Mangal, Corina S. Pasareanu, Haizhong Zheng and Beidi Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.707},
  url = {https://doi.org/10.18653/v1/2026.acl-long.707}
}

References

49 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2023. Claude llm family. https://www.anthropic.com/. Accessed: 2025-09-14. ↗
  2. Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, David Molnar, Spencer Whitman, and Joshua Saxe. 2024. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. Prepri… — [arXiv:2404.13161](https://arxiv.org/abs/2404.13161) find
  3. Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, Sasha Frolov, Ravi Prakash Giri, Dhaval Kapil, Yiannis Kozyrakis, David LeBlanc, James Milazzo, Aleksandar Straumann, Gabriel Synnaeve, Var… — [arXiv:2312.04724](https://arxiv.org/abs/2312.04724) find
  4. Marc Bruni, Fabio Gabrielli, Mohammad Ghafari, and Martin Kropp. 2025. Benchmarking prompt engineering techniques for secure code generation with gpt models. In 2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge), pages 93–103. IEEE. find
  5. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2025. Jailbreaking black box large language models in twenty queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 23–42. IEEE.
  6. Sahana Chennabasappa, Cyrus Nikolaidis, Daniel Song, David Molnar, Stephanie Ding, Shengye Wan, Spencer Whitman, Lauren Deason, Nicholas Doucette, Abraham Montilla, and 1 others. 2025. Llamafirewall: An open source guardrail system for building secure ai agents. arXiv preprint arXiv:2505.03574. — [arXiv:2505.03574](https://arxiv.org/abs/2505.03574) find
  7. Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. 2024. Introducing swe-bench verified. https://openai.com/index/ int… ↗
  8. Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, and Yizheng Chen. 2025. Secrepobench: Benchmarking llms for secure code generation in real-world repositories. arXiv preprint arXiv:2504.21205. — [arXiv:2504.21205](https://arxiv.org/abs/2504.21205) find
  9. Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. 2024a. Llm agents can autonomously exploit one-day vulnerabilities. Preprint, arXiv:2404.08144. — [arXiv:2404.08144](https://arxiv.org/abs/2404.08144) find
  10. Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. 2024b. Llm agents can autonomously hack websites. arXiv preprint arXiv:2402.06664. — [arXiv:2402.06664](https://arxiv.org/abs/2402.06664) find
  11. Dawei Gao, Zitao Li, Xuchen Pan, Weirui Kuang, Zhijian Ma, Bingchen Qian, Fei Wei, Wenhao Zhang, Yuexiang Xie, Daoyuan Chen, Liuyi Yao, Hongyi Peng, Zeyu Zhang, Lin Zhu, Chen Cheng, Hongzhu Shi, Yaliang Li, Bolin Ding, and Jingren Zhou. 2024. Agentscope: A flexible yet robust multi-agent platform. C… — [arXiv:2402.14034](https://arxiv.org/abs/2402.14034) find
  12. Dawei Gao, Zitao Li, Yuexiang Xie, Weirui Kuang, Liuyi Yao, Bingchen Qian, Zhijian Ma, Yue Cui, Haohao Luo, Shen Li, Lu Yi, Yi Yu, Shiqi He, Zhiling Luo, Wenmeng Zhou, Zhicheng Zhang, Xuguang He, Ziqian Chen, Weikai Liao, and 4 others. 2025. Agentscope 1.0: A developer-centric framework for building… — [arXiv:2508.16279](https://arxiv.org/abs/2508.16279) find
  13. Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, and Bo Li. 2024. Redcode: Risky code execution and generation benchmark for code agents. find
  14. Dongyoon Hahm, Woogyeol Jin, June Suk Choi, Sungsoo Ahn, and Kimin Lee. 2025a. Enhancing llm agent safety via causal influence prompting. arXiv preprint arXiv:2507.00979. find
  15. Dongyoon Hahm, Taywon Min, Woogyeol Jin, and Kimin Lee. 2025b. Unintended misalignment from agentic fine-tuning: Risks and mitigation. arXiv preprint arXiv:2508.14031. — [arXiv:2508.14031](https://arxiv.org/abs/2508.14031) find
  16. Hossein Hajipour, Keno Hassler, Thorsten Holz, Lea Schönherr, and Mario Fritz. 2023. Codelmsec benchmark: Systematically evaluating and finding security vulnerabilities in black-box code language models. Preprint, arXiv:2302.04012. — [arXiv:2302.04012](https://arxiv.org/abs/2302.04012) find
  17. Slobodan Jenko, Niels Mündler, Jingxuan He, Mark Vero, and Martin Vechev. 2025. Black-box adversarial attacks on llm-based code completion. Preprint, arXiv:2408.02509. — [arXiv:2408.02509](https://arxiv.org/abs/2408.02509) find
  18. Changyue Jiang, Xudong Pan, and Min Yang. 2025. Think twice before you act: Enhancing agent behavioral safety with thought correction. arXiv preprint arXiv:2505.11063. — [arXiv:2505.11063](https://arxiv.org/abs/2505.11063) find
  19. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.
  20. Juhee Kim, Woohyuk Choi, and Byoungyoung Lee. 2025. Prompt flow integrity to prevent privilege escalation in llm agents. arXiv preprint arXiv:2503.15547. — [arXiv:2503.15547](https://arxiv.org/abs/2503.15547) find
  21. Hwiwon Lee, Ziqi Zhang, Hanxiao Lu, and Lingming Zhang. 2025. Sec-bench: Automated benchmarking of llm agents on real-world software security tasks. arXiv preprint arXiv:2506.11791. — [arXiv:2506.11791](https://arxiv.org/abs/2506.11791) find
  22. Xinghang Li, Jingzhe Ding, Chao Peng, Bing Zhao, Xiang Gao, Hongwan Gao, and Xinchen Gu. 2025. Safegenbench: A benchmark framework for security vulnerability detection in llm-generated code. arXiv preprint arXiv:2506.05692. — [arXiv:2506.05692](https://arxiv.org/abs/2506.05692) find
  23. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Conference on Learning Representations.
  24. Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. 2024. How to understand whole software repository? arXiv preprint arXiv:2406.01422. mini-swe-agent. 2025. mini-swe-agent: The 100-line ai tool for devs. https://github.com/SWE-agent/. Accessed: 2025-10-05. — [arXiv:2406.01422](https://arxiv.org/abs/2406.01422) ↗
  25. MITRE. n.d. CWE-699: Software Development. https://cwe.mitre.org/data/definitions/ 699.html. Accessed: 2025-09-24. ↗
  26. MITRE Corporation. 2024. 2024 cwe top 25 most dangerous software weaknesses. https://cwe.mitre.org/top25/archive/2024/ 2024_cwe_top25.html. Accessed: 2025-09-29. ↗
  27. Wenjie Jacky Mo, Qin Liu, Xiaofei Wen, Dongwon Jung, Hadi Askari, Wenxuan Zhou, Zhe Zhao, and Muhao Chen. 2025. Redcoder: Automated multi-turn red teaming for code llms. arXiv preprint arXiv:2507.22063.
  28. Yuzhou Nie, Zhun Wang, Yu Yang, Ruizhe Jiang, Yuheng Tang, Xander Davies, Yarin Gal, Bo Li, Wenbo Guo, and Dawn Song. 2025. Secodeplt: A unified platform for evaluating the security of code genai. Preprint, arXiv:2410.11096. find
  29. Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanović, Jingxuan He, and Martin Vechev. 2025. Baxbench: Can llms generate correct and secure backends? arXiv preprint arXiv:2502.11844. — [arXiv:2502.11844](https://arxiv.org/abs/2502.11844) find
  30. OpenAI. 2025. Gpt-5 system card. https://cdn.openai.com/gpt-5-system-card.pdf. Accessed: 2025-10-05. ↗
  31. Sanidhya Vijayvargiya, Aditya Bharat Soni, Xuhui Zhou, Zora Zhiruo Wang, Nouha Dziri, Graham Neubig, and Maarten Sap. 2025. Openagentsafety: A comprehensive framework for evaluating real-world ai agent safety. Preprint, arXiv:2507.06134. find
  32. Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. 2025. Training software engineering agents and verifiers with swe-gym. Preprint, arXiv:2412.21139. find
  33. Team Qwen. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  34. Alfin Wijaya Rahardja, Junwei Liu, Weitong Chen, Zhenpeng Chen, and Yiling Lou. 2025. Can agents fix agent issues? Preprint, arXiv:2505.20749. — [arXiv:2505.20749](https://arxiv.org/abs/2505.20749) find
  35. Maxime Robeyns, Martin Szummer, and Laurence Aitchison. 2025. A self-improving coding agent. arXiv preprint arXiv:2504.15228. — [arXiv:2504.15228](https://arxiv.org/abs/2504.15228) find
  36. Pat Rondon, Renyao Wei, José Cambronero, Jürgen Cito, Aaron Sun, Siddhant Sanyam, Michele Tufano, and Satish Chandra. 2025. Evaluating agent-based program repair at google. Preprint, arXiv:2501.07531. find
  37. Abhik Roychoudhury, Corina Pasareanu, Michael Pradel, and Baishakhi Ray. 2025. Agentic ai software engineer: Programming with trust. arXiv preprint arXiv:2502.13767. — [arXiv:2502.13767](https://arxiv.org/abs/2502.13767) find
  38. Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawendé F Bissyandé. 2024. Codeagent: Autonomous communicative agents for code review. arXiv preprint arXiv:2402.02172. find
  39. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, and 150 others. 2025a. Kimi k2: Open agentic intelligence. Preprint,…
  40. Trae Research Team, Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, Yun Lin, Yingfei Xiong, Chao Peng, and Xia Liu. 2025b. Trae agent: An llm-based agent for software engineering with test-time scaling. find
  41. Ada Defne Tur, Nicholas Meade, Xing Han Lù, Alejandra Zambrano, Arkil Patel, Esin Durmus, Spandana Gella, Karolina Stańczak, and Siva Reddy. 2025. Safearena: Evaluating the safety of autonomous web agents. arXiv preprint arXiv:2503.04957. — [arXiv:2503.04957](https://arxiv.org/abs/2503.04957) find
  42. Muntasir Wahed, Xiaona Zhou, Kiet A. Nguyen, Tianjiao Yu, Nirav Diwan, Gang Wang, Dilek Hakkani-Tür, and Ismini Lourentzou. 2025. Mocha: Are code language models robust against multi-turn malicious coding prompts? Preprint, arXiv:2507.19598. find
  43. Haoyu Wang, Christopher M Poskitt, and Jun Sun. 2025a. Agentspec: Customizable runtime enforcement for safe and reliable llm agents. arXiv preprint arXiv:2503.18666. — [arXiv:2503.18666](https://arxiv.org/abs/2503.18666) find
  44. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, and 5 others. 2025b. Openhands: An open platform…
  45. Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. 2025. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449. find
  46. Fangzhou Wu, Xiaogeng Liu, and Chaowei Xiao. 2023. Deceptprompt: Exploiting llm-driven code generation via adversarial natural language instructions. arXiv preprint arXiv:2312.04730. — [arXiv:2312.04730](https://arxiv.org/abs/2312.04730) find
  47. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. arXiv preprint. find
  48. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in…
  49. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2026-when-correct-is-not-safe/