Elamur Research⌕
☰
home / papers / 2026 / Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks

Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks

Paper · 2026 · Shoumik Saha, Jifan Chen, Sam Mayers et al.

Summary

AI coding agents can read, edit and run files, so a successful jailbreak can yield working malware rather than just objectionable text. Earlier safety evaluations mostly checked whether a model refused, not whether the code it produced would run. The authors built JAWS-Bench to close this gap. It tests agents in three workspace settings that reflect increasing attacker skill: an empty directory with only a prompt (JAWS-0), one partially written malicious file (JAWS-1), and a multi-file malicious project with one function body removed (JAWS-M).

A four-stage judge pipeline checks each output in turn:

  1. Did the agent comply?
  2. Is the result harmful?
  3. Does it parse?
  4. Does it execute to completion?

Seven LLM backends from five families were tested, mainly inside OpenHands. Adding code context made refusals rarer. Agents were notably more vulnerable than the same models used directly, partly because they abandon initial refusals during planning and tool use. The work argues for defenses that account for execution and that keep refusals in place across agent steps.

Key points

  • Benchmark size: 182 textual prompts (JAWS-0), 100 single-file codebases (JAWS-1) and 180 multi-file codebases (JAWS-M). Code is released on GitHub (amazon-science/JAWS-Bench).
  • JAWS-0 funnel: about 61% compliance, 58% harmful, 52% parsable and only 27% fully runnable.
  • Code context raises risk: in JAWS-1, compliance nears 100% for capable models and mean ASR is about 71%. In JAWS-M, mean ASR reaches about 75%, with 32% of outputs deployable.
  • Agent wrapping and attack types: putting an LLM inside an agent raises ASR about 1.6×. Spyware, phishing and privilege escalation are the easiest categories to weaponize. SWE-Agent and OpenAI Codex show similar trends to OpenHands.
  • Simple defense: adding a security-check instruction to the system prompt cuts JAWS-M compliance by roughly 19–43 points. Mistral Large improves the most, while Qwen3-235B barely changes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations🏢AmazonSeattle · United States112 works · 40,565 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 517 works

Cite

Cite as Shoumik Saha, Jifan Chen, Sam Mayers, Sanjay Krishna Gouda, Zijian Wang and Varun Kumar. 2026. Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.792

@article{saha2026breaking,
  title = {{Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks}},
  author = {Shoumik Saha, Jifan Chen, Sam Mayers, Sanjay Krishna Gouda, Zijian Wang and Varun Kumar},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.792},
  url = {https://doi.org/10.1162/tacl.a.792}
}

References

22 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, J Zico Kolter, Matt Fredrikson, Yarin Gal, and Xander Davies. 2025. Agentharm: A benchmark for CodeQL. Codeql. https://codeql.github.com/. Static analysis engine for querying c… ↗
  2. Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents. Advances in Neural Information Processing Systems, 37:82895–82920. find
  3. Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yun-Dehaene, Joel Lamy-Poirier, Joao Monte… find
  4. Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dolphin. 2024. Dolphin-Mistral-Dao, Mayank Mishra, Alex Gu, Carolyn Jane 24B-Venice-Edition. https: Anderson, Brendan Dolan-Gavitt, Danish Con-//huggingface.co/dphn/ tractor, Siva Reddy, Daniel Fried, Dzmit… find
  5. Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. 2023. Star-Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida coder: may the source be with you! Transac-Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, tions on Machine Learning Research. Repro-Scott Yih, Luke Zettlemoyer, and Mik… find
  6. Yao, Tongliang Liu, and Bo Han. 2024. Deep-inception: Hypnotize large language model to Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao be jailbreaker. In Neurips Safe Generative AI Du, Qian Liu, Ye Wang, Jing Jiang, and Min Workshop 2024. find
  7. Lin. 2024. Agent smith: A single image can jailbreak one million multimodal LLM agents Yujia Li, David Choi, Junyoung Chung, Nate Kush-exponentially fast. In ICLR 2024 Workshop on man, Julian Schrittwieser, Rémi Leblond, Tom Large Language Model (LLM) Agents. Eccles, James Keeling, Felix Gimeno, Agu… find
  8. Xinyun Chen, Po-Sen Huang, Johannes Welbl, 2024a. Redcode: Risky code execution and Sven Gowal, Alexey Cherepanov, James Mol-generation benchmark for code agents. Ad-loy, Daniel J. Mankowitz, Esme Sutherland Rob-vances in Neural Information Processing Sys-son, Pushmeet Kohli, Nando de Freitas, Ko-te… find
  9. Qin, and Bin Hu. 2024b. COLD-attack: Jailbreaking LLMs with stealthiness and controlla- Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei bility. In Forty-first International Conference on Xiao. 2024. AutoDAN: Generating stealthy jail-Machine Learning. break prompts on aligned large language models. In… find
  10. Jacob Steinhardt. 2023. Automatically auditing large language models via discrete optimization. Yuzhou Nie, Zhun Wang, Yu Yang, Ruizhe Jiang, In International Conference on Machine Learn-Yuheng Tang, Xander Davies, Yarin Gal, Bo Li, ing, pages 15307–15329. find
  11. PMLR. Wenbo Guo, and Dawn Song. 2025. Secodeplt: A unified platform for evaluating the security of code genai. engineering. Advances in Neural Information Processing Systems, 37:50528–50652. find
  12. OpenAI. 2025. Codex. https://developers.openai.com/codex. Accessed: 2025-11-21. ↗
  13. Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2024. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts.
  14. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. In NeurIPS ML Safety Workshop.
  15. Kevin Power. 2025. Drop the guardrails: Tool-primed prompt pairing and refusal behavior in gpt-oss. Kaggle. find
  16. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10471–10506, Bangkok, Thailand. Association for Computation…
  17. Bernardo Quintero. 2023. Introducing virustotal code insight: Empowering threat Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei analysis with generative ai. https: Yao, Zhenting Wang, Chenlu Zhan, Hongwei //blog.virustotal.com/2023/04/ Wang, and Yongfeng Zhang. 2025. Agent secu-introducing-virustotal-… find
  18. Xin Tan, Wai Lam, and Lizhuang Ma. 2024. Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, CodeAttack: Revealing safety generalization Joe Barrow, Zichao Wang, Furong Huang, Ani challenges of large language models via code Nenkova, and Tong Sun. 2023. AutoDAN: Au-completion. In Findings of the Association… find
  19. Vinu Sankar Sadasivan, Shoumik Saha, Gau-rang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi. 2024. Fast adversarial attacks on language models in one GPU minute. In Forty-first International Conference on Machine Learning. find
  20. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
  21. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, a… — OpenHands: An Open Platform for AI Software Developers as Generalist Agents
  22. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saha-2026-breaking-code-security-assessment-ai/