Elamur Research⌕
☰
home / papers / 2026 / AutoSUIT Bench - Automated Security UnIt Test Benchmark for LLM Coding

AutoSUIT Bench - Automated Security UnIt Test Benchmark for LLM Coding

Paper · 2026 · Samuel Osebe, Fan Yang, Junyi Li et al.

Summary

Benchmarks that check whether LLM-written code is secure tend to rely on one of two approaches. Some use static analyzers such as CodeQL, which make many errors. Others use hand-written security tests, which limits them to a few dozen of the most common CWE categories. AutoSUIT addresses both problems with an automated pipeline. It takes the incomplete demonstration code from MITRE CWE entries and has an oracle LLM rewrite it into executable vulnerable functions. It then generates:

  • task descriptions,
  • functional unit tests,
  • patched secure versions,
  • security unit tests.

Each artifact is repeatedly refined using compiler messages and test failures until it validates. The resulting AutoSUIT Bench supports four tasks: code generation, generation with CWE context, security patching, and code completion. Across eight proprietary and open-weight LLMs, functional pass rates were always higher than security pass rates. Models also handled the well-known top CWEs much better than rarer ones. The authors conclude that security evaluation needs far broader CWE coverage than earlier benchmarks offered.

Key points

  • AutoSUIT Bench contains 952 samples covering 232 CWE categories in C, C++, Java and Python. Earlier dynamic, unit-test-based benchmarks covered fewer than 50 CWEs, for example SecCodePLT with 44.
  • Security patching was harder than plain generation. Security scores dropped for every language except C, and functional pass rates for Java and Python fell by roughly 10%. Java showed the most syntax breakage.
  • Giving models 30% of the solution as a prefix (code completion) consistently raised both functional and security pass rates.
  • Several CWEs that models frequently produced, such as CWE 250, 256 and 326, are absent from the 2022–2024 top-25 most dangerous list. Among the most affected categories, 25% are missing from that list for C and 70% for Python.
  • Stated limitations:
    • The benchmark works at the function level only, so it cannot capture repository-level vulnerabilities.
    • Each sample carries a single primary CWE label.
    • Dynamic test execution adds computational and engineering cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Massachusetts AmherstAmherst Center · United States106 works · 66,692 citations🏢AmazonSeattle · United States112 works · 40,565 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 worksSoftware · 92 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Samuel Osebe, Fan Yang, Junyi Li, Yue Gu, Yongxin Wang, Satyapriya Krishna, Kai-Wei Chang, Aram Galstyan, Rahul Gupta and Weitong Ruan. 2026. AutoSUIT Bench - Automated Security UnIt Test Benchmark for LLM Coding. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1735

@inproceedings{osebe2026autosuit,
  title = {{AutoSUIT Bench - Automated Security UnIt Test Benchmark for LLM Coding}},
  author = {Samuel Osebe, Fan Yang, Junyi Li, Yue Gu, Yongxin Wang, Satyapriya Krishna, Kai-Wei Chang, Aram Galstyan, Rahul Gupta and Weitong Ruan},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1735},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1735}
}

References

22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  2. Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, and 1 others. 2024. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. arXiv preprint arXiv:2404.13161. — [arXiv:2404.13161](https://arxiv.org/abs/2404.13161) find
  3. Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, and 1 others. 2023. Purple llama cyberseceval: A secure coding benchmark for language models. arXiv preprint arXiv:2312.04724. — [arXiv:2312.04724](https://arxiv.org/abs/2312.04724) find
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  5. Shih-Chieh Dai, Jun Xu, and Guanhong Tao. 2025. Rethinking the evaluation of secure code generation. Preprint, arXiv:2503.15554. — [arXiv:2503.15554](https://arxiv.org/abs/2503.15554) find
  6. Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, and Yizheng Chen. 2025. Secrepobench: Benchmarking llms for secure code generation in real-world repositories. arXiv preprint arXiv:2504.21205. — [arXiv:2504.21205](https://arxiv.org/abs/2504.21205) find
  7. Hossein Hajipour, Keno Hassler, Thorsten Holz, Lea Schönherr, and Mario Fritz. 2024. Codelmsec benchmark: Systematically evaluating and finding security vulnerabilities in black-box code language models. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 684–709. IEEE. find
  8. Joseph R. Horgan, Saul London, and Michael R Lyu. 2002. Achieving software quality with testing coverage measures. Computer, 27(9):60–69. find
  9. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.
  10. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.
  11. Avishree Khare, Saikat Dutta, Ziyang Li, Alaia Solko-Breslin, Rajeev Alur, and Mayur Naik. 2025. Understanding the effectiveness of large language models in detecting security vulnerabilities. In 2025 IEEE Conference on Software Testing, Verification and Validation (ICST), pages 103–114. IEEE. find
  12. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558–21572.
  13. Jinjun Peng, Leyi Cui, Kele Huang, Junfeng Yang, and Baishakhi Ray. 2025. Cweval: Outcome-driven evaluation on functionality and security of llm code generation. In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), pages 33–40. IEEE. find
  14. Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, and Wei Le. 2023. An empirical study of deep learning models for vulnerability detection. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 2237–2248. IEEE. find
  15. Catherine Tony, Markus Mutas, Nicolás E Díaz Ferreyra, and Riccardo Scandariato. 2023. Llmseceval: A dataset of natural language prompts for security evaluations. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pages 588–592. IEEE. find
  16. Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanović, Jingxuan He, and Martin Vechev. 2025. Baxbench: Can llms generate correct and secure backends? arXiv preprint arXiv:2502.11844. — [arXiv:2502.11844](https://arxiv.org/abs/2502.11844) find
  17. Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, and 1 others. 2024. Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models. arXiv preprint… — [arXiv:2408.01605](https://arxiv.org/abs/2408.01605) find
  18. Yu Yang, Yuzhou Nie, Zhun Wang, Yuheng Tang, Wenbo Guo, Bo Li, and Dawn Song. 2024. Seccodeplt: A unified platform for evaluating the security of code genai. arXiv preprint arXiv:2410.11096. find
  19. Zhaojian Yu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. 2024. Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation. arXiv preprint arXiv:2412.21199. — [arXiv:2412.21199](https://arxiv.org/abs/2412.21199) find
  20. Hong Zhu, Patrick AV Hall, and John HR May. 1997. Software unit test coverage and adequacy. Acm computing surveys (csur), 29(4):366–427. find
  21. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, and 1 others. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877.
  22. Hwiwon Lee, Ziqi Zhang, Hanxiao Lu, and Lingming Zhang. 2025. Sec-bench: Automated benchmarking of llm agents on real-world software security tasks. arXiv preprint arXiv:2506.11791. — [arXiv:2506.11791](https://arxiv.org/abs/2506.11791) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/osebe-2026-autosuit-bench-automated-security-unit/