Elamur Research⌕
☰
home / papers / 2026 / RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories

RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories

Paper · 2026 · Yanlin Wang, Ziyao Zhang, Chong Wang et al.

Summary

Most benchmarks for secure code generation use synthetic flaws or isolated snippets. They also tend to judge functionality and security separately, which misses vulnerabilities arising from data flows across a whole repository. RealSec-bench addresses this with Java tasks taken from real repositories. The authors chose popular Maven projects, scanned them with CodeQL, and kept the vulnerability-dense ones. They then filtered candidates in several stages:

  • only functions that compile and are covered by runnable unit tests were kept;
  • GPT-4.1 removed likely false positives, and security experts checked the result;
  • an LLM rewrote docstrings to be security-neutral, and programmers reviewed them.

Five LLMs were evaluated with Pass@k, Secure@k and the combined SecurePass@k. No model reached 6% SecurePass@1, and all failed completely on cryptography tasks. Retrieval augmentation (BM25, dense, dataflow) helped functionality somewhat but did little for security. Adding security guidelines to prompts gave inconsistent results and often broke compilation. The benchmark shows a clear gap between code that works and code that is verifiably secure.

Key points

  • Construction funnel: top 4,000 starred Java repos → 929 compilable Maven projects → 532 high-risk repos → ~20,000 raw CodeQL alerts → 160 executable instances → 105 final tasks.
  • The tasks cover 19 CWE types, with inter-procedural data-flow chains of up to 34 hops.
  • Evaluated models: Claude-3.7-Sonnet, Deepseek-V3, GPT-4.1, GPT-4.1-mini and Qwen3-235B. SecurePass@1 stayed below 6% for all of them.
  • Dataflow-focused retrieval often did worse than broader text-based retrieval, apparently because it misses functional context.
  • Limitations: the security oracle (CodeQL plus LLM adjudication) is weaker for logic, concurrency and crypto flaws, and the benchmark covers only open-source Maven/Java projects. Code and data are on GitHub (DeepSoftwareAnalytics/Realsec-code-Bench).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China168 works · 15,386 citations🎓Nanyang Technological UniversitySingapore · Singapore156 works · 25,644 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🎓Sun Yat-sen UniversityGuangzhou · China53 works · 9,185 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksInformation Systems · 1,437 worksSoftware · 92 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksCode & programming languages · 129 works

Cite

Cite as Yanlin Wang, Ziyao Zhang, Chong Wang, Xinyi Xu, Mingwei Liu, Yong Wang, Jiachi Chen and Zibin Zheng. 2026. RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1786

@inproceedings{wang2026realsec,
  title = {{RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories}},
  author = {Yanlin Wang, Ziyao Zhang, Chong Wang, Xinyi Xu, Mingwei Liu, Yong Wang, Jiachi Chen and Zibin Zheng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1786},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1786}
}

References

39 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2024. github-rest-api. https://docs.github.com/ en/rest. 2024. maven. https://maven.apache.org/.2025. codeql. https://codeql.github.com/. ↗
  2. Anthropic. 2025. Claude 3.7 Sonnet. https://ww w.anthropic.com/news/claude-3-7-sonnet. Accessed: 2025-09-08. find
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  4. Wachiraphan Charoenwet, Patanamon Thongtanunam, Van-Thuan Pham, and Christoph Treude. 2024a. An empirical study of static analysis tools for secure code review. In Proceedings of the 33rd ACM SIG-SOFT international symposium on software testing and analysis, pages 691–703. find
  5. Wachiraphan Charoenwet, Patanamon Thongtanunam, Van-Thuan Pham, and Christoph Treude. 2024b. Toward effective secure code reviews: an empirical study of security-related coding weaknesses. Empirical Software Engineering, 29(4):88. find
  6. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  7. Yizhou Chen, Zeyu Sun, Guoqing Wang, Qingyuan Liang, Xiao Yu, and Dan Hao. 2025. From cryptic to clear-training on llm explanations to detect smart contract vulnerabilities. ACM Transactions on Software Engineering and Methodology. find
  8. Wei Cheng, Yuhan Wu, and Wei Hu. 2024. Dataflow-guided retrieval augmentation for repository-level code completion. arXiv preprint arXiv:2405.19782. find
  9. DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437. find
  10. Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, and Yizheng Chen. 2025. Secrepobench: Benchmarking llms for secure code generation in real-world repositories. arXiv preprint arXiv:2504.21205. — [arXiv:2504.21205](https://arxiv.org/abs/2504.21205) find
  11. Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2022. Cocomic: Code completion by jointly modeling in-file and cross-file context. arXiv preprint arXiv:2212.10007.
  12. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. find
  13. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. find
  14. Jinyao Guo, Chengpeng Wang, Xiangzhe Xu, Zian Su, and Xiangyu Zhang. 2025. Repoaudit: An autonomous llm-agent for repository-level code auditing. arXiv preprint arXiv:2501.18160. — [arXiv:2501.18160](https://arxiv.org/abs/2501.18160) find
  15. Hossein Hajipour, Keno Hassler, Thorsten Holz, Lea Schönherr, and Mario Fritz. 2024. Codelmsec benchmark: Systematically evaluating and finding security vulnerabilities in black-box code language models. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 684–709. IEEE. find
  16. Nima Shiri Harzevili, Jiho Shin, Junjie Wang, Song Wang, and Nachiappan Nagappan. 2023. Characterizing and understanding software security vulnerabilities in machine learning libraries. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pages 27–38. IEEE. find
  17. Jingxuan He and Martin Vechev. 2023. Large language models for code: Security hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 1865–1879. find
  18. Ziyou Jiang, Lin Shi, Guowei Yang, and Qing Wang. 2024. Patuntrack: Automated generating patch examples for issue reports without tracked insecure code. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pages 1–13. find
  19. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.
  20. Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. 2024. Evocodebench: An evolving code generation benchmark aligned with real-world code repositories. arXiv preprint arXiv:2404.00599. — [arXiv:2404.00599](https://arxiv.org/abs/2404.00599) find
  21. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, and 1 others. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161. find
  22. Wei Liu, Ailun Yu, Daoguang Zan, Bo Shen, Wei Zhang, Haiyan Zhao, Zhi Jin, and Qianxiang Wang. 2024. Graphcoder: Enhancing repository-level code completion via code context graph-based retrieval and language model. arXiv preprint arXiv:2406.07003. — [arXiv:2406.07003](https://arxiv.org/abs/2406.07003) find
  23. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474. find
  24. OpenAI. 2025a. gpt-4.1. https://platform.openai.com/docs/models/gpt-4.1. ↗
  25. OpenAI. 2025b. gpt-4.1-mini. https://platform.o penai.com/docs/models/gpt-4.1-mini. ↗
  26. Oracle. 2025. Documentation Comment Specification for the Standard Doclet (JDK 24). https://docs.o racle.com/en/java/javase/24/docs/specs/j avadoc/doc-comment-spec.html. ↗
  27. Jinjun Peng, Leyi Cui, Kele Huang, Junfeng Yang, and Baishakhi Ray. 2025. Cweval: Outcome-driven evaluation on functionality and security of llm code generation. In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), pages 33–40. IEEE. find
  28. Stephen Robertson, Hugo Zaragoza, and 1 others. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. find
  29. Atanas Rountev, Scott Kagan, and Michael Gibas. 2004. Static and dynamic analysis of call chains in java. In Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis, pages 1–11. find
  30. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, and 1 others. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.
  31. The MITRE Corporation. 2025. Common Weakness Enumeration (CWE). https://cwe.mitre.org. Accessed: 2025-09-08. ↗
  32. The OWASP Foundation. 2025. OWASP Secure Coding Practices-Quick Reference Guide. https://owasp.org/www-project-secure-coding-practices-q uick-reference-guide/. ↗
  33. Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanović, Jingxuan He, and Martin Vechev. 2025. Baxbench: Can llms generate correct and secure backends? arXiv preprint arXiv:2502.11844. — [arXiv:2502.11844](https://arxiv.org/abs/2502.11844) find
  34. Cheng Wen, Yuandao Cai, Bin Zhang, Jie Su, Zhiwu Xu, Dugang Liu, Shengchao Qin, Zhong Ming, and Tian Cong. 2024. Automatically inspecting thousands of static bug warnings with large language model: How far are we? ACM Transactions on Knowledge Discovery from Data, 18(7):1–34. find
  35. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, and 1 others. 2025. Qwen3 Technical Report. arXiv preprint.
  36. Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570. find
  37. Zhuo Zhang, Yan Lei, Meng Yan, Yue Yu, Jiachi Chen, Shangwen Wang, and Xiaoguang Mao. 2022. Reentrancy vulnerability detection and localization: A deep learning based two-phase approach. In Proceedings of the 37th IEEE/ACM international conference on automated software engineering, pages 1–13. find
  38. Li Zhong and Zilong Wang. 2024. Can llm replace stack overflow? a study on robustness and reliability of large language model code generation. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 21841–21849. find
  39. Shuyan Zhou, Uri Alon, Frank F Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2022. Docprompting: Generating code by retrieving the docs. arXiv preprint arXiv: 2207.05987. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-realsec-bench-benchmark-evaluating-secure/