Elamur Research⌕
☰
home / papers / 2026 / Learning from Failures: Error Notebook-guided Secure Code Generation

Learning from Failures: Error Notebook-guided Secure Code Generation

Paper · 2026 · Xinyu Zhong, Peng Lan and Zhifang Liao

Summary

LLMs can write working programs but often introduce security vulnerabilities. Existing retrieval-augmented defenses have three weaknesses: they tend to give up functionality for safety, they retrieve mainly by security semantics, and they do not absorb new mistakes, so they generalize poorly to unseen CWE types. The authors propose SafeNote, which combines hierarchical security-knowledge retrieval (built from the SafeCoder dataset) with two "error notebooks" that log the model's own failures.

  • Offline stage: a Security Error Notebook is built by generating code for SafeCoder-derived tasks and flagging vulnerabilities with Semgrep. Each failure is paired with sliced secure code and LLM-written correction advice.
  • Online stage: a Function Error Notebook records unit-test failures and repair suggestions. Both notebooks are updated dynamically.

Relevant notes are fetched with Reciprocal Rank Fusion over code and task-description similarity. They serve as contrastive "what to avoid" context. Across five LLMs on CodeGuard+ and LiveCodeBench, SafeNote improves secure-and-correct output with little or no functional loss and handles unseen CWEs better than baselines. It offers a training-free route to safer code assistants.

Key points

  • On CodeGuard+, GPT-4o-mini's SP@1 rises from 60.21% to 66.70% with SafeNote.
  • Ablation on DeepSeek-V3.2 shows the role of each notebook:
    • Without the Security Error Notebook, SR drops from 84.39% to 80.85% and SP@1 falls to 66.06%.
    • Without the Function Error Notebook, Pass@1 falls from 79.26% to 75.21%. Kimi-K2-Instruct-0905 shows a similar drop, from 80.00% to 75.32%.
  • Varying the number of retrieved notes k from 0 to 5:
    • Most models peak at k between 2 and 4 (DeepSeek-V3.2 reaches 74.04% at k=4).
    • The biggest gain comes from moving from 0 to 1 note, so k=1 is the default.
  • Evaluation covers Python, C, and C++ in single-file settings only.
  • Detection is limited by the static analyzer, and logic flaws without predefined patterns may be missed. The note-merging step may also blur distinct error patterns.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksInformation Systems · 1,730 worksSoftware · 119 works

🧭 Topics 1Research subjects and languages the work is about.Code & programming languages · 168 works

Cite

Cite as Xinyu Zhong, Peng Lan and Zhifang Liao. 2026. Learning from Failures: Error Notebook-guided Secure Code Generation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1200

@inproceedings{zhong2026learning,
  title = {{Learning from Failures: Error Notebook-guided Secure Code Generation}},
  author = {Xinyu Zhong, Peng Lan and Zhifang Liao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1200},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1200}
}

References

37 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Weiheng Bai, Keyang Xuan, Pengxiang Huang, Qiushi Wu, Jianing Wen, Jingjing Wu, and Kangjie Lu. 2024. Apilot: Navigating large language models to generate secure code by sidestepping outdated api pitfalls. arXiv preprint arXiv:2409.16526. — [arXiv:2409.16526](https://arxiv.org/abs/2409.16526) find
  2. Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, and 1 others. 2023. Purple llama cyberseceval: A secure coding benchmark for language models. arXiv preprint arXiv:2312.04724. — [arXiv:2312.04724](https://arxiv.org/abs/2312.04724) find
  3. Mark Chen. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  4. Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 758–759.
  5. Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. 2025. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 57(6):1–39.
  6. Yanjun Fu, Ethan Baker, Yu Ding, and Yizheng Chen. 2024. Constrained decoding for secure code generation. arXiv preprint arXiv:2405.00218. — [arXiv:2405.00218](https://arxiv.org/abs/2405.00218) find
  7. GitHub. 2025. Codeql. https://codeql.github.com/. Accessed: 2025-10. ↗
  8. Hossein Hajipour, Keno Hassler, Thorsten Holz, Lea Schönherr, and Mario Fritz. 2024. Codelmsec benchmark: Systematically evaluating and finding security vulnerabilities in black-box code language models. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 684–709. IEEE. find
  9. Jingxuan He and Martin Vechev. 2023. Large language models for code: Security hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 1865–1879. find
  10. Jingxuan He, Mark Vero, Gabriela Krasnopolska, and Martin Vechev. 2024. Instruction tuning for secure code generation. arXiv preprint arXiv:2402.09497. — [arXiv:2402.09497](https://arxiv.org/abs/2402.09497) find
  11. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.
  12. Raphaël Khoury, Anderson R Avila, Jacob Brunelle, and Baba Mamadou Camara. 2023. How secure is code generated by chatgpt? In 2023 IEEE international conference on systems, man, and cybernetics (SMC), pages 2445–2451. IEEE. find
  13. Sung Yong Kim, Zhiyu Fan, Yannic Noller, and Abhik Roychoudhury. 2024. Codexity: secure ai-assisted code generation. arXiv preprint arXiv:2405.03927. — [arXiv:2405.03927](https://arxiv.org/abs/2405.03927) find
  14. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626.
  15. Hung Le, Hailin Chen, Amrita Saha, Akash Gokul, Doyen Sahoo, and Shafiq Joty. 2023. Codechain: Towards modular code generation through chain of self-revisions with representative sub-modules. arXiv preprint arXiv:2310.08992. — [arXiv:2310.08992](https://arxiv.org/abs/2310.08992) find
  16. Hung Le, Doyen Sahoo, Yingbo Zhou, Caiming Xiong, and Silvio Savarese. 2024. Indict: Code generation with internal dialogues of critiques for both security and helpfulness. Advances in Neural Information Processing Systems, 37:85546–85582. find
  17. Dong Li, Meng Yan, Yaosheng Zhang, Zhongxin Liu, Chao Liu, Xiaohong Zhang, Ting Chen, and David Lo. 2024. Cosec: On-the-fly security hardening of code llms via supervised co-decoding. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1428– 1439. find
  18. Bo Lin, Shangwen Wang, Yihao Qin, Liqian Chen, and Xiaoguang Mao. 2025. Give llms a security course: Securing retrieval-augmented code generation via knowledge injection. arXiv preprint arXiv:2504.16429. — [arXiv:2504.16429](https://arxiv.org/abs/2504.16429) find
  19. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
  20. Meta. 2024. Meta-llama-3-8b. https://huggingface.co/meta-llama/Meta-Llama-3-8B. ↗
  21. Moonshot AI. 2024. Kimi k2 (0905 preview). https://huggingface.co/moonshotai/ Kimi-K2-Instruct-0905. Preview version released September 2024, Accessed: 2025-01-XX. ↗
  22. Manisha Mukherjee and Vincent J Hellendoorn. 2025. Sosecure: Safer code generation with rag and stackoverflow discussions. arXiv preprint arXiv:2503.13654. — [arXiv:2503.13654](https://arxiv.org/abs/2503.13654) find
  23. Mahmoud Nazzal, Issa Khalil, Abdallah Khreishah, and NhatHai Phan. 2024. Promsec: Prompt optimization for secure generation of functional source code with large language models (llms). In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 2266–2280. find
  24. Ana Nunez, Nafis Tanveer Islam, Sumit Kumar Jha, and Peyman Najafirad. 2024. Autosafecoder: A multi-agent framework for securing llm code generation through static analysis and fuzz testing. arXiv preprint arXiv:2409.10737. — [arXiv:2409.10737](https://arxiv.org/abs/2409.10737) find
  25. Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641–649. find
  26. Xiangzhe Xu, Zian Su, Jinyao Guo, Kaiyuan Zhang, Zhenting Wang, and Xiangyu Zhang. 2024. Prosec: Fortifying code llms with proactive security alignment. arXiv preprint arXiv:2411.12882. — [arXiv:2411.12882](https://arxiv.org/abs/2411.12882) find
  27. Weichen Yu, Ravi Mangal, Terry Zhuo, Matt Fredrikson, and Corina S Pasareanu. 2025. A mixture of linear corrections generates secure code. arXiv preprint arXiv:2507.09508. — [arXiv:2507.09508](https://arxiv.org/abs/2507.09508) find
  28. Boyu Zhang, Tianyu Du, Junkai Tong, Xuhong Zhang, Kingsum Chow, Sheng Cheng, Xun Wang, and Jianwei Yin. 2024. Seccoder: Towards generalizable and robust secure code generation. arXiv preprint arXiv:2410.01488. find
  29. Jianguo Zhao, Yuqiang Sun, Cheng Huang, Chengwei Liu, YaoHui Guan, Yutong Zeng, and Yang Liu. 2025. Towards secure code generation with llms: A study on common weakness enumeration. IEEE Transactions on Software Engineering. find
  30. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, and 1 others. 2023a. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery a…
  31. Wenqing Zheng, SP Sharan, Ajay Kumar Jaiswal, Kevin Wang, Yihan Xi, Dejia Xu, and Zhangyang Wang. 2023b. Outline, then details: Syntactically guided coarse-to-fine code generation. In International Conference on Machine Learning, pages 42403–42419. PMLR. find
  32. OpenAI. 2024. Gpt-4o mini. https://platform.openai.com/docs/models/gpt-4o-mini. Accessed: 2025-01-XX. ↗
  33. Qwen. 2025. Qwen3-8b. https://huggingface.co/ Qwen/Qwen3-8B. ↗
  34. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, and 1 others. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.
  35. Semgrep. 2025. Semgrep. https://semgrep.dev/. Accessed: 2025-10. ↗
  36. Jiahao Shi and Tianyi Zhang. 2025. Rescue: Retrieval augmented secure code generation. arXiv preprint arXiv:2510.18204. — [arXiv:2510.18204](https://arxiv.org/abs/2510.18204) find
  37. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhong-2026-learning-failures-error-notebook-guided/