Elamur Research⌕
☰
home / papers / 2026 / Towards Knowledge Graph-Grounded Evaluation of Agentic LLMs on Cybersecurity Capture-the-Flag Challenges

Towards Knowledge Graph-Grounded Evaluation of Agentic LLMs on Cybersecurity Capture-the-Flag Challenges

Paper · 2026 · Daniel Schlör, Marius Bohn, Maximilian Wolf et al.

Summary

Agentic LLMs for offensive security are usually scored on whether they capture the flag, or against ad hoc checkpoints defined per challenge. These methods say little about partial progress or about valid alternative solutions. The authors address this with BraceGreen, which turns human-written CTF writeups into knowledge graphs that serve as gold-standard rubrics.

Each attack step is a node tagged with:

  • a MITRE ATT&CK tactic
  • a natural-language goal
  • a reference command and its expected output
  • a semantic result

Edges capture attack order, equivalent alternative commands or multi-command workflows, data-flow prerequisites, and conditions that make a command a poor choice.

A LangGraph workflow uses an LLM-as-judge with chain-of-thought reasoning. The judge checks an agent's prediction against every alternative the graph encodes, not just one reference answer.

The released resource covers seven VulnHub machines and connects to live targets through virtual machines and an MCP server. The excerpt contains no quantitative evaluation results. Its main value is a standardized, reproducible schema for fine-grained, step-level assessment of security agents, independent of the agent framework used.

Key points

  • Benchmark: 7 VulnHub CTF machines with full knowledge graph annotations, released at github.com/LSX-UniWue/brace-ctf-data.
  • Graph schema: RDF-compatible, with three node types (Step, Command, Command group).
    • Structural edges: next_step, or and has_subcmd.
    • Relations: has_tactic, has_goal, has_cmd, has_output, has_result, requires, contraindicated_by and is_optional.
  • Tactic coverage: steps map to 10 MITRE ATT&CK tactic categories, from Reconnaissance and Discovery through Privilege Escalation, Lateral Movement and Command and Control.
  • Evaluation modes: three modes (command prediction, goal inference, anticipated result), each scored by an LLM-as-judge inside a LangGraph pipeline.
  • Positioning: unlike binary benchmarks (NYU CTF Bench), milestone-based ones (AutoPenBench, Cybench) or rubric trees (PentestJudge), BraceGreen encodes alternative solution paths and prerequisites explicitly. It is limited to technical attack steps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WürzburgWürzburg · Germany8 works · 60 citations

Also on the byline: Coburg University of Applied Sciences.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 worksComputer Networks and Communications · 64 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksKnowledge graphs & ontologies · 192 works

Cite

Cite as Daniel Schlör, Marius Bohn, Maximilian Wolf, Kevin Bergner, Christian Goldschmied and Andreas Hotho. 2026. Towards Knowledge Graph-Grounded Evaluation of Agentic LLMs on Cybersecurity Capture-the-Flag Challenges. Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) @ LREC26.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4ne74wm45iti

@inproceedings{schlor2026knowledge,
  title = {{Towards Knowledge Graph-Grounded Evaluation of Agentic LLMs on Cybersecurity Capture-the-Flag Challenges}},
  author = {Daniel Schlör, Marius Bohn, Maximilian Wolf, Kevin Bergner, Christian Goldschmied and Andreas Hotho},
  booktitle = {Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) @ LREC26},
  year = {2026},
  doi = {10.63317/4ne74wm45iti},
  url = {https://doi.org/10.63317/4ne74wm45iti}
}

References

31 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang, Carlos E. Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press. 2025. Enigma: Interactive tool… find
  2. Razvan Beuran. 2025. Capture the flag platforms. In Cybersecurity Education and Training, pages 193–219. Springer. find
  3. Shane Caldwell, Max Harley, Michael Kouremetis, Vincent Abruzzo, and Will Pearce. 2025. PentestJudge: Judging Agent Behavior Against Operational Requirements. ArXiv:2508.02921 [cs]. — [arXiv:2508.02921](https://arxiv.org/abs/2508.02921) find
  4. Brian Challita and Pierre Parrend. 2025. RedTeam-LLM: an Agentic AI framework for offensive security. ArXiv:2505.06913 [cs]. — [arXiv:2505.06913](https://arxiv.org/abs/2505.06913) find
  5. Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. 2024. PentestGPT: Evaluating and harnessing large language models for automated penetration testing. In 33rd USENIX Security Symposium (USENIX Security 24), pages 847–… find
  6. Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. 2024. Llm agents can autonomously exploit one-day vulnerabilities. arXiv preprint arXiv:2404.08144. — [arXiv:2404.08144](https://arxiv.org/abs/2404.08144) find
  7. Rikhiya Ghosh, Hans-Martin von Stockhausen, Martin Schmitt, George Marica Vasile, Sanjeev Kumar Karn, and Oladimeji Farri. 2025. Cve-llm: Ontology-assisted automatic vulnerability evaluation using large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pag… find
  8. Yasod Ginige, Akila Niroshan, Sajal Jain, and Suranga Seneviratne. 2025. AutoPentester: An LLM Agent-based Framework for Automated Pentesting. In 2025 IEEE 24th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), pages 163–174, Guiyang, China. IEEE. find
  9. Luca Gioacchini, Marco Mellia, Idilio Drago, Alexander Delsanto, Giuseppe Siracusano, and Roberto Bifulco. 2024. Autopenbench: Benchmarking generative agents for penetration testing. arXiv preprint arXiv:2410.03225. — [arXiv:2410.03225](https://arxiv.org/abs/2410.03225) find
  10. Zimo Ji, Daoyuan Wu, Wenyuan Jiang, Pingchuan Ma, Zongjie Li, and Shuai Wang. 2025. Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, pages 603–617, Taipei Taiwan. ACM. find
  11. Mudita Khurana and Raunak Jain. 2025. SoK: Measuring What Matters for Closed-Loop Security Agents. ArXiv:2510.01654 [cs]. — [arXiv:2510.01654](https://arxiv.org/abs/2510.01654) find
  12. He Kong, Die Hu, Jingguo Ge, Liangxiong Li, Tong Li, and Bingzhen Wu. 2025. Vulnbot: Autonomous penetration testing for a multi-agent collaborative framework. arXiv preprint arXiv:2501.13411. — [arXiv:2501.13411](https://arxiv.org/abs/2501.13411) find
  13. Knowledge Graph from Cyber Threat Intelligence Reports. ArXiv:2111.07093 [cs]. — [arXiv:2111.07093](https://arxiv.org/abs/2111.07093) find
  14. Bruno Lourenço, Pedro Adão, João F. Ferreira, Mario Monteiro Marques, and Cátia Vaz. 2025. Structuring Security: A Survey of Cybersecurity Ontologies, Semantic Log Processing, and LLMs Application. ArXiv:2510.16610 [cs]. — [arXiv:2510.16610](https://arxiv.org/abs/2510.16610) find
  15. Haokai Ma, Javier Yong, Yunshan Ma, Kuei Chen, Anis Yusof, Zhenkai Liang, and Ee-Chien Chang. 2025. AttackSeqBench: Benchmarking Large Language Models in Analyzing Attack Sequences within Cyber Threat Intelligence. ArXiv:2503.03170 [cs]. — [arXiv:2503.03170](https://arxiv.org/abs/2503.03170) find
  16. MITRE Corporation. 2023. MITRE ATT&CK Framework. https://attack.mitre.org/. Accessed: 2026-02-27. ↗
  17. Alessandro Palma and Marco Angelini. 2024. It is time to steer: A scalable framework for analysis-driven attack graph generation. In European Symposium on Research in Computer Security, pages 229–250. Springer. find
  18. RDI Foundation. 2024. AgentBeats: Agent-to-Agent Protocol for Multi-Agent Systems. https://github.com/RDI-Foundation/ AgentBeats. ↗
  19. Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, et al. 2024. Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security. Advances in Neural Info… find
  20. Minghao Shao, Nanda Rani, Kimberly Milner, Haoran Xi, Meet Udeshi, Saksham Aggarwal, Venkata Sai Charan Putrevu, Sandeep K Shukla, Prashanth Krishnamurthy, Farshad Khorrami, et al. 2026. Towards effective offensive security llm agents: Hyperparameter tuning, llm as a judge, and a lightweight ctf ben… find
  21. Alyzia-Maria Konsta, Alberto Lluch Lafuente, Beatrice Spiga, and Nicola Dragoni. 2024. Survey: Automatic generation of attack trees and attack graphs. Computers & Security, 137:103602. find
  22. Minghao Shao, Nanda Rani, Kimberly Milner, Haoran Xi, Meet Udeshi, Saksham Aggarwal, Venkata Sai Charan Putrevu, Sandeep Kumar Shukla, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. 2025. Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as… — [arXiv:2508.05674](https://arxiv.org/abs/2508.05674) find
  23. Zhenyuan Li, Jun Zeng, Yan Chen, and Zhenkai Liang. 2022. find
  24. AttacKG: Constructing Technique Xiangmin Shen, Lingzhi Wang, Zhenyuan Li, Yan Chen, Wencheng Zhao, Dawei Sun, Jiashui Wang, and Wei Ruan. 2025. Pentestagent: Incorporating llm agents to automated penetration testing. In Proceedings of the 20th ACM Asia Conference on Computer and Communications Secur… find
  25. Leslie F. Sikos. 2023. Cybersecurity knowledge graphs. Knowledge and Information Systems, 65(9):3511–3531. find
  26. VulnHub. 2026. VulnHub: Vulnerable By Design. https://www.vulnhub.com/. Accessed: 2026-02-27. ↗
  27. Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, et al. 2024. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. arXiv preprint arXiv:2408.08926. find
  28. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.
  29. Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, et al. 2025. Cve-bench: a benchmark for ai agents’ ability to exploit real-world web application vulnerabilities. arXiv preprint arXiv:2503.17332. — [arXiv:2503.17332](https://arxiv.org/abs/2503.17332) find
  30. Yuxuan Zhu, Antony Kellermann, Akul Gupta, Philip Li, Richard Fang, Rohan Bindu, and Daniel Kang. 2024. Teams of llm agents can exploit zero-day vulnerabilities. arXiv preprint arXiv:2406.01637.
  31. Yuwen Zou, Jia Liu, and Wenjun Fan. 2026. Ctfa-gent: An llm-powered agent for ctf challenge solving. Journal of Information Security and Applications, 96:104305. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schlor-2026-knowledge-graph-grounded-evaluation-agentic/