Elamur Research⌕
☰
home / papers / 2026 / SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning

SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning

Paper · 2026 · Kaiwen Zhou, Ahmed Elgohary, A S M Iftekhar et al.

Summary

LLM agents that plan and call tools can cause real-world harm, but current safety evaluations have two gaps. They test only broad risk categories, and existing red-teaming tools either need white-box access or target narrow threat types. SIRAJ is a red-teaming framework that needs only black-box access to the agent. It works in two stages. First, given an agent's tool definitions, it lists fine-grained risk outcomes within each safety category. It then prompts a generator for seed test cases that differ in expected tool-call trajectory and risk source (malicious user or malicious environment). Second, a red-teamer model rewrites each seed into adversarial variants over several rounds, using the agent's execution traces as feedback. To make this cheaper, the authors reformat a large teacher model's reasoning into a concise structured form. They then train Qwen3-8B on successful attacks, first with SFT and then with RL. The distilled 8B model roughly doubles its attack success rate and slightly beats DeepSeek-R1 (671B) while producing far fewer tokens. This makes thorough agent safety testing more affordable.

Key points

  • The seed generation step gives about 2x more diverse tool-calling trajectories and about 2.5x more diverse risk outcomes than baseline prompting.
  • Distillation improves the attack success rate (ASR) of Qwen3-8B by more than 100%:
    • +25.8% ASR over the untrained Qwen-8B baseline and +6.0% over R1.
    • RL after SFT adds a further 3.1% average ASR.
  • Structured reasoning helps in two ways: it raises the success of training-free red-teamers, and it beats distillation from raw R1 reasoning traces.
  • On efficiency, the distilled red-teamer outputs about 3k tokens per seed case, compared with roughly 9–11k for R1 or Structured-R1, using at most K = 3 refinement iterations.
  • The model was trained only on GPT-5-mini-based agents, yet it transfers to other agent backbones, including safer ones such as Safe o4-mini and Safe GPT-5-mini.
    • It also improves ASR by more than 20% on agents with unseen tools.
    • However, it still trails R1 on agents whose tools are all unseen.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksBias, fairness & safety · 587 works

Cite

Cite as Kaiwen Zhou, Ahmed Elgohary, A S M Iftekhar and Amin Saied. 2026. SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.171

@inproceedings{zhou2026siraj,
  title = {{SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning}},
  author = {Kaiwen Zhou, Ahmed Elgohary, A S M Iftekhar and Amin Saied},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.171},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.171}
}

References

29 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, et al. 2024. Agentharm: A benchmark for measuring harmfulness of llm agents. arXiv preprint arXiv:2410.09024. — [arXiv:2410.09024](https://arxiv.org/abs/2410.09024) find
  2. Alex Beutel, Kai Xiao, Johannes Heidecke, and Lilian Weng. 2024. Diverse and effective red teaming with auto-generated rewards and multi-step reinforcement learning. arXiv preprint arXiv:2412.18693. — [arXiv:2412.18693](https://arxiv.org/abs/2412.18693) find
  3. Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. 2024. Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases. Advances in Neural Information Processing Systems, 37:130185–130213. find
  4. Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. Agentdojo: A dynamic environment to evaluate attacks and defenses for llm agents. arXiv preprint arXiv:2406.13352. — [arXiv:2406.13352](https://arxiv.org/abs/2406.13352) find
  5. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. In NeurIPS Datasets and Benchmarks Track.
  6. Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, and Yuning Mao. 2023. Mart: Improving llm safety with multi-round automatic red-teaming. arXiv preprint arXiv:2311.07689.
  7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025a. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  8. Weiyang Guo, Zesheng Shi, Zhuo Li, Yequan Wang, Xuebo Liu, Wenya Wang, Fangming Liu, Min Zhang, and Jing Li. 2025b. Jailbreak-r1: Exploring the jailbreak capabilities of llms via reinforcement learning. arXiv preprint arXiv:2506.00782. — [arXiv:2506.00782](https://arxiv.org/abs/2506.00782) find
  9. Xiaogeng Liu, Peiran Li, Edward Suh, Yevgeniy Vorobeychik, Zhuoqing Mao, Somesh Jha, Patrick McDaniel, Huan Sun, Bo Li, and Chaowei Xiao. 2024. Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms. arXiv preprint arXiv:2410.05295. — [arXiv:2410.05295](https://arxiv.org/abs/2410.05295) find
  10. Yanjiang Liu, Shuhen Zhou, Yaojie Lu, Huijia Zhu, Weiqiang Wang, Hongyu Lin, Ben He, Xianpei Han, and Le Sun. 2025. Auto-rt: Automatic jailbreak strategy exploration for red-teaming large language models. arXiv preprint arXiv:2501.01830. — [arXiv:2501.01830](https://arxiv.org/abs/2501.01830) find
  11. Yingning Ma. 2025. Realsafe: Quantifying safety risks of language agents in real-world. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9586–9617. find
  12. Jingyi Yang, Shuai Shao, Dongrui Liu, and Jing Shao. 2025b. Riosworld: Benchmarking the risk of multimodal compter-use agents. arXiv preprint arXiv:2506.00618. — [arXiv:2506.00618](https://arxiv.org/abs/2506.00618) find
  13. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2023. Webshop: Towards scalable real-world web interaction with grounded language agents. In NeurIPS.
  14. OpenAI. 2025a. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/. System Card; accessed 2025-08-21. ↗
  15. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. arXiv preprint arXiv:2403.02691.
  16. OpenAI. 2025b. OpenAI o3 and o4-mini System Card. https://openai.com/index/ o3-o4-mini-system-card/. System Card; accessed 2025-08-21. ↗
  17. Jiawei Zhang, Shuang Yang, and Bo Li. 2025. Udora: A unified red teaming framework against llm agents by dynamically hijacking their own reasoning. arXiv preprint arXiv:2503.01908. — [arXiv:2503.01908](https://arxiv.org/abs/2503.01908) find
  18. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789.
  19. Yanzhe Zhang, Tao Yu, and Diyi Yang. 2024a. Attacking vision-language computer agents via pop-ups. arXiv preprint arXiv:2411.02391.
  20. Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, et al. 2024. Rainbow teaming: Open-ended generation of diverse adversarial prompts. Advances in Neural Information Processing Systems, 37:… find
  21. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  22. Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. 2024b. Agent-safetybench: Evaluating the safety of llm agents. arXiv preprint arXiv:2412.14470. — [arXiv:2412.14470](https://arxiv.org/abs/2412.14470) find
  23. Andy Zhou, Kevin Wu, Francesco Pinto, Zhaorun Chen, Yi Zeng, Yu Yang, Shuang Yang, Sanmi Koyejo, James Zou, and Bo Li. 2025. Autoredteamer: Autonomous red teaming with lifelong attack integration. arXiv preprint arXiv:2503.15754. — [arXiv:2503.15754](https://arxiv.org/abs/2503.15754) find
  24. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
  25. Ada Defne Tur, Nicholas Meade, Xing Han Lù, Alejandra Zambrano, Arkil Patel, Esin Durmus, Spandana Gella, Karolina Stańczak, and Siva Reddy. 2025. Safearena: Evaluating the safety of autonomous web agents. arXiv preprint arXiv:2503.04957. — [arXiv:2503.04957](https://arxiv.org/abs/2503.04957) find
  26. Sanidhya Vijayvargiya, Aditya Bharat Soni, Xuhui Zhou, Zora Zhiruo Wang, Nouha Dziri, Graham Neubig, and Maarten Sap. 2025. Openagentsafety: A comprehensive framework for evaluating real-world ai agent safety. arXiv preprint arXiv:2507.06134. find
  27. Zhun Wang, Vincent Siu, Zhe Ye, Tianneng Shi, Yuzhou Nie, Xuandong Zhao, Chenguang Wang, Wenbo Guo, and Dawn Song. 2025. Agentvigil: Generic black-box red-teaming for indirect prompt injection against llm agents. arXiv preprint arXiv:2505.05849. find
  28. Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, and Bo Li. 2024. Advagent: Controllable blackbox red-teaming on web agents. arXiv preprint arXiv:2410.17401. — [arXiv:2410.17401](https://arxiv.org/abs/2410.17401) find
  29. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-siraj-diverse-efficient-red-teaming/