Elamur Research⌕
☰
home / papers / 2026 / Adaptive Instruction Composition for Automated LLM Red-Teaming

Adaptive Instruction Composition for Automated LLM Red-Teaming

Paper · 2026 · Jesse Zymet, Andy Luo, Swapnil Shinde et al.

Summary

Automated red-teaming uses an attacker LLM to find jailbreaks against a target model. Existing methods face a trade-off. Attackers that invent their own strategies by trial and error tend to settle on a narrow set of attack types. WildTeaming gets variety by filling its attacker prompt with harmful queries and tactics from the crowdsourced WildJailbreak corpus, but it pairs them randomly and never learns from earlier successes. The authors propose Adaptive Instruction Composition (AIC), which treats choosing a query–tactic combination as a contextual bandit problem. A small neural network trained with Neural Thompson Sampling learns to predict the evaluator's verdict for candidate compositions, which are represented by compressed sentence embeddings. It then samples compositions so that exploiting known weaknesses is balanced against further exploration. Compared with random combination, AIC scores substantially higher on both effectiveness and diversity metrics, and it keeps this advantage under model transfer. It also beats several recent adaptive methods on Harmbench. The work matters for safety teams that need broad, target-specific attack sets for safety training, since AIC runs in roughly the same time as WildTeaming.

Key points

  • Search space: Built on WildJailbreak data, which contains 50,500 vanilla harmful queries and 13,311 jailbreak tactics. Instructions combining one query with two tactics yield over 8 trillion possibilities.
  • Feature pipeline: Texts are embedded with the contrastively pretrained Sentence Transformer all-mpnet-base-v2 (768 dimensions) and reduced to 10 dimensions with UMAP. The embeddings of each composition's components are concatenated to form the bandit input. Ablations suggest the contrastive embeddings let the network generalize across this huge space.
  • Experimental setup: Mixtral-8x7B serves as attacker and Llama-Guard-2 as judge. Targets are Mistral-7B, Llama-3-70B-Instruct and Llama-3.3-70B-Instruct. Results were additionally checked with the Harmbench evaluator (Harmbench-Llama-2-13b-cls).
  • Compute cost: A training simulation takes about 10–12 hours, or roughly 70–120 GPU hours. A 10,000-trial run against Mistral-7B took about 10 hours, mainly because three LLMs are queried in sequence.
  • Limitations and future work: Only three open-source targets were tested, and the evaluators can produce false positives and negatives. The authors suggest richer instruction templates and adapting AIC to indirect prompt injection against RAG agents.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Capital OneTysons Corner · United States7 works · 16 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksBias, fairness & safety · 587 works

Cite

Cite as Jesse Zymet, Andy Luo, Swapnil Shinde, Sahil Wadhwa and Emily Chen. 2026. Adaptive Instruction Composition for Automated LLM Red-Teaming. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2174

@inproceedings{zymet2026adaptive,
  title = {{Adaptive Instruction Composition for Automated LLM Red-Teaming}},
  author = {Jesse Zymet, Andy Luo, Swapnil Shinde, Sahil Wadhwa and Emily Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2174},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2174}
}

References

55 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shipra Agrawal and Navin Goyal. 2013. Thompson sampling for contextual bandits with linear payoffs. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 127–135, Atlanta, Georgia, USA. PMLR. find
  2. Peter Auer. 2002. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397–422. find
  3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — Constitutional AI: Harmlessness from AI Feedback
  4. Mohsen Bayati, Nima Hamidi, Ramesh Johari, and Khashayar Khosravi. 2024. The unreasonable effectiveness of greedy algorithms in multi-armed bandit with many arms. find
  5. Alex Beutel, Kai Xiao, Johannes Heidecke, and Lilian Weng. 2024. Diverse and effective red teaming with auto-generated rewards and multi-step reinforcement learning. find
  6. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2024. Jailbreaking black box large language models in twenty queries.
  7. Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2024. Masterkey: Automated jailbreaking of large language model chatbots. In Proceedings 2024 Network and Distributed System Security Symposium, NDSS 2024. Internet Society. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding.
  9. Yi Dong, Ronghui Mu, Yanghao Zhang, Siqi Sun, Tianle Zhang, Changshun Wu, Gaojie Jin, Yi Qi, Jinwei Hu, Jie Meng, Saddek Bensalem, and Xiaowei Huang. 2024. Safeguarding large language models: A survey.
  10. Michael Feffer, Anusha Sinha, Wesley H Deng, Zachary C Lipton, and Hoda Heidari. 2024. Red-teaming for generative ai: Silver bullet or security theater? In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pages 421–437. find
  11. Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. 2010. Parametric bandits: The generalized linear case. Advances in neural information processing systems, 23. find
  12. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan… — Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
  13. Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, and Yuning Mao. 2023. Mart: Improving llm safety with multi-round automatic red-teaming. arXiv preprint arXiv:2311.07689.
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao,… — The Llama 3 Herd of Models
  15. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security,…
  16. Taehyun Hwang, Kyuwook Chai, and Min hwan Oh. 2023. Combinatorial neural bandits. find
  17. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations.
  18. Arthur Jacot, Franck Gabriel, and Clément Hongler. 2020. Neural tangent kernel: Convergence and generalization in neural networks. find
  19. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianmario Calafiore, Guillaume Bour, Guillaume Lample, Jean-Baptiste Bordes, Jose Cano, Lauri Sintonen, Madeleine-Clémence Gros, Matthieu Cord, Moïse Debbabi, Paul-Emm… find
  20. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  21. Bojian Jiang, Yi Jing, Tianhao Shen, Tong Wu, Qing Yang, and Deyi Xiong. 2024b. Automated progressive red teaming. arXiv preprint arXiv:2407.03876.
  22. Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. 2024c. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models.
  23. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. 2023. Pre-training language models with human preferences. find
  24. George Kour, Naama Zwerdling, Marcel Zalmanovici, Ateret Anaby-Tavor, Ora Nova Fandina, and Eitan Farchi. 2024. Exploring straightforward conversational red-teaming. find
  25. Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661–670. find
  26. Lizhi Lin, Honglin Mu, Zenan Zhai, Minghan Wang, Yuxia Wang, Renxi Wang, Junjie Gao, Yixuan Zhang, Wanxiang Che, Timothy Baldwin, et al. 2024. Against the achilles’ heel: A survey on red teaming for generative models. arXiv preprint arXiv:2404.00629. — [arXiv:2404.00629](https://arxiv.org/abs/2404.00629) find
  27. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. Wanli: Worker and ai collaboration for natural language inference dataset creation.
  28. Xiaogeng Liu, Peiran Li, Edward Suh, Yevgeniy Vorobeychik, Zhuoqing Mao, Somesh Jha, Patrick Mc-Daniel, Huan Sun, Bo Li, and Chaowei Xiao. 2024a. Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms. find
  29. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024b. Autodan: Generating stealthy jailbreak prompts on aligned large language models.
  30. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, Kailong Wang, and Yang Liu. 2024c. Jailbreaking chatgpt via prompt engineering: An empirical study.
  31. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  32. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.
  33. Leland McInnes, John Healy, and James Melville. 2020. Umap: Uniform manifold approximation and projection for dimension reduction.
  34. Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2023. Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119.
  35. Lingbo Mo, Zeyi Liao, Boyuan Zheng, Yu Su, Chaowei Xiao, and Huan Sun. 2024. A trembling house of cards? mapping adversarial attacks against language agents. find
  36. John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp.
  37. Gary D. Lopez Munoz, Amanda J. Minnich, Roman Lutz, Richard Lundeen, Raja Sekhar Rao Dheekonda, Nina Chikanov, Bolor-Erdene Jagdagdorj, Martin Pouliot, Shiven Chawla, Whitney Maxwell, Blake Bullwinkel, Katherine Pratt, Joris de Gruyter, Charlotte Siska, Pete Bryan, Tori Westerhoff, Chang Kawaguchi,… find
  38. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  39. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. arXiv preprint arXiv:2202.03286.
  40. Alberto Purpura, Sahil Wadhwa, Jesse Zymet, Akshay Gupta, Andy Luo, Melissa Kazemi Rad, Swapnil Shinde, and Mohammad Shahed Sorower. 2025. Building safe genai applications: An end-to-end overview of red teaming for large language models. find
  41. Ambrish Rawat, Stefan Schoepf, Giulio Zizzo, Giandomenico Cornacchia, Muhammad Zaid Hameed, Kieran Fraser, Erik Miehling, Beat Buesser, Elizabeth M Daly, Mark Purcell, et al. 2024. Attack atlas: A practitioner’s perspective on challenges and pitfalls in red teaming genai. arXiv preprint arXiv:2409.1… — [arXiv:2409.15398](https://arxiv.org/abs/2409.15398) find
  42. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  43. Elias Abad Rocamora, Yongtao Wu, Fanghui Liu, Grigorios G. Chrysos, and Volkan Cevher. 2024. Revisiting character-level adversarial attacks for language models. find
  44. Daniel Russo and Benjamin Van Roy. 2020. Satisficing in time-sensitive bandit learning. find
  45. Chantal Shaib, Joe Barrow, Jiuding Sun, Alexa F. Siu, Byron C. Wallace, and Ani Nenkova. 2025. Standardizing the measurement of text diversity: A tool and a comparative analysis of scores. find
  46. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1671–1685.
  47. Guy Tevet and Jonathan Berant. 2021. Evaluating the evaluation of diversity in natural language generation.
  48. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2021. Universal adversarial triggers for attacking and analyzing nlp. find
  49. Yizao Wang, Jean-Yves Audibert, and Rémi Munos. 2008. Algorithms for infinitely many-armed bandits. Advances in Neural Information Processing Systems, 21. find
  50. Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. 2024. A comprehensive study of jailbreak attack versus defense for large language models. find
  51. Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. 2024. Low-resource languages jailbreak gpt-4.
  52. Tianhui Zhang, Bei Peng, and Danushka Bollegala. 2024. Improving diversity of commonsense generation by large language models via in-context learning. find
  53. Weitong Zhang, Dongruo Zhou, Lihong Li, and Quanquan Gu. 2021. Neural thompson sampling. find
  54. Dongruo Zhou, Lihong Li, and Quanquan Gu. 2020. Neural contextual bandits with ucb-based exploration. find
  55. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zymet-2026-adaptive-instruction-composition-automated-llm/