Elamur Research⌕
☰
home / papers / 2024 / FLIRT: Feedback Loop In-context Red Teaming

FLIRT: Feedback Loop In-context Red Teaming

Paper · 2024 · Ninareh Mehrabi, Palash Goyal, Christophe Dupuy et al.

Summary

The paper tackles automated red teaming: finding inputs that push generative models into producing unsafe content, without expensive human probing. The authors present FLIRT, a black-box framework where a red language model writes attack prompts through in-context learning. It starts from a handful of human-written seed examples. Each candidate prompt goes to a target model, and the output is judged by safety classifiers (Q16 and NudeNet for images, TOXIGEN for text). When an attack succeeds, the red LM revises its exemplar set using one of several strategies: FIFO queue, LIFO stack, a weighted scoring objective (effectiveness, diversity, low toxicity), or a Scoring-LIFO hybrid. Against Stable Diffusion, including safety-enhanced versions, FLIRT succeeds far more often than the stochastic few-shot (SFS) in-context baseline of Perez et al. (2022). Its prompts also transfer between models and can be tuned to slip past toxicity filters. The work shows that cheap in-context feedback is enough to expose weaknesses in deployed safety mechanisms.

Key points

  • Average attack success is about 80% on vanilla Stable Diffusion and 60% on safe SD variants. The Perez et al. (2022) baseline reaches roughly 30% and 20%.
  • The scoring strategy optimizes a weighted sum of objectives over the exemplar list. Limiting prompt toxicity lets attacks evade content-moderation filters.
  • The findings hold when 5%, 10% or 20% of classifier labels are randomly flipped. Scoring stays the most effective strategy, while SFS, LIFO and Scoring-LIFO produce more diverse prompts.
  • On the GPT-Neo 2.7B text model, the share of toxic outputs is 52.4% for Scoring-LIFO, 50.9% for Scoring, 46.2% for LIFO, 38.8% for FIFO and only 9.9% for SFS.
  • Code is released at github.com/amazon-science/FLIRT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 517 works

Cite

Cite as Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan and Rahul Gupta. 2024. FLIRT: Feedback Loop In-context Red Teaming. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.41

@inproceedings{mehrabi2024flirt,
  title = {{FLIRT: Feedback Loop In-context Red Teaming}},
  author = {Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan and Rahul Gupta},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.41},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.41}
}

References

14 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-t… find
  2. Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata. find
  3. Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. 2023. Explore, establish, exploit: Red teaming language models from scratch. arXiv preprint arXiv:2306.09442.
  4. Nouha Dziri, Andrea Madotto, Osmar Zaïane, and Avishek Joey Bose. 2021. Neural path hunter: Reducing hallucination in dialogue systems via path grounding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2197–2214, Online and Punta Cana, Do-Deep Gangul…
  5. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. find
  6. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. The magazine archive includes every article published in communications of the acm for over the past 50 years. Communications of the ACM, 64(12):86–92. find
  7. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:…
  8. Deokjae Lee, JunYeong Lee, Jung-Woo Ha, Jin-Hwa Kim, Sang-Woo Lee, Hwaran Lee, and Hyun Oh Song. 2023. Query-efficient black-box red teaming via Bayesian optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11551– 115… find
  9. Ninareh Mehrabi, Ahmad Beirami, Fred Morstatter, and Aram Galstyan. 2022. Robust conversational agents against imperceptible toxicity triggers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 283… find
  10. Pamela Mishkin, Lama Ahmad, Miles Brundage, Gretchen Krueger, and Girish Sastry. 2022. Dall·e 2 preview - risks and limitations. find
  11. OpenAI. 2023. Gpt-4 technical report.
  12. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419–3448, Abu…
  13. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125.
  14. Javier Rando, Daniel Paleka, David Lindner, Lennard Heim, and Florian Tramèr. 2022. Red-teaming the stable diffusion safety filter. arXiv preprint arXiv:2210.04610. — [arXiv:2210.04610](https://arxiv.org/abs/2210.04610) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mehrabi-2024-flirt-feedback-loop-context-red/