Elamur Research⌕
☰
home / papers / 2026 / Red-Teaming NSFW Image Classifiers as Text-to-Image Safeguards

Red-Teaming NSFW Image Classifiers as Text-to-Image Safeguards

Paper · 2026 · Tinghao Xie, Yueqi Xie, Alireza Zareian et al.

Summary

This paper examines a weakness in NSFW image classifiers that act as output filters for text-to-image (T2I) generators. The authors call it "context shifts": adding harmless surrounding elements to an explicit image can stop the filter from flagging it. An example is a nude figure placed among clothed people. The authors build an automated red-teaming pipeline in two stages. In the exploration stage, an LLM pads short unsafe seed prompts with varied benign details, and an open-weight image generator renders them. This produces a synthetic dataset of about 36K images covering nudity and violence. In the exploitation stage, an MLLM captions the images that classifiers wrongly passed as safe. An LLM is then fine-tuned to map seed prompts to these captions, so it learns to rewrite new prompts into versions that evade detection. Strong classifiers such as GPT-4o and Gemini miss a noticeable share of the dataset, and the attacks carry over to commercial T2I and text-to-video services. The work shows that output classifiers can give a misleading sense of safety. It also suggests a fix: retrain filters on the failure cases it uncovers.

Key points

  • The synthetic dataset has roughly 36K NSFW images (nude/sexual and violent). Depending on the classifier, 4.1% to 36.2% of the nude/sexual images were labeled "Safe". Classifiers studied include Q16, LlavaGuard, NudeNet, GPT-4o and Gemini.
  • Prompts produced by the fine-tuned rewriter LLM make detection evasion up to about 6 times more likely than the original seed prompts.
  • Against real systems, rewritten prompts obtained nude/sexual images from DALL-E 3 and Nano Banana in 40–53% of attempts. For DALL-E 3, the rate rose from 0 to over 50%, while plain seed prompts reached only 0–10%. Similar failures appeared in Firefly, Sora and Veo 3.
  • As a mitigation, the authors adapted a Llama-3.2-Vision model as a classifier. Fine-tuning it on the misclassified images reduced failures under context shifts, though the authors note possible false-positive trade-offs.
  • Stated limitations:
    • Only two harm categories were studied.
    • The data is fully synthetic.
    • The "Safe" rate is used as a proxy for errors, which may overstate failures.
    • For safety reasons, only seed prompts are released, not adversarial ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 811 worksBias, fairness & safety · 587 works

Cite

Cite as Tinghao Xie, Yueqi Xie, Alireza Zareian, Shuming Hu, Felix Juefei-Xu, Xiaowen Lin, Ankit Jain, Prateek Mittal and Li Chen. 2026. Red-Teaming NSFW Image Classifiers as Text-to-Image Safeguards. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.506

@inproceedings{xie2026red,
  title = {{Red-Teaming NSFW Image Classifiers as Text-to-Image Safeguards}},
  author = {Tinghao Xie, Yueqi Xie, Alireza Zareian, Shuming Hu, Felix Juefei-Xu, Xiaowen Lin, Ankit Jain, Prateek Mittal and Li Chen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.506},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.506}
}

References

62 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adobe. 2025. Adobe firefly. find
  2. Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. 2020. Square attack: a query-efficient black-box adversarial attack via random search. In European conference on computer vision, pages 484–501. Springer. find
  3. Anish Athalye, Nicholas Carlini, and David Wagner. 2018. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning, pages 274–283. PMLR. find
  4. Abeba Birhane and Vinay Uday Prabhu. 2021. Large image datasets: A pyrrhic win for computer vision? In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1536–1546. IEEE. find
  5. Black Forest Labs. 2024a. Announcing black forest labs. find
  6. Black Forest Labs. 2024b. Flux.1-dev. find
  7. Elias Bouzidi. 2024. Distilbert nsfw text classifier. find
  8. Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. 2023. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM transactions on Graphics (TOG), 42(4):1–10. find
  9. Zhaoyu Chen, Bo Li, Shuang Wu, Kaixun Jiang, Shouhong Ding, and Wenqiang Zhang. 2024. Content-based unrestricted adversarial attack. Advances in Neural Information Processing Systems, 36. find
  10. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  11. CompVis. 2022. Stable diffusion safety checker model card. find
  12. Anudeep Das, Vasisht Duddu, Rui Zhang, and N Asokan. 2024. Espresso: Robust concept filtering in text-to-image models. arXiv preprint arXiv:2404.19227. — [arXiv:2404.19227](https://arxiv.org/abs/2404.19227) find
  13. Yingkai Dong, Zheng Li, Xiangtao Meng, Ning Yu, and Shanqing Guo. 2024. Jailbreaking text-to-image models with llm-based agents. Preprint, arXiv:2408.00523. — [arXiv:2408.00523](https://arxiv.org/abs/2408.00523) find
  14. Falcons.ai. 2023. Fine-tuned vision transformer (vit) for nsfw image classification. find
  15. Rojit George. 2020. Nsfw words list. find
  16. Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016. Deep learning, volume 1. MIT Press. find
  17. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
  18. Google. 2025a. Break the silence with veo 3.1. find
  19. Google. 2025b. Nano banana - gemini ai image generator & photo editor. find
  20. Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. 2019. Simple black-box adversarial attacks. In International conference on machine learning, pages 2484–2493. PMLR. find
  21. Lukas Helff, Felix Friedrich, Manuel Brack, Kristian Kersting, and Patrick Schramowski. 2024. Llavaguard: Vlm-based safeguards for vision dataset curation and safety assessment. arXiv preprint arXiv:2406.05113. — [arXiv:2406.05113](https://arxiv.org/abs/2406.05113) find
  22. Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. 2021. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15262–15271. find
  23. Michelle Li. 2022. Fine-tuned distilroberta-base for nsfw classification. find
  24. Xinfeng Li, Yuchen Yang, Jiangyi Deng, Chen Yan, Yanjiao Chen, Xiaoyu Ji, and Wenyuan Xu. 2024. Safegen: Mitigating sexually explicit content generation in text-to-image models. Preprint, arXiv:2404.06666. — [arXiv:2404.06666](https://arxiv.org/abs/2404.06666) find
  25. Han Liu, Yuhao Wu, Shixuan Zhai, Bo Yuan, and Ning Zhang. 2023a. Riatig: Reliable and imperceptible adversarial text-to-image generation with natural prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20585–20594. find
  26. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. In NeurIPS.
  27. Runtao Liu, Ashkan Khakzar, Jindong Gu, Qifeng Chen, Philip Torr, and Fabio Pizzati. 2024. Latent guard: a safety framework for text-to-image generation. arXiv preprint arXiv:2404.08031. — [arXiv:2404.08031](https://arxiv.org/abs/2404.08031) find
  28. Jiachen Ma, Anda Cao, Zhiqing Xiao, Jie Zhang, Chao Ye, and Junbo Zhao. 2024. Jailbreaking prompt attack: A controllable adversarial attack against diffusion models. arXiv preprint arXiv:2404.02928. find
  29. Hossein Hosseini and Radha Poovendran. 2018. Semantic adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1614–1619. find
  30. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations.
  31. Yihao Huang, Le Liang, Tianlin Li, Xiaojun Jia, Run Wang, Weikai Miao, Geguang Pu, and Yang Liu. 2024. Perception-guided jailbreak against text-to-image models. arXiv preprint arXiv:2408.10848. — [arXiv:2408.10848](https://arxiv.org/abs/2408.10848) find
  32. Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. 2022. Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460. — [arXiv:2205.07460](https://arxiv.org/abs/2205.07460) find
  33. Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. 2019. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32. find
  34. Haibo Jin, Peiyan Zhang, Man Luo, and Haohan Wang. 2025. Reasoning can hurt the inductive abilities of large language models. arXiv preprint arXiv:2505.24225. — [arXiv:2505.24225](https://arxiv.org/abs/2505.24225) find
  35. Duo Peng, Qiuhong Ke, and Jun Liu. 2024. Upam: Unified prompt attack in text-to-image generation models against both textual filters and visual checkers. arXiv preprint arXiv:2405.11336. — [arXiv:2405.11336](https://arxiv.org/abs/2405.11336) find
  36. Ameya Joshi, Amitangshu Mukherjee, Soumik Sarkar, and Chinmay Hegde. 2019. Semantic adversarial attacks: Parametric transformations that fool deep classifiers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4773– 4783. find
  37. Minh Pham, Kelly O Marshall, Niv Cohen, Govind Mittal, and Chinmay Hegde. 2023. Circumventing concept erasure methods for text-to-image generative models. In The Twelfth International Conference on Learning Representations. find
  38. Alex Kim. 2022. Nsfw data scraper. find
  39. Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. 2018. Adversarial examples in the physical world. In Artificial intelligence safety and security, pages 99–112. Chapman and Hall/CRC. find
  40. LAION. 2023. Clip-based-nsfw-detector. find
  41. OpenAI. 2024a. Gpt-4o system card.
  42. Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2024. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations.
  43. Bedapudi Praneeth. 2024. Nudenet: lightweight nudity detection. find
  44. Yiting Qu, Xinyue Shen, Yixin Wu, Michael Backes, Savvas Zannettou, and Yang Zhang. 2024. Unsafebench: Benchmarking image safety classifiers on real-world and ai-generated images. Preprint, arXiv:2405.03486. — [arXiv:2405.03486](https://arxiv.org/abs/2405.03486) find
  45. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8…
  46. Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr. 2022. Red-teaming the stable diffusion safety filter. arXiv preprint arXiv:2210.04610. — [arXiv:2210.04610](https://arxiv.org/abs/2210.04610) find
  47. Naquee Rizwan, Paramananda Bhaskar, Mithun Das, Swadhin Satyaprakash Majhi, Punyajoy Saha, and Animesh Mukherjee. 2024. Zero shot vlms for hate meme detection: Are we there yet? arXiv preprint arXiv:2402.12198. — [arXiv:2402.12198](https://arxiv.org/abs/2402.12198) find
  48. Patrick Schramowski, Christopher Tauchmann, and Kristian Kersting. 2022. Can machines help us answering question 16 in datasheets, and in turn reflecting on inappropriate content? In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1350–1361. find
  49. Ali Shahin Shamsabadi, Ricardo Sanchez-Matilla, and Andrea Cavallaro. 2020. Colorfool: Semantic adversarial colorization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1151–1160. find
  50. Michelle Shu, Chenxi Liu, Weichao Qiu, and Alan Yuille. 2020. Identifying model weakness with adversarial examiner. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11998–12006. find
  51. C Szegedy. 2013. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199.
  52. Florian Tramer. 2022. Detecting adversarial examples is (nearly) as hard as classifying them. In International Conference on Machine Learning, pages 21692–21702. PMLR. find
  53. Yu-Lin Tsai, Chia-Yi Hsu, Chulin Xie, Chih-Hsun Lin, Jia-You Chen, Bo Li, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. 2023. Ring-a-bell! how reliable are concept removal methods for diffusion models? arXiv preprint arXiv:2310.10012. — [arXiv:2310.10012](https://arxiv.org/abs/2310.10012) find
  54. Victor Wang, Michael JQ Zhang, and Eunsol Choi. 2025. Improving LLM-as-a-judge inference with the judgment distribution. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 23173–23199, Suzhou, China. Association for Computational Linguistics. find
  55. Eric Wong, Leslie Rice, and J Zico Kolter. 2020. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994. — [arXiv:2001.03994](https://arxiv.org/abs/2001.03994) find
  56. Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. 2018. Generating adversarial examples with adversarial networks. arXiv preprint arXiv:1801.02610. find
  57. Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Prateek Mittal. 2025. SORRY-bench: Systematically evaluating large language model safety refusal. In The Thi…
  58. Yijun Yang, Ruiyuan Gao, Xiaosen Wang, Tsung-Yi Ho, Nan Xu, and Qiang Xu. 2024a. Mma-diffusion: Multimodal attack on diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7737–7746. find
  59. Yijun Yang, Ruiyuan Gao, Xiao Yang, Jianyuan Zhong, and Qiang Xu. 2024b. Guardt2i: Defending t2i models from adversarial prompts. arXiv preprint arXiv:2403.01446. — [arXiv:2403.01446](https://arxiv.org/abs/2403.01446) find
  60. Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao. 2024c. Sneakyprompt: Jailbreaking text-to-image generative models. In 2024 IEEE symposium on security and privacy (SP), pages 897–912. IEEE. find
  61. Shengming Yuan, Qilong Zhang, Lianli Gao, Yaya Cheng, and Jingkuan Song. 2022. Natural color fool: Towards boosting black-box unrestricted attacks. Advances in Neural Information Processing Systems, 35:7546–7560. find
  62. Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. 2019. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems, 30(9):2805–2824. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xie-2026-red-teaming-nsfw-image-classifiers/