Elamur Research⌕
☰
home / papers / 2026 / Beyond the Safety Tax: Mitigating Unsafe Text-to-Image Generation via External Safety Rectification

Beyond the Safety Tax: Mitigating Unsafe Text-to-Image Generation via External Safety Rectification

Paper · 2026 · Xiangtao Meng, Yingkai Dong, Ning Yu et al.

Summary

Most safety defenses for text-to-image diffusion models change the generator itself, either by fine-tuning harmful concepts away (e.g., ESD, SafeGEN) or by steering sampling at inference (e.g., SLD). Unsafe and benign concepts are entangled in the latent space, so these edits also harm harmless content. The authors call this quality loss the "Safety Tax." Their alternative, SafePatch, is a separate safety module attached to a frozen base model. Its backbone is a trainable copy of the base model's encoder. It joins the generator through zero-initialized convolutions, so it has no effect when training begins. An instruction-aware spatial projection helps it pinpoint where unsafe content appears. Training relies on ACS, a new counterfactual dataset. An LLM minimally rewrites unsafe Lexica prompts into safe versions, and the target model renders safe images. A three-stage filter (automated auditors, semantic check, human review) then keeps only tightly aligned pairs. Against six defenses, SafePatch suppresses unsafe outputs more strongly, including under adversarial prompts, while preserving image quality and text alignment. This suggests safety can be added without sacrificing benign fidelity.

Key points

  • On I2P, SafePatch lowers the overall unsafe probability to 7%, while the baselines stay around 20%.
  • Under adversarial attacks, manual checks found 0% unsafe images for SneakyPrompt. Ring-A-Bell yielded 24% unsafe for violence prompts and 6% for nudity prompts, better than every baseline.
  • Scaling ACS from 1K to 10K pairs improves final defense performance. It also halves the iterations needed to reach near-saturation, from 50K to 25K.
  • On one RTX 5090 with SD v1.4, the module adds about 1122 ms of latency per generation and 766 MB of GPU memory.
  • Unsafe rates are measured with NudeNet and Q16, which the authors acknowledge are imperfect proxies. The ACS prompts are shared only on request for research use.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shandong UniversityJinan · China19 works · 952 citations

Also on the byline: Netflix.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 811 worksBias, fairness & safety · 587 works

Cite

Cite as Xiangtao Meng, Yingkai Dong, Ning Yu, Li Wang, Zheng Li and Shanqing Guo. 2026. Beyond the Safety Tax: Mitigating Unsafe Text-to-Image Generation via External Safety Rectification. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1394

@inproceedings{meng2026beyond,
  title = {{Beyond the Safety Tax: Mitigating Unsafe Text-to-Image Generation via External Safety Rectification}},
  author = {Xiangtao Meng, Yingkai Dong, Ning Yu, Li Wang, Zheng Li and Shanqing Guo},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1394},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1394}
}

References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2023. Nsfw gpt. https://www.reddit.com/r/ ChatGPT/comments/11vlp7j/nsfwgpt_that_ nsfw_prompt/. 2023. Usage policies of openai. https://openai. com/policies/usage-policies/. ↗
  2. Ibtihel Amara, Ahmed Imtiaz Humayun, Ivana Kajic, Zarana Parekh, Natalie Harris, Sarah Young, Chirag Nagpal, Najoung Kim, Junfeng He, Cristina Nader Vasconcelos, and 1 others. 2025. Erasing more than intended? how concept erasure degrades the generation of non-target concepts. In Proceedings of the… find
  3. P Bedapudi. 2019. Nudenet: Neural nets for nudity classification, detection and selective censoring. find
  4. Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. 2021. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963. — [arXiv:2110.01963](https://arxiv.org/abs/2110.01963) find
  5. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, and 1 others. 2022. Toy models of superposition. arXiv preprint arXiv:2209.10652. find
  6. Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. 2023. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2426–2436. find
  7. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86– 92. find
  8. Gabriel Goh, James Betker, Li Jing, and Aditya Ramesh. 2024. Dall·e 3. https://openai.com/index/ dall-e-3/. ↗
  9. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718.
  10. Byung Hyun Lee, Sungjin Lim, and Se Young Chun. 2025. Localized concept erasure for text-to-image diffusion models using training-free gated low-rank adaptation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18596– 18606. find
  11. Hang Li, Chengzhi Shen, Philip Torr, Volker Tresp, and Jindong Gu. 2024a. Self-discovering interpretable diffusion latent directions for responsible text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  12. Xinfeng Li, Yuchen Yang, Jiangyi Deng, Chen Yan, Yanjiao Chen, Xiaoyu Ji, and Wenyuan Xu. 2024b. Safegen: Mitigating sexually explicit content generation in text-to-image models. In arXiv preprint arXiv:2404.06666. — [arXiv:2404.06666](https://arxiv.org/abs/2404.06666) find
  13. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pag…
  14. Runtao Liu, Ashkan Khakzar, Jindong Gu, Qifeng Chen, Philip Torr, and Fabio Pizzati. 2024. Latent guard: a safety framework for text-to-image generation. In European Conference on Computer Vision, pages 93– 109. Springer. find
  15. Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. 2024. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6430–6440. find
  16. ML Research. 2023. Q16: A multi-category safety classifier. https://github.com/ml-research/Q16. Accessed: 2024. notAI Tech. 2019. Nudenet: Neural nets for nudity detection. https://github.com/notAI-tech/ ↗
  17. NudeNet. Accessed: 2024. find
  18. Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. 2022. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11410–11420. find
  19. Javier Rando, Daniel Paleka, David Lindner, Lennard Heim, and Florian Tramèr. 2022. Red-teaming the stable diffusion safety filter. ArXiv, abs/2210.04610. — [arXiv:2210.04610](https://arxiv.org/abs/2210.04610) find
  20. Robin Rombach, Patrick Esser. 2022. Stable diffusion v1-4. https://huggingface.co/CompVis/ stable-diffusion-v1-4. ↗
  21. Robin Rombach, Patrick Esser. 2023a. Stable diffusion safety checker. https://huggingface.co/ CompVis/stable-diffusion-safety-checker. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/meng-2026-beyond-safety-tax-mitigating-unsafe/