Elamur Research⌕
☰
home / papers / 2023 / FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts

FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts

Paper · 2023 · Yichen Gong, Delong Ran, Jinyuan Liu et al.

Summary

Vision-language models (LVLMs) usually inherit their safety guardrails from the text-only LLM at their core. This paper tests whether those guardrails still hold once images enter the input. The authors present FigStep, a black-box jailbreak that requires no optimization or gradient access. The harmful request is not typed as text. Instead, it is rendered typographically into an image, and a harmless-looking text prompt asks the model to carry out what the picture describes. The authors also built SafeBench, a new safety benchmark, and attacked six open-source LVLMs with it. FigStep reached an average attack success rate of 82.50%. It also beat five text-only and four image-based jailbreak methods while costing almost nothing to run. Ablations and an analysis of embedding distributions suggest why it works: visual embeddings are aligned with the LLM's semantic space but not with its safety training. Simple defenses such as OCR filtering, image noise and stricter system prompts did little to stop the attack. The authors conclude that safety alignment must be designed across modalities.

Key points

  • Resources: the SafeBench benchmark (plus a smaller SafeBench-Tiny subset) and code are released at github.com/ThuCCSLab/FigStep. The extended paper accompanies an AAAI 2025 publication.
  • Main result: FigStep reached an average ASR of 82.50% across six open-source LVLMs, including LLaVA, MiniGPT4 and CogVLM.
  • System-prompt defense: a safety-focused system prompt only partly helped. ASR dropped from 90% to 80% on MiniGPT4 and from 82% to 64% on CogVLM, and stayed at 92% on LLaVA.
  • Noise defense: Gaussian noise barely lowered ASR, and it slightly hurt accuracy on benign Llava-bench-in-the-wild questions.
  • Variants for commercial models: FigStep-adv and FigStep-hide target OCR filters. FigStep-pro splits the harmful text across several harmless sub-images.
    • On GPT-4V, ASR rose from 18% (baseline) to 34% (FigStep) and 70% (FigStep-pro).
    • On GPT-4o, ASR rose from 28% to 48% and 62%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,226 citations🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🔬Beijing National Research Center for Information Science and TechnologyBeijing · China33 works · 6,706 citations🎓Shandong UniversityJinan · China19 works · 952 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 worksBias, fairness & safety · 587 works

Cite

Cite as Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan and Xiaoyun Wang. 2023. FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts. arXiv:2311.05608.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.05608 · arXiv 2311.05608

@misc{gong2023figstep,
  title = {{FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts}},
  author = {Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan and Xiaoyun Wang},
  year = {2023},
  doi = {10.48550/arxiv.2311.05608},
  eprint = {2311.05608},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.05608}
}

References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jaided AI. Easyocr 1.7.1. https://pypi.org/project/easyocr/1.7.1/, 2023. Accessed: 2024-02-09. ↗
  2. Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132, 2023.
  3. Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26–38, 2017. find
  4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. find
  5. Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. Image hijacking: Adversarial images can control generative models at runtime. arXiv preprint arXiv:2309.00236, 2023. — [arXiv:2309.00236](https://arxiv.org/abs/2309.00236) find
  6. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  8. Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447, 2023.
  9. Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks, 2017. find
  10. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. find
  11. Microsoft Corporation. The new bing, 2023. Accessed: 2023-12-05. find
  12. Yashar Deldjoo, Cristina Frà, Massimo Valla, Antonio Paladini, Davide Anghileri, Mustafa Anil Tuncil, Franca Garzotta, Paolo Cremonesi, et al. Enhancing children’s experience with recommendation systems. In CEUR Workshop Proceedings, pages N–A, 2017. find
  13. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual Jailbreak Challenges in Large Language Models. In International Conference on Learning Representations (ICLR), 2024.
  14. Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358–19369, 2023. find
  15. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022.
  16. Georgi Gerganov. llama.cpp/example/embedding, 2023. find
  17. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. arXiv preprint arXiv:2302.12173, 2023.
  18. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source llms via exploiting generation. arXiv preprint arXiv:2310.06987, 2023.
  19. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506–17533. PMLR, 2023.
  20. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
  21. Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepinception: Hypnotize large language model to be jailbreaker, 2024. Deep reinforcement learning: An overview. arXiv preprint
  22. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gong-2023-figstep-jailbreaking-large-vision-language/