Elamur Research⌕
☰
home / papers / 2023 / MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models

MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models

Paper · 2023 · Xin Liu, Yichen Zhu, Jindong Gu et al.

Summary

Multimodal LLMs (MLLMs) rely on language backbones that are often safety-aligned, but it is unclear whether that protection holds once images enter the dialogue. The authors observe that when a harmful request is paired with an image related to it, models frequently comply, while an unrelated picture usually triggers a refusal. They suggest the vision-language alignment stage, trained without safety data, overrides the LLM's caution. Building on this, they construct MM-SafetyBench. GPT-4 writes prohibited questions for 13 scenarios derived from usage policies and extracts each unsafe key phrase. That phrase is then turned into a Stable Diffusion image, a typographic rendering, or a blend of both, and the text query is reworded to point at the picture. The resulting 5,040 text-image pairs are used to test 12 state-of-the-art MLLMs. Typographic and combined images markedly raise attack success, even for models with aligned backbones. A simple safety prompt lowers success, mainly for models that follow instructions well. The benchmark offers an early systematic way to measure image-driven jailbreaks.

Key points

  • MM-SafetyBench contains 5,040 text-image pairs across 13 forbidden-content scenarios, including illegal activity, hate speech, and physical harm. It is built in four stages: question generation, unsafe key-phrase extraction, query-to-image conversion, and question rephrasing.
  • In a pilot study with 300 harmful queries, LLaVA-1.5 rarely complied when shown irrelevant images from LLaVA-150K. Attack success was about 5.1% for illegal activity and 2.3% for hate speech, and it rose sharply when query-relevant images were used instead.
  • On LLaVA-1.5, typography images increased average attack success rate (ASR) by more than 30% for the 7B model and 28.3% for the 13B model. Combining Stable Diffusion images with typography generally worked best.
  • Some models only seem safe because of weaknesses rather than alignment, such as overfitting (e.g., CogVLM), poor OCR or visual understanding, or weak instruction following (e.g., IDEFICS giving off-topic answers). No model tested combined a low ASR with a high Refusal Rate (RR).
  • A safety prompt reduced LLaVA-1.5's ASR from roughly 77% to about 15%. The effect was smaller for MiniGPT-4, InstructBLIP, Otter, and IDEFICS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom122 works · 160,277 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China152 works · 13,164 citations🎓East China Normal UniversityShanghai · China69 works · 2,490 citations

Also on the byline: Midea Group.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 worksVision & language · 811 worksBias, fairness & safety · 587 works

Cite

Cite as Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang and Yu Qiao. 2023. MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models. arXiv:2311.17600.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.17600 · arXiv 2311.17600

@misc{liu2023mm,
  title = {{MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models}},
  author = {Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang and Yu Qiao},
  year = {2023},
  doi = {10.48550/arxiv.2311.17600},
  eprint = {2311.17600},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.17600}
}

References

52 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy B… — Flamingo: a Visual Language Model for Few-Shot Learning
  2. Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. OpenFlamingo: An Open-Source Framework for Training Large… — OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
  3. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023.
  4. Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jingren Zhou. TouchStone: Evaluating Vision-Language Models by Language Models. arXiv preprint, page arXiv:2308.16890, 2023. — [arXiv:2308.16890](https://arxiv.org/abs/2308.16890) find
  5. Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions. arXiv preprint, page arXiv:2309.07875, 2023.
  6. Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schimdt. VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use. arXiv preprint, page arXiv:2308.06595, 2023. — [arXiv:2308.06595](https://arxiv.org/abs/2308.06595) find
  7. Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM. arXiv preprint, page arXiv:2309.14348, 2023.
  8. Delong Chen, Jianfeng Liu, Wenliang Dai, and Baoyuan Wang. Visual Instruction Tuning with Polite Flamingo. arXiv preprint, page arXiv:2307.01003, 2023. — [arXiv:2307.01003](https://arxiv.org/abs/2307.01003) find
  9. Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023.
  10. Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic. arXiv preprint, page arXiv:2306.15195, 2023.
  11. Yang Chen, Ethan Mendes, Sauvik Das, Wei Xu, and Alan Ritter. Can language models be instructed to protect personal information? arXiv preprint arXiv:2310.02224, 2023. — [arXiv:2310.02224](https://arxiv.org/abs/2310.02224) find
  12. Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. arXiv preprint arXiv:2311.10081, 2023. — [arXiv:2311.10081](https://arxiv.org/abs/2311.10081) find
  13. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/… ↗
  14. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv preprint, page arXiv:2305.06500, 2023.
  15. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint, page arXiv:2306.13394, 2023.
  16. Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model. arXiv preprint, page arXiv:2304.15010, 2023.
  17. Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Figstep: Jailbreaking large vision-language models via typographic visual prompts. arXiv preprint arXiv:2311.05608, 2023.
  18. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904– 6913, 2017.
  19. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
  20. Peter Henderson, Eric Mitchell, Christopher D. Manning, Dan Jurafsky, and Chelsea Finn. Self-destructing models: Increasing the costs of harmful dual uses of foundation models. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 287–296, 2023. find
  21. Dominik Hintersdorf, Lukas Struppek, and Kristian Kersting. Balancing Transparency and Risk: The Security and Privacy Risks of Open-Source Machine Learning Models. arXiv preprint, page arXiv:2308.09490, 2023. — [arXiv:2308.09490](https://arxiv.org/abs/2308.09490) find
  22. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation. arXiv preprint, page arXiv:2310.06987, 2023.
  23. Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.
  24. Yuanfeng Ji, Chongjian Ge, Weikai Kong, Enze Xie, Zhengying Liu, Zhengguo Li, and Ping Luo. Large language models as automated aligners for benchmarking vision-language models. arXiv preprint arXiv:2311.14580, 2023. — [arXiv:2311.14580](https://arxiv.org/abs/2311.14580) find
  25. Shuyu Jiang, Xingshu Chen, and Rui Tang. Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attacks. arXiv preprint, page arXiv:2310.10077, 2023. — [arXiv:2310.10077](https://arxiv.org/abs/2310.10077) find
  26. Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents. arXiv preprint, page arXiv:2…
  27. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A Multi-Modal Model with In-Context Instruction Tuning. arXiv preprint, page arXiv:2305.03726, 2023.
  28. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension. arXiv preprint, page arXiv:2307.16125, 2023.
  29. Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. RAIN: Your Language Models Can Align Themselves without Finetuning. arXiv preprint, page arXiv:2309.07124, 2023. — [arXiv:2309.07124](https://arxiv.org/abs/2309.07124) find
  30. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning. arXiv preprint, page arXiv:2310.03744, 2023.
  31. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. arXiv preprint, page arXiv:2304.08485, 2023.
  32. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is Your Multi-modal Model an All-around Player? arXiv preprint, page arXiv:2307.06281, 2023.
  33. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. find
  34. OpenAI. Gpt-4v(ision) system card., 2023. find
  35. Renjie Pi, Tianyang Han, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, and Tong Zhang. Mllm-protector: Ensuring mllm’s safety without hurting performance. arXiv preprint arXiv:2401.02906, 2024.
  36. Wenqi Shao, Yutao Hu, Peng Gao, Meng Lei, Kaipeng Zhang, Fanqing Meng, Peng Xu, Siyuan Huang, Hongsheng Li, Yu Qiao, and Ping Luo. Tiny LVLM-eHub: Early Multimodal Experiments with Bard. arXiv preprint, page arXiv:2308.03729, 2023. — [arXiv:2308.03729](https://arxiv.org/abs/2308.03729) find
  37. Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. On the Exploitability of Instruction Tuning. arXiv preprint, page arXiv:2306.17194, 2023. — [arXiv:2306.17194](https://arxiv.org/abs/2306.17194) find
  38. Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. PandaGPT: One Model To Instruction-Follow Them All. arXiv preprint, page arXiv:2305.16355, 2023.
  39. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. ↗
  40. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  41. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023.
  42. Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks. arXiv preprint, page arXiv:2305.11175, 2023. — [arXiv:2305.11175](https://arxiv.org/abs/2305.11175) find
  43. Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations. arXiv preprint, page arXiv:2310.06387, 2023.
  44. Yuanwei Wu, Xiang Li, Yixin Liu, Pan Zhou, and Lichao Sun. Jailbreaking gpt-4v via self-adversarial attacks with system prompts. arXiv preprint arXiv:2311.09127, 2023. — [arXiv:2311.09127](https://arxiv.org/abs/2311.09127) find
  45. Liang Xu, Kangkang Zhao, Lei Zhu, and Hang Xue. SC-Safety: A Multi-round Open-ended Question Adversarial Safety Benchmark for Large Language Models in Chinese. arXiv preprint, page arXiv:2310.05818, 2023. — [arXiv:2310.05818](https://arxiv.org/abs/2310.05818) find
  46. Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models. arXiv preprint, page arXiv:2310.02949, 2023.
  47. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality. arXiv preprint, pa…
  48. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023.
  49. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. arXiv preprint, page arXiv:2308.02490, 2023.
  50. Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong. What Matters in Training a GPT4-Style Language Model with Multimodal Inputs? arXiv preprint, page arXiv:2307.02469, 2023. find
  51. Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and-language generation via generative vokens. arXiv preprint arXiv:2310.02239, 2023. — [arXiv:2310.02239](https://arxiv.org/abs/2310.02239) find
  52. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv preprint, page arXiv:2304.10592, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2023-mm-safetybench-benchmark-safety-evaluation/