Elamur Research⌕
☰
home / papers / 2026 / MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models

MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models

Paper · 2026 · Rishabh Makwana, Mamta Mamta, Deeksha Varshney et al.

Summary

Earlier work showed that harmful instructions rendered as flowcharts can jailbreak vision-language models (VLMs), but those studies tested English only. This paper presents MLingualFC, a benchmark for checking whether that attack works across languages. The authors start from an existing English set of flowchart-encoded harmful requests derived from AdvBench. They machine-translate the flowchart text into Hindi, Punjabi, Spanish, Romanian and German and render each one in three layouts. Each image is paired with a neutral instruction to complete the depicted steps. The attack is run black-box and single-turn against Qwen2.5-VL, Gemma-4 and Pangea, and an LLM judge scores the outputs.

Attack success is high for the Latin-script languages and much lower for Hindi and Punjabi. Qualitative analysis shows the low Indic scores do not reflect better safety. Instead, the models misread the script and produce unrelated content, for example treating a bomb-making flowchart as an exam-study plan. The work shows that multimodal safety alignment does not carry over reliably across languages and scripts.

Key points

  • Data pipeline:
    • The source is AdvBench-based harmful queries, already decomposed into step graphs (DOT files) by Zhang et al. (2025).
    • Translation used GoogleTranslator via the deep-translator package.
    • Images were compiled to PNG with Graphviz in horizontal, vertical and tortuous (S-shaped) layouts.
  • Models evaluated: Qwen2.5-VL (including the 3B-Instruct variant in the examples), Gemma-4 (E4B-it) and the multilingual Pangea.
  • Evaluation:
    • GPT-oss-120b serves as the judge.
    • Native speakers rated Hindi and Punjabi responses on a 1–5 harmfulness scale.
    • Human and automatic ASR differed by 0% for Pangea and 1–6% for Gemma-4.
  • Main finding: Spanish, Romanian and German yield consistently high ASR. Hindi and Punjabi yield low ASR, which the authors attribute to weak visual text recognition rather than robust refusal.
  • Access and scope:
    • Code is on GitHub (Rishabhpm23/MLingualFC).
    • The full dataset is available only to vetted researchers under an acceptable-use agreement.
    • Defense methods are left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King's College LondonLondon · United Kingdom42 works · 1,617 citations🎓Indian Institute of Technology JodhpurJodhpur · India3 works · 3 citations🎓Dwarkadas J. Sanghvi College of EngineeringMumbai · India3 works · 9 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 worksVision & language · 657 worksBias, fairness & safety · 517 works

Cite

Cite as Rishabh Makwana, Mamta Mamta, Deeksha Varshney and Oana Cocarascu. 2026. MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models. Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.mellm-1.22

@inproceedings{makwana2026mlingualfc,
  title = {{MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models}},
  author = {Rishabh Makwana, Mamta Mamta, Deeksha Varshney and Oana Cocarascu},
  booktitle = {Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.mellm-1.22},
  url = {https://doi.org/10.18653/v1/2026.mellm-1.22}
}

References

44 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, and 1 others. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925.
  2. Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2025. Jailbreaking leading safety-aligned llms with simple adaptive attacks. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  3. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question answering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425–2433. IEEE Computer Society… find
  4. Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, and Vitaly Shmatikov. 2023. (ab)using images and sounds for indirect instruction injection in multimodal llms. CoRR, abs/2307.10490. — [arXiv:2307.10490](https://arxiv.org/abs/2307.10490) find
  5. Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. 2024a. Image hijacks: adversarial images can control generative models at runtime. In Proceedings of the 41st International Conference on Machine Learning, pages 2443–2455. find
  6. Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. 2024b. Image hijacks: Adversarial images can control generative models at runtime. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. Open-Review.net. find
  7. Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei Koh, Daphne Ippolito, Florian Tramèr, and Ludwig Schmidt. 2023. Are aligned neural networks adversarially aligned? In Advances in Neural Information Processing Systems 36: Annual Conference on Neural… find
  8. Huanran Chen, Yichi Zhang, Yinpeng Dong, Xiao Yang, Hang Su, and Jun Zhu. 2023. Rethinking model ensemble in transfer-based adversarial attacks. arXiv preprint arXiv:2303.09105. — [arXiv:2303.09105](https://arxiv.org/abs/2303.09105) find
  9. Shuo Chen, Zhen Han, Bailan He, Zifeng Ding, Wenqian Yu, Philip Torr, Volker Tresp, and Jindong Gu. 2024. Red teaming GPT-4V: are GPT-4V safe against uni/multi-modal jailbreak attacks? CoRR, abs/2404.03411. — [arXiv:2404.03411](https://arxiv.org/abs/2404.03411) find
  10. Erik Derner and Kristina Bastič. 2025. Beyond Words: Multilingual and Multimodal Red Teaming of MLLMs. In Proceedings of the ACL Workshop on LLM Security (LLMSec), pages 198–206. find
  11. Yi Ding, Bolian Li, and Ruqi Zhang. 2025. ETA: evaluating then aligning safety of vision language models at inference time. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  12. Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. 2023. How robust is google’s bard to adversarial image attacks? CoRR, abs/2309.11751. — [arXiv:2309.11751](https://arxiv.org/abs/2309.11751) find
  13. Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. 2025. Figstep: Jailbreaking large vision-language models via typographic visual prompts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23951–23959. find
  14. Graphviz Team. 2025. Graphviz – Graph Visualization Software. find
  15. Tianle Gu, Zeyang Zhou, Kexin Huang, and 1 others. 2024. MLLMGuard: A Multi-Dimensional Safety Evaluation Suite for Multimodal Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 7256–7295. find
  16. Dongchen Han, Xiaojun Jia, Yang Bai, Jindong Gu, Yang Liu, and Xiaochun Cao. 2023. Ot-attack: Enhancing adversarial transferability of vision-language models via optimal transport optimization. arXiv preprint arXiv:2312.04403. — [arXiv:2312.04403](https://arxiv.org/abs/2312.04403) find
  17. Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2021. Scaling up vision-language pre-training for image captioning. CoRR, abs/2111.12233. find
  18. Zaid Khan, B. G. Vijay Kumar, Samuel Schulter, Xiang Yu, Yun Fu, and Manmohan Chandraker. 2023. Q: how to specialize large vision-language models to data-scarce VQA tasks? A: self-train on unlabeled images! In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC,… find
  19. Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024a. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. CoRR, abs/2407.07895. find
  20. Jiaxuan Li, Duc Minh Vo, Akihiro Sugimoto, and Hideki Nakayama. 2024b. Evcap: Retrieval-augmented image captioning with external visual-name memory for open-world comprehension. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 13… find
  21. Yifan Li, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. 2024c. Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024,… find
  22. Yifan Li, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. 2024d. Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models. In European Conference on Computer Vision, pages 174–189. Springer. find
  23. Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. 2024. Jailbreakv-28k: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks. CoRR, abs/2404.03027. find
  24. Siyuan Ma, Weidi Luo, Yu Wang, and Xiaogeng Liu. 2024. Visual-roleplay: Universal jailbreak attack on multimodal large language models via role-playing image character. arXiv preprint arXiv:2405.20773. — [arXiv:2405.20773](https://arxiv.org/abs/2405.20773) find
  25. Priyaranjan Pattnayak and Sanchari Chowdhuri. 2026. Indicjr: A judge-free benchmark of jailbreak robustness in south asian languages. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track), pages 649–668.
  26. Xiangyu Qi, Yi Zeng, Tinghao Xie, and 1 others. 2024. Visual Adversarial Examples Jailbreak Aligned Large Language Models. In Proceedings of AAAI. find
  27. Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023. Prompting large language models with answer heuristics for knowledge-based visual question answering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 14974– 14983. IEEE. find
  28. Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. 2023. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models. arXiv preprint arXiv:2307.14539. find
  29. Erfan Shayegani, Yue Dong, and Nael B. Abu-Ghazaleh. 2024. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. find
  30. Enyi Shi, Pengyang Shao, Yanxin Zhang, Chenhang Cui, Jiayi Lyu, Xu Xie, Xiaobo Xia, Fei Shen, and Tat-Seng Chua. 2026. Lingua-safetybench: A benchmark for safety evaluation of multilingual vision-language models. CoRR, abs/2601.22737. — [arXiv:2601.22737](https://arxiv.org/abs/2601.22737) find
  31. Jiayang Song, Yuheng Huang, Zhehua Zhou, and Lei Ma. 2025. Multilingual blending: Large language model safety alignment evaluation with language mixture. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3433–3449. find
  32. Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. 2021. Visualmrc: Machine reading comprehension on document images. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium o… find
  33. Siyuan Wang, Zhuohan Long, Zhihao Fan, and Zhongyu Wei. 2024. From LLMs to MLLMs: Exploring the landscape of multimodal jailbreaking. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17568–17582, Miami, Florida, USA. Association for Computational Lingu… find
  34. Yu Wang, Xiaofei Zhou, Yichen Wang, Geyuan Zhang, and Tianxing He. 2025. Jailbreak large vision-language models through multi-modal linkage. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1466–1494.
  35. Fenghua Weng, Yue Xu, Chengyan Fu, and Wenjie Wang. 2025. Mmj-bench: A comprehensive study on jailbreak attacks and defenses for vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27689– 27697. find
  36. An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, and 9 others. 2025. Qwen2.5-1m technical report. CoRR, abs/2501.15383.
  37. Ziyi Yin, Muchao Ye, Tianrong Zhang, Tianyu Du, Jinguo Zhu, Han Liu, Jinghui Chen, Ting Wang, and Fenglong Ma. 2023. Vlattack: Multimodal adversarial attacks on vision-language tasks via pre-trained models. Advances in Neural Information Processing Systems, 36:52936–52956. find
  38. Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023. Low-Resource Languages Jailbreak GPT-4. arXiv preprint arXiv:2310.02446.
  39. Haneul Yoo, Yongjin Yang, and Hwaran Lee. 2024. Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding. arXiv preprint arXiv:2406.15481.
  40. Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. 2024. Pangea: A fully open multilingual multimodal llm for 39 languages. In The Thirteenth International Conference on Learning… find
  41. Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. 2025. Pangea: A fully open multilingual multimodal LLM for 39 languages. In The Thirteenth International Conference on Learning… find
  42. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6720–6731. Computer Vision Foundation / IEEE. find
  43. Ziyi Zhang, Zhen Sun, Zongmin Zhang, Jihui Guo, and Xinlei He. 2025. FC-Attack: Jailbreaking Large Vision-Language Models via Auto-Generated Flowcharts. arXiv preprint arXiv:2502.21059. — [arXiv:2502.21059](https://arxiv.org/abs/2502.21059) find
  44. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv preprint arXiv:2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/makwana-2026-mlingualfc-evaluating-jailbreak-vulnerabilities-multilingual/