Elamur Research⌕
☰
home / papers / 2026 / Defeating Cerberus: Privacy-Leakage Mitigation in Vision Language Models

Defeating Cerberus: Privacy-Leakage Mitigation in Vision Language Models

Paper · 2026 · Boyang Zhang, Istemi Ekin Akkus, Ruichuan Chen et al.

Summary

Vision Language Models can read personal data from images such as scans or ID cards and perform PII extraction at scale. Their safety guardrails can be bypassed through jailbreaks or backdoors, and the visual channel adds another attack surface.

Existing PII benchmarks are text-only, so the authors first build multimodal test sets by turning text PII corpora into images. They use two strategies: direct conversion (rendering Enron emails or synthetic records as documents and tables, with simulated scanning noise) and context injection.

They then propose a concept-guided mitigation inspired by interpretability research. It locates internal states of the LLM backbone linked to PII-related content and amplifies them with a steering coefficient, so the model declines such requests. No retraining or fine-tuning is needed.

The steered models refuse PII tasks 93.3% of the time on average, with little harm to unrelated tasks. One fixed configuration works across all datasets and on PII types absent from the demonstration data. This offers a cheap, deployable privacy safeguard that may transfer to other multimodal LLMs built on similar backbones.

Key points

  • New multimodal PII datasets include PII-Table (tables from ai4privacy synthetic data, with optional "scanned" artifacts), image versions of the Enron email corpus, and CelebA-Info. The authors note that text-to-image models cannot yet render legible PII realistically, which is why they converted text corpora instead.
  • Average refusal rate on PII-related tasks is 93.3%, while refusals on non-PII tasks stay negligible. Steering settings were not re-tuned between datasets, and the method also caught unseen types such as social security and credit card numbers.
  • Results vary by PII type on CelebA-Info. Address extraction is refused in over 97% of cases for all three backbones, but email is weaker (79.1% for Vicuna-7B), and other email and phone results sit above 80%.
  • The steering coefficient study on the Mistral-7B backbone found a clear gap, around 0.4–0.6, between where mitigation weakens and where benign tasks start to suffer. This makes it possible to pick a safe minimal value.
  • The method only works if the model already has refusal behavior to amplify, and smaller models may lack the needed concepts. Code and dataset tools are released at github.com/Nokia-Bell-Labs/cerberus.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Helmholtz Center for Information SecuritySaarbrücken · Germany20 works · 1,172 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 works

Cite

Cite as Boyang Zhang, Istemi Ekin Akkus, Ruichuan Chen, Alice Dethise, Klaus Satzke, Ivica Rimac and Yang Zhang. 2026. Defeating Cerberus: Privacy-Leakage Mitigation in Vision Language Models. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.154

@inproceedings{zhang2026defeating,
  title = {{Defeating Cerberus: Privacy-Leakage Mitigation in Vision Language Models}},
  author = {Boyang Zhang, Istemi Ekin Akkus, Ruichuan Chen, Alice Dethise, Klaus Satzke, Ivica Rimac and Yang Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.154},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.154}
}

References

42 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. https://github.com/meta-llama/llama3/. ↗
  2. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. Preprint, arXiv:2406.11717.
  3. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities. CoRR abs/2308.12966.
  4. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. CoRR abs/2302.0402…
  5. James Betker, Gabriel Goh, Li Jing, † TimBrooks, Jianfeng Wang, Linjie Li, † LongOuyang, † JuntangZhuang, † JoyceLee, † YufeiGuo, † Wesam-Manassra, † PrafullaDhariwal, † CaseyChu, † YunxinJiao, and Aditya Ramesh. 2023. Improving image generation with better captions. find
  6. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  7. Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023. Jailbreaker: Automated Jailbreak Across Multiple Large Language Model Chatbots. CoRR abs/2307.08715. — [arXiv:2307.08715](https://arxiv.org/abs/2307.08715) find
  8. GDPR. 2016. Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec (general data protection regulation) (text w… find
  9. Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. 2023. FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts. CoRR abs/2311.05608.
  10. Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, and Min Lin. 2024. Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast. In International Conference on Machine Learning (ICML). PMLR. find
  11. Bryan Klimt and Yiming Yang. 2004. The Enron Corpus: A New Dataset for Email Classification Research. In European Conference on Machine Learning (ECML), pages 217–226. Springer. find
  12. Iro Laina, Christian Rupprecht, and Nassir Navab. 2019. Towards Unsupervised Image Captioning With Shared Multimodal Embeddings. In IEEE International Conference on Computer Vision (ICCV), pages 7413–7423. IEEE. find
  13. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. 2024. A mechanistic understanding of alignment algorithms: A case study on DPO and toxicity. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024.…
  14. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024. Llava-onevision: Easy visual task transfer. Preprint, arXiv:2408.03326.
  15. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved Baselines with Visual Instruction Tuning. CoRR abs/2310.03744.
  16. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual Instruction Tuning. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS.
  17. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Langdon Holmes, Scott Crossley, Perpetual Baffour, Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, Jules King, Lauryn Burleigh, Maggie Demkin, and Yang Liu. 2023c. Jailbreaking ChatGPT via Ryan Holbrook, Walter Reade, and Addison Prompt Engineering: An… find
  18. CoRR Howard. 2024. The learning agency lab - pii data abs/2305.13860. detection. https://kaggle.com/competitions/ pii-detection-removal-from-educational-data. ↗
  19. Zhendong Liu, Yuanbi Nie, Yingshui Tan, Xiangyu Yue, Kaggle. Qiushi Cui, Chongjun Wang, Xiaoyong Zhu, and Bo Zheng. 2024. Safety Alignment for Vision Lan-Hai Huang, Zhengyu Zhao, Michael Backes, Yun guage Models. CoRR abs/2405.13581. — [arXiv:2405.13581](https://arxiv.org/abs/2405.13581) find
  20. Shen, and Yang Zhang. 2023. Composite Backdoor Attacks Against Large Language Models. CoRR Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. abs/2310.07676. 2015. Deep Learning Face Attributes in the Wild. In IEEE International Conference on Computer Vision Jie Huang, Hanyin Shao, and Kevin Chen-… find
  21. IEEE. 2022. Are Large Pre-Trained Language Models Leaking Your Personal Information? In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2038–2047. ACL. find
  22. Wen Huang, Hongbin Liu, Minxin Guo, and Neil Zhenqiang Gong. 2024. Visual hallucinations of multimodal large language models. arXiv preprint arXiv:2402.14683. — [arXiv:2402.14683](https://arxiv.org/abs/2402.14683) find
  23. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  24. Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella Béguelin. 2023. Analyzing Leakage of Personally Identifiable Information in Language Models. In IEEE Symposium on Security and Privacy (S&P), pages 346–363. IEEE.
  25. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.
  26. Lingbo Mo, Zeyi Liao, Boyuan Zheng, Yu Su, Chaowei Xiao, and Huan Sun. 2024. A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents. CoRR abs/2402.10196. — [arXiv:2402.10196](https://arxiv.org/abs/2402.10196) find
  27. Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. 2011. Multimodal Deep Learning. In International Conference on Machine Learning (ICML), pages 689–696. Omnipress. find
  28. Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 technical report. Preprint, arXiv:2407.10671. find
  29. Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. 2024. Llm self defense: By self examination, llms know they are being tricked. Preprint, arXiv:2308.07308.
  30. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A Survey on Multimodal Large Language Models. CoRR abs/2306.13549.
  31. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. Do Anything Now: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM.
  32. Bowei Tian, Xuntao Lyu, Meng Liu, Hongyi Wang, and Ang Li. 2025. Why representation engineering works: A theoretical and empirical study in vision-language models. Preprint, arXiv:2503.22720. — [arXiv:2503.22720](https://arxiv.org/abs/2503.22720) find
  33. ZheZhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhan… — The Rise and Potential of Large Language Model Based Agents: A Survey
  34. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending ChatGPT against jailbreak attack via self-reminders. Nature Machine Intelligence. find
  35. Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. 2023. Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models. CoRR abs/2305.14710. find
  36. Jun Yan, Vansh Gupta, and Xiang Ren. 2023. BITE: Textual Backdoor Attacks with Iterative Trigger Injection. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 12951– 12968. ACL. find
  37. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… — Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification
  38. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Annual Conference on Neural Information Processing Syst…
  39. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. CoRR abs/2304.10592.
  40. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendry… — Representation Engineering: A Top-Down Approach to AI Transparency
  41. Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. 2024. Improving alignment and robustness with circuit breakers. Preprint, arXiv:2406.04313.
  42. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023b. Universal and Transferable Adversarial Attacks on Aligned Language Models. CoRR abs/2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-defeating-cerberus-privacy-leakage-mitigation/