Elamur Research⌕
☰
home / papers / 2026 / PII-VisBench: Evaluating Personally Identifiable Information Safety in Vision Language Models Along a Continuum of Visibility

PII-VisBench: Evaluating Personally Identifiable Information Safety in Vision Language Models Along a Continuum of Visibility

Paper · 2026 · G M Shahariar, Zabir Al Nazi, Md Olid Hasan Bhuiyan et al.

Summary

Most privacy tests for vision language models check whether a model hands over personal data on request. They ignore how much is already known about the pictured person online. PII-VisBench fills this gap. Each image is sorted by how findable its subject is on the web, and models are asked for personal details based on the picture alone.

The benchmark has 200 faces split evenly into four tiers:

  • High and Medium: real faces from CelebA and the web, ranked by Google hit counts.
  • Low: real faces from MMDT and FFHQ with no Google Lens matches.
  • Zero: StyleGAN-generated faces of people who do not exist.

Each face is crossed with 20 PII attributes, which range from easy to hard. The authors tested 18 open-source VLMs and scored them with two metrics, Refusal Rate and Conditional PII Disclosure Rate.

The main result is that models become more cautious as subject visibility falls. Well-known people are therefore the most exposed, which suggests that memorized knowledge can override safety tuning. These findings support privacy evaluation that accounts for visibility, as well as safety training that protects both famous and obscure individuals.

Key points

  • Scale: 4,000 unique probes (200 subjects × 20 PII attributes), with 50 subjects per visibility tier. The 18 evaluated VLMs range from 0.3B to 32B parameters.
  • Main trend: Conditional PII Disclosure Rate falls from 9.10% for high-visibility subjects to 5.34% for low-visibility ones, while refusals rise.
  • Zero-visibility anomaly: Synthetic faces received slightly fewer refusals and more disclosures than the low tier. The authors attribute this to models inventing or guessing attributes for unfamiliar faces.
  • Uneven protection: Results vary widely across model families. Models refuse structured identifiers (SSNs, emails) more reliably than demographic or descriptive traits (race, gender). Paraphrased and jailbreak-style prompts could bypass refusals, with failures depending on the attack and the model.
  • Evaluation and release: Responses were judged by GPT-4.1-mini, Qwen3-Guard and keyword matching. The code is public on GitHub, but the dataset is available to vetted researchers only on request.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, RiversideRiverside · United States7 works · 339 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 worksBias, fairness & safety · 517 works

Cite

Cite as G M Shahariar, Zabir Al Nazi, Md Olid Hasan Bhuiyan and Zhouxing Shi. 2026. PII-VisBench: Evaluating Personally Identifiable Information Safety in Vision Language Models Along a Continuum of Visibility. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.501

@inproceedings{shahariar2026pii,
  title = {{PII-VisBench: Evaluating Personally Identifiable Information Safety in Vision Language Models Along a Continuum of Visibility}},
  author = {G M Shahariar, Zabir {Al Nazi}, Md Olid Hasan Bhuiyan and Zhouxing Shi},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.501},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.501}
}

References

44 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024. Phi-… — Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
  2. Paweł Borsukiewicz, Fadi Boutros, Iyiola E Olatunji, Charles Beumier, Wendkûuni C Ouedraogo, Jacques Klein, and Tegawendé F Bissyandé. 2025. Beyond real faces: Synthetic datasets can achieve reliable recognition performance without privacy compromise. arXiv preprint arXiv:2510.17372. — [arXiv:2510.17372](https://arxiv.org/abs/2510.17372) find
  3. Simone Caldarella, Massimiliano Mancini, Elisa Ricci, and Rahaf Aljundi. 2024. The phantom menace: unmasking privacy leakages in vision-language models. In European Conference on Computer Vision, pages 435–451. Springer. find
  4. Chuxue Cao, Han Zhu, Jiaming Ji, Qichao Sun, Zhenghao Zhu, Wu Yinyu, Josef Dai, Yaodong Yang, Sirui Han, and Yike Guo. 2025. SafeLawBench: Towards safe alignment of large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14015–14048, Vienna, Austria. Asso… find
  5. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others. 2021. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650…
  6. Tiejin Chen, Pingzhi Li, Kaixiong Zhou, Tianlong Chen, and Hua Wei. 2025. Unveiling privacy risks in multimodal large language models: Task-specific vulnerabilities and mitigation challenges. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4573–4586, Vienna, Austria. As… find
  7. Yang Chen, Ethan Mendes, Sauvik Das, Wei Xu, and Alan Ritter. 2023. Can language models be instructed to protect personal information? arXiv preprint arXiv:2310.02224. — [arXiv:2310.02224](https://arxiv.org/abs/2310.02224) find
  8. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  9. Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences. routledge. find
  10. Ruggero Donida Labati, Arun Ross, and Antitza Dantcheva. 2022. Soft biometrics. In Encyclopedia of Cryptography, Security and Privacy, pages 1–4. Springer. find
  11. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  12. Milton Friedman. 1937. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the american statistical association, 32(200):675–701. find
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  14. Iryna Hartsock and Ghulam Rasool. 2024. Vision-language models for medical report generation and visual question answering: A review. Frontiers in artificial intelligence, 7:1430984. find
  15. Andong Hua, Kenan Tang, Chenhe Gu, Jindong Gu, Eric Wong, and Yao Qin. 2025. Flaw or artifact? rethinking prompt sensitivity in evaluating LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19889–19899, Suzhou, China. Association for Computational… find
  16. Erika McCallister. 2010. Guide to protecting the confidentiality of personally identifiable information. Diane Publishing. find
  17. Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410. find
  18. David Sánchez, Laura Martínez-Sanahuja, and Montserrat Batet. 2018. Survey and evaluation of web search engine hit counts as research tools in computational linguistics. Information Systems, 73:50–60. find
  19. Younggun Kim, Sirnam Swetha, Fazil Kagdi, and Mubarak Shah. 2025. Safe-llava: A privacy-preserving vision-language dataset and benchmark for biometric safety. arXiv preprint arXiv:2509.00192. — [arXiv:2509.00192](https://arxiv.org/abs/2509.00192) find
  20. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, and 1 others. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information proce…
  21. Michal Kosinski, David Stillwell, and Thore Graepel. 2013. Private traits and attributes are predictable from digital records of human behavior. Proceedings of the national academy of sciences, 110(15):5802– 5805. find
  22. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306.
  23. Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. 2024b. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models. In European Conference on Computer Vision, pages 386–403. Springer. find
  24. Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. 2024c. Safety of multimodal large language models on images and texts. arXiv preprint arXiv:2402.00357. — [arXiv:2402.00357](https://arxiv.org/abs/2402.00357) find
  25. Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV). find
  26. Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. 2024. Jailbreakv: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks. arXiv preprint arXiv:2404.03027. find
  27. Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, and 1 others. 2025. Smolvlm: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299. find
  28. Laura Martínez-Sanahuja, David Sánchez, and Montserrat Batet. 2016. Evaluating the suitability of web search engines as proxies for knowledge discovery from the web. In Procedia Computer Science, volume 96, pages 1360–1369. Elsevier. find
  29. OpenAI. 2024. Gpt-4.1-mini. https://platform.openai.com/docs/models. Accessed via OpenAI API. ↗
  30. Charles Spearman. 1987. The proof and measurement of association between two things. The American journal of psychology, 100(3/4):441–471. find
  31. Latanya Sweeney. 2000. Simple demographics often identify people uniquely. Health (San Francisco), 671(2000):1–34. find
  32. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  33. Qwen Team. 2025a. Qwen2.5-vl. find
  34. Qwen Team. 2025b. Qwen3 technical report. Preprint, arXiv:2505.09388.
  35. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the w…
  36. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.
  37. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36:80079–80110.
  38. Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80–83. find
  39. Chejian Xu, Jiawei Zhang, Zhaorun Chen, Chulin Xie, Mintong Kang, Yujin Potter, Zhun Wang, Zhuowen Yuan, Alexander Xiong, Zidi Xiong, and 1 others. 2025. Mmdt: Decoding the trustworthiness and safety of multimodal foundation models. arXiv preprint arXiv:2503.14827. find
  40. Jie Zhang, Xiangkui Cao, Zhouyu Han, Shiguang Shan, and Xilin Chen. 2024. Multi-p2a: A multi-perspective benchmark on privacy assessment for large vision-language models. arXiv preprint arXiv:2412.19496. — [arXiv:2412.19496](https://arxiv.org/abs/2412.19496) find
  41. Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin, and 1 others. 2025. Qwen3guard technical report. arXiv preprint arXiv:2510.14276.
  42. Zhanhui Zhou, Jie Liu, Zhichen Dong, Jiaheng Liu, Chao Yang, Wanli Ouyang, and Yu Qiao. 2024. Emulated disalignment: Safety alignment for large language models may backfire! In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1581… find
  43. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, and 1 others. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479.
  44. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shahariar-2026-pii-visbench-evaluating-personally-identifiable/