Elamur Research⌕
☰
home / papers / 2026 / UMMF: Protecting Copyright of Large Vision-Language Models through Unlearning-based Multimodal Memorization Fingerprint

UMMF: Protecting Copyright of Large Vision-Language Models through Unlearning-based Multimodal Memorization Fingerprint

Paper · 2026 · Xiaofan Zheng, Xinghao Wang and Xiaojun Wan

Summary

Unauthorized fine-tuning and black-box redeployment of Large Vision-Language Models (LVLMs) are hard to detect. Existing fingerprinting relies on adversarial or backdoor trigger images that must produce a fixed output. Such explicit triggers force a compromise: they are either fragile after modification or easy to spot and purify. The authors propose UMMF, which needs no fixed input-output pair. For each fingerprint image, a semantically close paraphrase T′ of its caption T is generated. During training, the model learns (I, T) and unlearns (I, T′). This creates small regions of deliberate overfitting in the data manifold. To verify ownership, a membership inference attack measures the suspect model's memorization of both captions. The gap between them serves as an implicit fingerprint. The source model acts as a shadow model for difficulty calibration. Experiments mainly use LLaVA with VQA datasets and cover full and LoRA fine-tuning, pruning, and image transformations. UMMF outperforms trigger-based baselines. This matters because it offers a black-box, stealthier and sturdier way to prove model provenance.

Key points

  • The authors describe UMMF as the first fingerprinting method to use unlearning. It is evaluated with Fingerprint Success Rate (FSR) on V7W, ST-VQA and TextVQA.
  • Under the strongest uniform noise tested, FSR falls by at most 13% and still beats the best baseline. Mean blur was also tested.
  • After Taylor-expansion-based unstructured pruning of the smallest 10% of weights, UMMF stays stable, whereas the IF and RNA baselines degrade sharply. Pruning attention layers harms the fingerprint more than pruning MLP layers.
  • In the ablations, removing unlearning causes the largest drop (as extracted, e.g. about 87% → 72% FSR on V7W with LoRA). Removing calibration also hurts noticeably. Performance is stable once more than 50 fingerprint samples are used.
  • The main experiments use LLaVA, with extra LoRA results on IDEFICS2-8B and MiniGPT-4 in the appendix. Code is available at github.com/qingpingwan/UMMF.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China315 works · 43,268 citations🎓Xi'an Jiaotong UniversityXi'an · China42 works · 36,431 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksHardware and Architecture · 167 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 works

Cite

Cite as Xiaofan Zheng, Xinghao Wang and Xiaojun Wan. 2026. UMMF: Protecting Copyright of Large Vision-Language Models through Unlearning-based Multimodal Memorization Fingerprint. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.429

@inproceedings{zheng2026ummf,
  title = {{UMMF: Protecting Copyright of Large Vision-Language Models through Unlearning-based Multimodal Memorization Fingerprint}},
  author = {Xiaofan Zheng, Xinghao Wang and Xiaojun Wan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.429},
  url = {https://doi.org/10.18653/v1/2026.acl-long.429}
}

References

59 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Ha… — Qwen2.5-VL Technical Report
  2. Yi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu, Zheng Wang, Yang Yang, See-Kiong Ng, and Heng Tao Shen. 2024. Gallerygpt: Analyzing paintings with large multimodal models. arXiv preprint arXiv:2408.00491. — [arXiv:2408.00491](https://arxiv.org/abs/2408.00491) find
  3. Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marcal Rusinol, CV Jawahar, Ernest Valveny, and Dimosthenis Karatzas. 2019. Scene text Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. 2024. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE… find
  4. Carl Edwards, ChengXiang Zhai, and Heng Ji. 2021. Text2mol: Cross-modal molecule retrieval with natural language queries. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 595–607.
  5. Thibaud Gloaguen, Robin Staab, Nikola Jovanović, and Martin Vechev. 2025. Robust llm fingerprinting via domain-specific watermarks. Preprint, arXiv:2505.16723. — [arXiv:2505.16723](https://arxiv.org/abs/2505.16723) find
  6. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  7. Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. 2022. Membership inference attacks on machine learning: A survey. Preprint, arXiv:2103.07853. — [arXiv:2103.07853](https://arxiv.org/abs/2103.07853) find
  8. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations.
  9. Dmitri Iourovitski, Sanat Sharma, and Rakshak Talwar. 2024. Hide and seek: Fingerprinting large language models with evolutionary learning. Preprint, arXiv:2408.02871. — [arXiv:2408.02871](https://arxiv.org/abs/2408.02871) find
  10. Heng Jin, Chaoyu Zhang, Shanghao Shi, Wenjing Lou, and Y Thomas Hou. 2024. Proflingo: A fingerprinting-based intellectual property protection scheme for large language models. In 2024 IEEE Conference on Communications and Network Security (CNS), pages 1–9. IEEE. find
  11. Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? Preprint, arXiv:2405.02246.
  12. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023a. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125.
  13. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024a. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36.
  14. Jiaqi Li, Qianshan Wei, Chuanyi Zhang, Guilin Qi, Miaozeng Du, Yongrui Chen, Sheng Bi, and Fan Liu. 2025. Single image unlearning: Efficient machine unlearning in multimodal large language models. Preprint, arXiv:2405.12523. — [arXiv:2405.12523](https://arxiv.org/abs/2405.12523) find
  15. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024a. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations. find
  16. Qiao Li, Jing Chen, Kun He, Zijun Zhang, Ruiying Du, Jisi She, and Xinxin Wang. 2024b. Model-agnostic adversarial example detection via high-frequency amplification. Computers & Security, 141:103791. find
  17. Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. 2024b. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.17294.
  18. Shen Li, Liuyi Yao, Jinyang Gao, Lan Zhang, and Yaliang Li. 2024c. Double-i watermark: Protecting model copyright for LLM fine-tuning. CoRR, abs/2402.14883. — [arXiv:2402.14883](https://arxiv.org/abs/2402.14883) find
  19. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023b. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355.
  20. Yuqing Liang, Jiancheng Xiao, Wensheng Gan, and Philip S. Yu. 2024. Watermarking techniques for large language models: A survey. Preprint, arXiv:2409.00089. — [arXiv:2409.00089](https://arxiv.org/abs/2409.00089)
  21. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In NeurIPS.
  22. Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. 2024. Rethinking machine unlearning for large language models. Preprint, arXiv:2402.08787. find
  23. Zheyuan Liu, Guangyao Dou, Mengzhao Jia, Zhaoxuan Tan, Qingkai Zeng, Yongle Yuan, and Meng Jiang. 2025. Protecting privacy in multimodal large language models with mllmu-bench. Preprint, arXiv:2410.22108.
  24. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521. find
  25. Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. 2023. Membership inference attacks against language models via neighbourhood comparison. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11330– 11…
  26. Anshul Nasery, Jonathan Hayase, Creston Brooks, Peiyao Sheng, Himanshu Tyagi, Pramod Viswanath, and Sewoong Oh. 2025. Scalable fingerprinting of large language models. Preprint, arXiv:2502.07760. — [arXiv:2502.07760](https://arxiv.org/abs/2502.07760) find
  27. OpenAI. 2024. Gpt-4o system card. arXiv:2410.21276.
  28. Preprint, Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326. find
  29. Toan Tran, Ruixuan Liu, and Li Xiong. 2025. Tokens for learning, tokens for unlearning: Mitigating membership inference attacks in large language models via dual-purpose training. Preprint, arXiv:2502.19726. find
  30. Jiaqi Wang, Hanqi Jiang, Yiheng Liu, Chong Ma, Xu Zhang, Yi Pan, Mengyuan Liu, Peiran Gu, Sichen Xia, Wenjun Li, Yutong Zhang, Zihao Wu, Zhengliang Liu, Tianyang Zhong, Bao Ge, Tuo Zhang, Ning Qiang, Xintao Hu, Xi Jiang, Xin Zhang, Wei Zhang, Dinggang Shen, Tianming Liu, and Shu Zhang. 2024. A compr… — [arXiv:2408.01319](https://arxiv.org/abs/2408.01319) find
  31. Shida Wang, Chaohu Liu, Yubo Wang, and Linli Xu. 2025a. Fpedit: Robust llm fingerprinting through localized knowledge editing. Preprint, arXiv:2508.02092. find
  32. Yubo Wang, Jianting Tang, Chaohu Liu, and Linli Xu. 2025b. Tracking the copyright of large vision-language models through parameter learning adversarial images. In The Thirteenth International Conference on Learning Representations. find
  33. Lauren Watson, Chuan Guo, Graham Cormode, and Alexandre Sablayrolles. 2022. On the importance of difficulty calibration in membership inference attacks. In International Conference on Learning Representations. find
  34. Hengyu Wu and Yang Cao. 2025. Membership inference attacks on large-scale models: A survey. Preprint, arXiv:2503.19338. — [arXiv:2503.19338](https://arxiv.org/abs/2503.19338) find
  35. Jiaxuan Wu, Wanli Peng, Hang Fu, Yiming Xue, and Juan Wen. 2025a. Imf: Implicit fingerprint for large language models. Preprint, arXiv:2503.21805. — [arXiv:2503.21805](https://arxiv.org/abs/2503.21805) find
  36. Zehao Wu, Yanjie Zhao, and Haoyu Wang. 2025b. Gradient-based model fingerprinting for llm similarity detection and family classification. Preprint, arXiv:2506.01631. — [arXiv:2506.01631](https://arxiv.org/abs/2506.01631) find
  37. Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. 2024. Instructional fingerprinting of large language models. Preprint, arXiv:2401.12255. find
  38. Zhenhua Xu, Meng Han, and Wenpeng Xing. 2025a. Evertracer: Hunting stolen large language models via stealthy and robust probabilistic fingerprint. Preprint, arXiv:2509.03058. find
  39. Zhenhua Xu, Xubin Yue, Zhebo Wang, Qichen Liu, Xixiang Zhao, Jingxuan Zhang, Wenjun Zeng, Weng-peng Xing, Dezhang Kong, Changting Lin, and Meng Han. 2025b. Copyright protection for large language models: A survey of methods, challenges, and trends. Preprint, arXiv:2508.11548. — [arXiv:2508.11548](https://arxiv.org/abs/2508.11548) find
  40. Zhenhua Xu, Xixiang Zhao, Xubin Yue, Shengwei Tian, Changting Lin, and Meng Han. 2025c. Ctcc: A robust and stealthy fingerprinting framework for large language models via cross-turn contextual correlation backdoor. Preprint, arXiv:2509.09703.
  41. Shojiro Yamabe, Futa Waseda, Tsubasa Takahashi, and Koki Wataoka. 2025. Mergeprint: Merge-resistant fingerprints for robust black-box ownership verification of large language models. Preprint, arXiv:2410.08604. find
  42. Zhiguang Yang and Hanzhou Wu. 2024. A fingerprint for large language models. Preprint, arXiv:2407.01235. — [arXiv:2407.01235](https://arxiv.org/abs/2407.01235) find
  43. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. find
  44. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490.
  45. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2023. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502.
  46. Xubin Yue, Zhenhua Xu, Wenpeng Xing, Jiahui Yu, Mohan Li, and Meng Han. 2025. Pree: Towards harmless and adaptive fingerprint editing in large language models via knowledge prefix enhancement. Preprint, arXiv:2509.00918. find
  47. Jiale Zhang, Haoxuan Li, Di Wu, Xiaobing Sun, Qinghua Lu, and Guodong Long. 2025a. Beyond dataset watermarking: Model-level copyright protection for code summarization models. In THE WEB CONFERENCE 2025. find
  48. Jialong Zhang, Zhongshu Gu, Jiyong Jang, Hui Wu, Marc Ph. Stoecklin, Heqing Huang, and Ian Molloy. 2018. Protecting intellectual property of deep neural networks with watermarking. In Proceedings of the 2018 on Asia Conference on Computer and Communications Security, ASIACCS ’18, page 159–172, New Y… find
  49. Jie Zhang, Dongrui Liu, Chen Qian, Linfeng Zhang, Yong Liu, Yu Qiao, and Jing Shao. 2024b. Reef: Representation encoding fingerprints for large language models. Preprint, arXiv:2410.14273. — [arXiv:2410.14273](https://arxiv.org/abs/2410.14273) find
  50. Jingxuan Zhang, Zhenhua Xu, Rui Hu, Wenpeng Xing, Xuhong Zhang, and Meng Han. 2025b. Meraser: An effective fingerprint erasure approach for large language models. Preprint, arXiv:2506.12551.
  51. Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. 2025c. Min-k%++: Improved baseline for pre-training data detection from large language models. In The Thirteenth International Conference on Learning Representations. find
  52. Xianren Zhang, Hui Liu, Delvin Ce Zhang, Xianfeng Tang, Qi He, Dongwon Lee, and Suhang Wang. 2025d. Does multimodal large language model truly unlearn? stealthy mllm unlearning attack. Preprint, arXiv:2506.17265. — [arXiv:2506.17265](https://arxiv.org/abs/2506.17265) find
  53. Xiaofan Zheng, Minnan Luo, and Xinghao Wang. 2025a. Unveiling fake news with adversarial arguments generated by multimodal large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7862–7869, Abu Dhabi, UAE. Association for Computational Linguisti… find
  54. Xiaofan Zheng, Zinan Zeng, Heng Wang, Yuyang Bai, Yuhan Liu, and Minnan Luo. 2025b. From predictions to analyses: Rationale-augmented fake news detection with large vision-language models. In Proceedings of the ACM on Web Conference 2025, WWW ’25, page 5364–5375, New York, NY, USA. Association for C… find
  55. Boyi Zeng, Lizheng Wang, Yuncong Hu, Yi Xu, Chenghu Zhou, Xinbing Wang, Yu Yu, and Zhouhan Lin. 2025. Huref: Human-readable fingerprint for large language models. Preprint, arXiv:2312.04828. — [arXiv:2312.04828](https://arxiv.org/abs/2312.04828) find
  56. Xiaofan Zheng, Huixuan Zhang, and Xiaojun Wan. 2025c. Tracing training footprints: A calibration approach for membership inference attacks against multimodal large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 17179–17191, Suzhou, China. Association… find
  57. Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. 2024a. Mm-llms: Recent advances in multimodal large language models. Preprint, arXiv:2401.13601.
  58. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.
  59. Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995–5004.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2026-ummf-protecting-copyright-large-vision/