Elamur Research⌕
☰
home / papers / 2026 / Black-Box Membership Inference Attacks for Video Training Data in Multimodal Large Language Models

Black-Box Membership Inference Attacks for Video Training Data in Multimodal Large Language Models

Paper · 2026 · Jinrui Wang, Zhenfeng Gao, Wendan Wang et al.

Summary

The paper tackles a practical question: can one tell whether a particular video was used to train a multimodal large language model (MLLM) when only the model's text output is visible? Earlier video membership inference attacks (MIAs) have two weaknesses. Concept-based methods such as DIS-CO check whether the model recognises things like movie titles, but that knowledge may come from Wikipedia or other corpora. Vid-SME relies on logits, which closed commercial systems do not expose. The authors propose VideoMIA, a black-box framework that probes whether the model remembers the temporal order of frames, something general world knowledge cannot supply. It uses two tasks of rising difficulty:

  • picking the right candidate frame relative to a set of context frames;
  • reordering shuffled subsequent frames.

High success rates on these tasks indicate membership. The authors also build three new benchmarks, including YoutubeTection, and test on ten MLLMs across four benchmarks. Existing methods fall to near-chance on YoutubeTection, while VideoMIA beats all baselines. This makes copyright and privacy auditing of closed models more feasible.

Key points

  • Two temporal tasks: VideoMIA combines FrameIdentify (Nearest-Frame Identification) and TempRank (Temporal Frame Ranking). By default their scores are weighted equally, and results stay stable for weights from 0.3/0.7 to 0.7/0.3.
  • Benchmarks: Three new benchmarks are released. Two come from open-source video LLMs with documented training data, and YoutubeTection is built from YouTube videos so that closed-source models can be tested.
  • Robustness to scoring choices: Swapping exponential decay for linear decay, or pairwise AUC for Kendall's τ, gives similar results. Average AUC is about 0.59 on LLaVA-7B, LLaVA-34B, MiniCPM and LLaVA-Qwen2.
  • Set-level detection: Averaging membership scores over a set of 100 videos makes members and non-members almost perfectly separable.
  • Hyperparameters: Performance improves with more context frames. With more test samples per video it rises and then plateaus; the defaults are N=15 and x=3.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Beijing University of Posts and TelecommunicationsBeijing · China57 works · 32,843 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Jinrui Wang, Zhenfeng Gao, Wendan Wang, Huili Wang, Zichen Qin, Linjie Zhu, Hongke Fu, Shangguang Wang and Tao Qi. 2026. Black-Box Membership Inference Attacks for Video Training Data in Multimodal Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1820

@inproceedings{wang2026black,
  title = {{Black-Box Membership Inference Attacks for Video Training Data in Multimodal Large Language Models}},
  author = {Jinrui Wang, Zhenfeng Gao, Wendan Wang, Huili Wang, Zichen Qin, Linjie Zhu, Hongke Fu, Shangguang Wang and Tao Qi},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1820},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1820}
}

References

34 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. 2022. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. IEEE. find
  3. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2024. Sharegpt4v: Improving large multi-modal models with better captions. In Proceedings of European Conference on Computer Vision, pages 370– 387. Springer.
  4. Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. Do membership inference attacks work on large language models? In Proceedings of the First Conference on Language Modeling.
  5. André V Duarte, Xuandong Zhao, Arlindo L Oliveira, and Lei Li. 2025. Dis-co: Discovering copyrighted content in vlms training data. In Proceedings of International Conference on Machine Learning, pages 14807–14832. find
  6. Maria Dolores Esteban and Domingo Morales. 1995. A summary on entropy statistics. Kybernetika, 31(4):337–346. find
  7. Shahriar Golchin and Mihai Surdeanu. 2024. Time travel in llms: Tracing data contamination in large language models. In Proceedings of the 12th International Conference on Learning Representations.
  8. David J Hand and Robert J Till. 2001. A simple generalisation of the area under the roc curve for multiple class classification problems. Machine learning, 45(2):171–186. find
  9. Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. 2022. Membership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR), 54(11s):1–37. find
  10. Yuke Hu, Zheng Li, Zhihao Liu, Yang Zhang, Zhan Qin, Kui Ren, and Chun Chen. 2025. Membership inference attacks against vision-language models. In Proceedings of the 34th USENIX Security Symposium (USENIX Security 25), pages 1589–1608. find
  11. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  12. Bargav Jayaraman, Chuan Guo, and Kamalika Chaudhuri. 2024. Déjà vu memorization in vision–language models. In Proceedings of Advances in Neural Information Processing Systems, pages 50722–50749. find
  13. Narine Kokhlikyan, Bargav Jayaraman, Florian Bordes, Chuan Guo, and Kamalika Chaudhuri. 2024. Measuring deja vu memorization efficiently. In Proceedings of Advances in Neural Information Processing Systems, pages 30015–30040. find
  14. Qi Li, Cheng-Long Wang, Yinzhi Cao, and Di Wang. 2024a. Data lineage inference: Uncovering privacy vulnerabilities of dataset pruning. arXiv preprint arXiv:2411.15796. — [arXiv:2411.15796](https://arxiv.org/abs/2411.15796) find
  15. Qi Li, Runpeng Yu, and Xinchao Wang. 2025. Vid-sme: Membership inference attacks against large video understanding models. In Proceedings of Annual Conference on Neural Information Processing Systems. find
  16. Zhan Li, Yongtao Wu, Yihang Chen, Francesco Tonin, Elias Abad Rocamora, and Volkan Cevher. 2024b. Membership inference attacks against large vision-language models. In Proceedings of Advances in Neural Information Processing Systems, pages 98645–98674. understanding across millions of tokens of cont…
  17. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306.
  18. Cheng-Long Wang, Qi Li, Zihang Xiang, Yinzhi Cao, and Di Wang. 2025. Towards lifecycle unlearning commitment management: Measuring sample-level unlearning completeness. In Proceedings of 34th USENIX Security Symposium (USENIX Security 25), pages 6481–6500. find
  19. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Proceedings of Advances in Neural Information Processing Systems, pages 34892–34916.
  20. Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. 2018. Understanding membership inferences on well-generalized learning models. arXiv preprint arXiv:1802.04889. — [arXiv:1802.04889](https://arxiv.org/abs/1802.04889) find
  21. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. 2024. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 12585–12602.
  22. Yonatan Oren, Nicole Meister, Niladri S Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. 2024. Proving test set contamination in black-box language models. In Proceedings of the 12th International Conference on Learning Representations. find
  23. Francesco Pinto, Nathalie Rauschmayr, Florian Tramèr, Philip Torr, and Federico Tombari. 2024. Extracting training data from document-based vqa models. In Proceedings of International Conference on Machine Learning, pages 40813–40826. find
  24. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. In Proceedings of the 12th International Conference on Learning Representations. find
  25. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
  26. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, and 1 others. 2024b. Cogvlm: Visual expert for pretrained language models. In Proceedings of Advances in Neural Information Processing Systems, pages 121475–121499.
  27. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. In Proceedings of the 31st IEEE Computer Security Foundations Symposium (CSF), pages 268–282. IEEE.
  28. Shukang Yin, Chaoyou Fu, Sirui Zhao, Chunjiang Ge, Yan Yang, Yuhan Dai, Yongdong Luo, Tong Xu, Caifeng Shan, and Enhong Chen. 2026. Sparrow: Data-efficient video-llm with text-to-image augmentation. IEEE Transactions on Multimedia. find
  29. Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. 2024. Low-cost high-power membership inference attacks. In Proceedings of the 41st International Conference on Machine Learning, pages 58244–58282. find
  30. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2025. Llava-video: Video instruction tuning with synthetic data. Transactions on Machine Learning Research. find
  31. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3–18. IEEE.
  32. Liwei Song and Prateek Mittal. 2021. Systematic evaluation of privacy risks of machine learning models. In Proceedings of the 30th USENIX Security Symposium (USENIX Security 21), pages 2615–2632. find
  33. Liwei Song, Reza Shokri, and Prateek Mittal. 2019. Membership inference attacks against adversarially robust deep learning models. In 2019 IEEE Security and Privacy Workshops (SPW), pages 50–56. IEEE. find
  34. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. Gemini 1.5: Unlocking multimodal find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-black-box-membership-inference-attacks/