Elamur Research⌕
☰
home / papers / 2026 / Fast-MIA: Efficient and Scalable Membership Inference for LLMs

Fast-MIA: Efficient and Scalable Membership Inference for LLMs

Paper · 2026 · Hiromu Takahashi and Shotaro Ishihara

Summary

Membership inference attacks (MIA) test whether a given text was part of an LLM's training data. They are central to auditing privacy leaks, copyright issues and benchmark contamination. Evaluating them has become costly for two reasons. Newer attacks run the model several times per sample, and audits increasingly work at the level of whole document collections. Existing code also runs each method separately, so shared quantities such as token log-probabilities are recomputed many times.

The authors present Fast-MIA, an open-source Python library that tackles this in two ways:

  • It runs batched inference on the vLLM engine, which is roughly five times faster than a standard Transformers pipeline with almost identical scores.
  • It caches intermediate outputs once and reuses them across all selected attacks.

Their experiments, which include an ablation of both mechanisms, show shorter runtimes and fewer model calls with only small changes in attack performance. This matters because it makes fair side-by-side comparisons of attack methods, hyperparameter sweeps and dataset-scale audits practical.

Key points

  • Implemented attacks: LOSS, PPL/zlib, Min-K% Prob, DC-PDD, Lowercase, PAC, ReCaLL, Con-ReCall and SaMIA. They are grouped into four families: baseline, token-distribution, text-alteration and black-box.
  • Cache effect: once LOSS has been computed, the token-distribution methods need no additional inference. Methods that require several passes each save one model call.
  • Configuration and data: experiments are set up through YAML files. The library connects to the WikiMIA and MIMIR benchmarks and can load input from Hugging Face datasets as well as from Parquet, JSONL, JSON and CSV files. It also handles non-English text.
  • Models: supports Hugging Face models, LoRA adapters and quantized models served by vLLM. Version 0.3.0 is built on vLLM 0.15.1.
  • Release and limitations: published under Apache 2.0 at github.com/Nikkei/fast-mia. The library is limited to vLLM-compatible autoregressive models and does not support closed APIs. The experiments cover a single model and dataset setup.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Hiromu Takahashi and Shotaro Ishihara. 2026. Fast-MIA: Efficient and Scalable Membership Inference for LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-demo.9

@inproceedings{takahashi2026fast,
  title = {{Fast-MIA: Efficient and Scalable Membership Inference for LLMs}},
  author = {Hiromu Takahashi and Shotaro Ishihara},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-demo.9},
  url = {https://doi.org/10.18653/v1/2026.acl-demo.9}
}

References

24 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. 2022. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. IEEE. find
  2. Ryuto Koike, Liam Dugan, Masahiro Kaneko, Chris Callison-Burch, and Naoaki Okazaki. 2025. Machine text detectors are membership inference attacks. arXiv [cs.CL]. find
  3. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21)…
  4. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611…
  5. Bowen Chen, Namgi Han, and Yusuke Miyao. 2025. A statistical and multi-perspective revisiting of the membership inference attack in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22854–22874, Vienna, Au… find
  6. Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. 2023. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, WWW ’23, page 3637–3647, New York, NY, USA. Association for Computing Machinery. find
  7. Yuetian Chen, Yuntao Du, Kaiyuan Zhang, Ashish Kundu, Charles Fleming, Bruno Ribeiro, and Ninghui Li. 2026. Window-based membership inference attacks against fine-tuned large language models. arXiv [cs.CL]. find
  8. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 157–165, Dublin, Ireland. Association for Computational Linguistics. find
  9. Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. 2024. LLM dataset inference: Did you train on my dataset? In Advances in Neural Information Processing Systems, volume 37, pages 124… find
  10. Cheng Wang, Yiwei Wang, Bryan Hooi, Yujun Cai, Nanyun Peng, and Kai-Wei Chang. 2025. Con-ReCall: Detecting pre-training data in LLMs via contrastive decoding. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1013–1026, Abu Dhabi, UAE. Association for Computatio… find
  11. Matthieu Meeus, Igor Shilov, Shubham Jain, Manuel Faysse, Marek Rei, and Yves-Alexandre de Montjoye. 2025. SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It) . In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 385–401, Los Alamitos, CA,… find
  12. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  13. Sasi Kumar Murakonda and Reza Shokri. 2020. ML privacy meter: Aiding regulatory compliance by quantifying the privacy risks of machine learning. arXiv [cs.CR]. find
  14. Haritz Puerto, Martin Gubri, Sangdoo Yun, and Seong Joon Oh. 2025. Scaling up membership inference: When and how attacks succeed on large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4165–4182, Albuquerque, New Mexico. Association for Computational… find
  15. Hengyu Wu and Yang Cao. 2025. Membership inference attacks on large-scale models: A survey. arXiv [cs.LG]. find
  16. Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Zhenqiang Gong, and Bhuwan Dhingra. 2024. ReCaLL: Membership inference via relative conditional log-likelihoods. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8671– 8689, Mia… find
  17. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  18. Mathieu Ravaut, Bosheng Ding, Fangkai Jiao, Hailin Chen, Xingxuan Li, Ruochen Zhao, Chengwei Qin, Caiming Xiong, and Shafiq Joty. 2025. A comprehensive survey of contamination detection methods in large language models. Transactions on Machine Learning Research. find
  19. Wentao Ye, Jiaqi Hu, Liyao Li, Haobo Wang, Gang Chen, and Junbo Zhao. 2024. Data contamination calibration for black-box LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10845–10861, Bangkok, Thailand. Association for Computational Linguistics. find
  20. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations. find
  21. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pages 268–282.
  22. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3–18.
  23. Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. Pre-training data detection for large language models: A divergence-based calibration method. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5263–5274, Mia… find
  24. Hiromu Takahashi and Shotaro Ishihara. 2025. Quantifying memorization in continual pre-training with Japanese general or industry-specific corpora. In Proceedings of the First Workshop on Large Language Model Memorization (L2M2), pages 95–105, Vienna, Austria. Association for Computational Linguisti… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/takahashi-2026-fast-mia-efficient-scalable-membership/