Elamur Research⌕
☰
home / papers / 2026 / Hey, That's My Data! Token-Only Dataset Inference in Large Language Models

Hey, That's My Data! Token-Only Dataset Inference in Large Language Models

Paper · 2026 · Chen Xiong, Zihao Wang, Rui Zhu et al.

Summary

Owners who suspect an LLM was trained on their proprietary or copyrighted text usually need log-probabilities to test this. Many commercial APIs, however, return only greedy top-1 tokens. The authors address this token-only setting with CatShift, which uses catastrophic forgetting as an auditing signal.

Each document in the suspect dataset is cut in half, giving a prompt and a reference completion. The data is then divided into train and test parts. The model is fine-tuned on the train part through a standard fine-tuning API. Completions on the test part are compared before and after fine-tuning using lexical overlap or BERT-based similarity. The same procedure is run on a validation set known to be unseen. A KS or Mann–Whitney U test then checks whether the suspect data changed significantly more.

The underlying idea is that training members shift more, because fine-tuning reactivates patterns the model already stores. Unseen data has to be learned from scratch, so its outputs change less. On Pythia models and GPT-3.5, CatShift clearly outperforms existing dataset inference baselines. This makes copyright audits feasible without access to logits or other internal signals.

Key points

  • Threat model: black-box access with greedy decoding only, a fine-tuning API, and a guaranteed non-member validation set used for calibration.
  • Pythia-410M results: CatShift reaches an AUC of 0.979. This is over 0.49 AUC above PDD and 0.19 above LLM DI, and CatShift keeps high TPR at 5% FPR, where both baselines collapse.
  • GPT-3.5 on BookMIA: member books give p = 6.44 × 10⁻⁵, while non-members give p = 0.711.
  • Robustness to logit noise (Pythia-410M, YouTube Subtitles subset of The Pile):
    • Noise levels 0.5–1.5 keep p below 0.1.
    • Level 2.0 hides membership (p = 0.248) but raises perplexity from about 22.9 to 54.5.
  • Limitations:
    • The signal is weaker for data that heavily overlaps with the rest of the training corpus.
    • Results are sensitive to fine-tuning hyperparameters.
    • Providers could apply further output restrictions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States139 works · 70,691 citations🎓Yale UniversityNew Haven · United States95 works · 12,765 citations🎓Chinese University of Hong KongHong Kong · China126 works · 14,073 citations🎓Indiana University BloomingtonBloomington · United States37 works · 3,685 citations🎓University of California, DavisDavis · United States28 works · 41,916 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Chen Xiong, Zihao Wang, Rui Zhu, Tsung-Yi Ho, Pin-Yu Chen, Jingwei Xiong and Haixu Tang. 2026. Hey, That's My Data! Token-Only Dataset Inference in Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.353

@inproceedings{xiong2026hey,
  title = {{Hey, That's My Data! Token-Only Dataset Inference in Large Language Models}},
  author = {Chen Xiong, Zihao Wang, Rui Zhu, Tsung-Yi Ho, Pin-Yu Chen, Jingwei Xiong and Haixu Tang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.353},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.353}
}

References

31 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. https://www-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/ Model_Card_Claude_3.pdf. ↗
  2. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across… find
  3. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large-Scale Autoregressive Language Modeling with Mesh-TensorFlow. Software available from Zenodo. find
  4. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium, USENIX Security…
  5. Zitao Chen and Karthik Pattabiraman. 2023. Overconfidence is a dangerous thing: Mitigating membership inference attacks by enforcing less confident prediction. arXiv preprint arXiv:2307.01610. — [arXiv:2307.01610](https://arxiv.org/abs/2307.01610) find
  6. Debeshee Das, Jie Zhang, and Florian Tramèr. 2024. Blind baselines beat membership inference attacks for foundation models. arXiv preprint arXiv:2406.16201. — [arXiv:2406.16201](https://arxiv.org/abs/2406.16201) find
  7. Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841.
  8. Adam Dziedzic, Haonan Duan, Muhammad Ahmad Kaleem, Nikita Dhawan, Jonas Guan, Yannis Cattan, Franziska Boenisch, and Nicolas Papernot. 2022. Dataset inference for self-supervised models. Advances in Neural Information Processing Systems, 35:12058–12070. find
  9. Hao Fang, Yixiang Qiu, Hongyao Yu, Wenbo Yu, Jiawei Kong, Baoli Chong, Bin Chen, Xuan Wang, and Shu-Tao Xia. 2024. Privacy leakage on dnns: A survey of model inversion attacks and defenses. CoRR, abs/2402.04013. — [arXiv:2402.04013](https://arxiv.org/abs/2402.04013) find
  10. Filippo Galli, Luca Melis, and Tommaso Cucinotta. 2024. Noisy neighbors: Efficient membership inference attacks against llms. arXiv preprint arXiv:2406.16565. find
  11. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027. find
  12. Isha Garg, Deepak Ravikumar, and Kaushik Roy. 2023. Memorization through the lens of curvature of loss function around samples. arXiv preprint arXiv:2307.05831. — [arXiv:2307.05831](https://arxiv.org/abs/2307.05831) find
  13. Yu He, Boheng Li, Liu Liu, Zhongjie Ba, Wei Dong, Yiming Li, Zhan Qin, Kui Ren, and Chun Chen. 2025. Towards {Label-Only} membership inference attack against pre-trained large language models. In 34th USENIX Security Symposium (USENIX Security 25), pages 1609–1628. find
  14. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  15. Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. 2024. Rethinking machine unlearning for large language models. CoRR, abs/2402.08787. find
  16. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. CoRR, abs/2308.08747. find
  17. Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. 2024. Llm dataset inference: Did you train on my dataset? Advances in Neural Information Processing Systems, 37:124069–124092. find
  18. Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. 2021. Dataset inference: Ownership resolution in machine learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. find
  19. Matthieu Meeus, Shubham Jain, Marek Rei, and Yves-Alexandre de Montjoye. 2024a. Inherent challenges of post-hoc membership inference for large language models. arXiv preprint arXiv:2406.17975. — [arXiv:2406.17975](https://arxiv.org/abs/2406.17975) find
  20. Matthieu Meeus, Igor Shilov, Manuel Faysse, and Yves-Alexandre de Montjoye. 2024b. Copyright traps for large language models. In 41st International Conference on Machine Learning. find
  21. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
  22. Deepak Ravikumar, Efstathia Soufleri, Abolfazl Hashemi, and Kaushik Roy. 2024. Unveiling privacy, memorization, and input curvature links. arXiv preprint arXiv:2402.18726. — [arXiv:2402.18726](https://arxiv.org/abs/2402.18726) find
  23. Jie Ren, Han Xu, Pengfei He, Yingqian Cui, Shenglai Zeng, Jiankun Zhang, Hongzhi Wen, Jiayuan Ding, Hui Liu, Yi Chang, and Jiliang Tang. 2024. Copyright protection in generative AI: A technical perspective. CoRR, abs/2402.02333. — [arXiv:2402.02333](https://arxiv.org/abs/2402.02333) find
  24. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations. find
  25. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017, pages 3–18.
  26. The New York Times. 2023. The Times Sues OpenAI and Microsoft Over A.I. Use of Copyrighted Work. https://www.nytimes.com/2023/12/27/business/media/ new-york-times-open-ai-microsoft-lawsuit. html. ↗
  27. Johnny Tian-Zheng Wei, Ryan Yixiang Wang, and Robin Jia. 2024. Proving membership in LLM pre-training data via data watermarks. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 13306–13320. Association for Comput… find
  28. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. In 31st IEEE Computer Security Foundations Symposium, CSF 2018, Oxford, United Kingdom, July 9-12, 2018, pages 268–282.
  29. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 27263–27277.
  30. Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. Pre-training data detection for large language models: A divergence-based calibration method. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, F… find
  31. Rui Zhu, Di Tang, Siyuan Tang, XiaoFeng Wang, and Haixu Tang. 2022. Selective amnesia: On efficient, high-fidelity and blind suppression of backdoor effects in trojaned machine learning models. CoRR, abs/2212.04687. — [arXiv:2212.04687](https://arxiv.org/abs/2212.04687) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiong-2026-hey-that-s-my-data/