Elamur Research⌕
☰
home / papers / 2026 / PDR: A Plug-and-Play Positional Decay Framework for LLM Pre-training Data Detection

PDR: A Plug-and-Play Positional Decay Framework for LLM Pre-training Data Detection

Paper · 2026 · Jinhan Liu, Yibo Yang, Ruiying Lu et al.

Summary

The paper addresses pre-training data detection: deciding whether a given text was part of an LLM's training corpus. This matters for privacy audits, copyright checks and test-set contamination analysis. The setting assumes only output probabilities are available, with no training and no reference data. The authors note that likelihood-based detectors such as Loss, Min-k% and Min-k%++ weight every included token equally when combining token scores. Information theory suggests otherwise: extra context cannot raise entropy, so predictive uncertainty tends to fall along a sequence. Measurements on MIMIR subsets confirm that token entropy is high at the start and drops quickly. The authors conclude that confident predictions on early tokens carry the most membership evidence. They propose Positional Decay Reweighting (PDR), which multiplies token scores by a monotonically decreasing positional weight. The weight can follow a linear, exponential or polynomial shape. PDR needs no training and adds no compute. It can be attached to existing scores, including inside FSD, and it usually improves them on WikiMIA and MIMIR.

Key points

  • Core observation: Membership signal concentrates in the early, high-entropy tokens and weakens later. With more context, members and non-members both become easy to predict, so their difference shrinks.
  • Method: PDR replaces uniform aggregation with decaying positional weights. It is a drop-in prior for likelihood-based scores (Loss, Min-k%, Min-k%++) and also works within the fine-tuning-based FSD framework.
  • Main result: The largest reported gain is up to 4.7 AUROC points over Min-k%++ on WikiMIA with sequence length 128.
  • Evaluation setup:
    • Benchmarks: WikiMIA and MIMIR.
    • Models: open-weight Pythia, LLaMA (e.g., LLaMA-13B), OPT and others.
    • Access: only output logits or probabilities, under the black-/grey-box assumption.
  • Limitations:
    • Experiments are English-only. Tokenization differences across languages, including low-resource ones such as Ukrainian, may change the best decay shape.
    • On MIMIR, members and non-members are highly similar, so signals stay weak and detection remains hard.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations🎓King Abdullah University of Science and TechnologyJeddah · Saudi Arabia27 works · 1,084 citations🎓Jilin UniversityChangchun · China25 works · 5,252 citations🎓Xidian UniversityXi'an · China14 works · 1,357 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Jinhan Liu, Yibo Yang, Ruiying Lu, Piotr Piękos, Yimeng Chen, Peng Wang and Dandan Guo. 2026. PDR: A Plug-and-Play Positional Decay Framework for LLM Pre-training Data Detection. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.562

@inproceedings{liu2026pdr,
  title = {{PDR: A Plug-and-Play Positional Decay Framework for LLM Pre-training Data Detection}},
  author = {Jinhan Liu, Yibo Yang, Ruiying Lu, Piotr Piękos, Yimeng Chen, Peng Wang and Dandan Guo},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.562},
  url = {https://doi.org/10.18653/v1/2026.acl-long.562}
}

References

38 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Martin Bertran, Shuai Tang, Aaron Roth, Michael Kearns, Jamie H Morgenstern, and Steven Z Wu. 2023. Scalable membership inference attacks via quantile regression. Advances in Neural Information Processing Systems, 36:314–330. find
  3. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, and 1 others. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Co… find
  4. Sidney Black, Stella Biderman, Eric Hallahan, Quentin Gregory Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Martin Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT-neox-20b: An op…
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  6. Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. 2022. Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP), pages 1897–1914. IEEE. find
  7. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650…
  8. Hongyan Chang, Ali Shahin Shamsabadi, Kleomenis Katevas, Hamed Haddadi, and Reza Shokri. 2025. Context-aware membership inference attacks against pre-trained large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 7299–7321. find
  9. Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841.
  10. André V. Duarte, Xuandong Zhao, Arlindo L. Oliveira, and Lei Li. 2024. DE-COP: Detecting Copyrighted Content in Language Models Training Data. Preprint, arXiv:2402.09910. — [arXiv:2402.09910](https://arxiv.org/abs/2402.09910) find
  11. Jan Dubiński, Antoni Kowalczuk, Stanisław Pawlak, Przemyslaw Rokita, Tomasz Trzciński, and Paweł Morawiecki. 2024. Towards more realistic membership inference attacks on large diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4860–4… find
  12. Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. 2025. MIA-tuner: Adapting large language models as pre-training text detector. In Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, Pennsylvania, USA. find
  13. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, and 1 others. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. find
  14. Michael M Grynbaum and Ryan Mac. 2023. The times sues openai and microsoft over ai use of copyrighted work. The New York Times, 27(1). find
  15. Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. find
  16. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022a. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  17. Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. 2022b. Membership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR), 54(11s):1–37. find
  18. Zhiheng Huang, Yannan Liu, Daojing He, and Yu Li. 2025. Df-mia: A distribution-free membership inference attack on fine-tuned large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 343–351. find
  19. Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. 2023. Membership inference attacks against language models via neighbourhood comparison. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11330– 11…
  20. Matthieu Meeus, Shubham Jain, Marek Rei, and Yves-Alexandre de Montjoye. 2023. Did the neurons read your book? document-level membership inference for large language models. arXiv preprint arXiv:2310.15007. — [arXiv:2310.15007](https://arxiv.org/abs/2310.15007) find
  21. Lauren Watson, Chuan Guo, Graham Cormode, and Alexandre Sablayrolles. 2022. On the importance of difficulty calibration in membership inference attacks. In International Conference on Learning Representations. find
  22. Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. 2022. Quantifying privacy risks of masked language models using membership inference attacks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8332– 8347…
  23. Hengyu Wu and Yang Cao. 2025. Membership inference attacks on large-scale models: A survey. arXiv preprint arXiv:2503.19338. — [arXiv:2503.19338](https://arxiv.org/abs/2503.19338) find
  24. Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D Griffin. 2023. Use of llms for illicit purposes: Threats, prevention measures, and vulnerabilities. arXiv preprint arXiv:2308.12833. — [arXiv:2308.12833](https://arxiv.org/abs/2308.12833) find
  25. Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B Hashimoto. 2023. Proving test set contamination in black box language models. arXiv preprint arXiv:2310.17623. find
  26. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  27. Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423. find
  28. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations. find
  29. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE.
  30. Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Zhenqiang Gong, and Bhuwan Dhingra. 2024. ReCaLL: Membership inference via relative conditional log-likelihoods. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8671– 8689, Mia… find
  31. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268–282. IEEE.
  32. Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. 2023. Low-cost high-power membership inference attacks. arXiv preprint arXiv:2312.03262. — [arXiv:2312.03262](https://arxiv.org/abs/2312.03262) find
  33. Hengxiang Zhang, Songxin Zhang, Bingyi Jing, and Hongxin Wei. 2025a. Fine-tuning can help detect pretraining data from large language models. In The Thirteenth International Conference on Learning Representations. find
  34. Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. 2025b. Min-k%++: Improved baseline for pre-training data detection from large language models. In The Thirteenth International Conference on Learning Representations. find
  35. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, and 1 others. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
  36. Jiashu Tao and Reza Shokri. 2025. Range membership inference attacks. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 346–361. IEEE. find
  37. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-pdr-plug-play-positional-decay/