Elamur Research⌕
☰
home / papers / 2026 / Memorization or Lucky Guesses: Detecting Short Sequences from Copyrighted Dutch News in LLM Output

Memorization or Lucky Guesses: Detecting Short Sequences from Copyrighted Dutch News in LLM Output

Summary

Smaller publishers, especially outside the English-speaking world, have trouble proving that LLMs were trained on their content, because models rarely reproduce their articles at length. The authors ask whether brief, recurring fragments can serve as evidence of training exposure instead. They take Dutch-language articles from four Dutch news outlets in mC4 and run extraction attacks on GPT-4 and on mT5, which was trained on mC4. The models are given either randomly truncated articles or outlet-specific strings that recur often, and their continuations are checked for verbatim matches. As a control for chance matches, the authors build a comparison set of articles published after GPT-4's training cutoff. They then treat the length of a verbatim continuation as a time-to-event variable and apply survival analysis to compare the two sets. Randomly chosen articles show no reliable difference from the control. Repeated editorial boilerplate, however, such as column openings and calls to action, is completed verbatim significantly more often. This gives modest-sized outlets a practical way to audit models for use of their content.

Key points

  • The four outlets and their article counts in mC4 are: ad.nl (141,985), nrc.nl (594,438), volkskrant.nl (162,668) and groene.nl (32,877).
  • The baseline uses the CC-MAIN-2023-23 Common Crawl snapshot, restricted to articles from 2022 onward and cleaned according to mC4's procedure. Articles for nrc.nl, which blocks Common Crawl, came from the Media-Cloud API instead. The baseline contains 500 articles per outlet.
  • GPT-4 (gpt-4-0613) was queried at temperature zero with no system prompt. For random articles, 1,000 were sampled per outlet, each cut at five random points.
  • mT5 models of several sizes were tested by masking spans of up to three tokens. Memorization increased with model size, but only for strings that were repeated in mC4.
  • Editorial boilerplate showed the strongest signal, notably for ad.nl and nrc.nl. groene.nl had only 25 duplicate strings and showed no evidence of memorization, which suggests some minimum publication volume is needed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3iqgydmfqiju

@inproceedings{veerbeek2026memorization,
  title = {{Memorization or Lucky Guesses: Detecting Short Sequences from Copyrighted Dutch News in LLM Output}},
  author = {Veerbeek, Joris and Berendsen, Kas and Polimeno, Alessandra and Bosch, Antal van den},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3iqgydmfqiju},
  url = {https://doi.org/10.63317/3iqgydmfqiju}
}

References

14 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. ↗
  2. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650. ↗ our card
  3. Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023. Speak, memory: An archaeology of books known to chatgpt/gpt-4. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7312–7327. find
  4. Nick Diakopoulos. 2024. Fair use, copyright, and the challenge of memorization in the nyt vs. openai. https://medium.com/generative-a i-in-the-newsroom/fair-use-copyrig ht-and-the-challenge-of-memorizat ion-in-the-nyt-vs-openai-7f6c0a13f 703. Published in Generative AI in the Newsroom, Medium, February 1, 2024. ↗
  5. Thomas R Fleming and David P Harrington. 1981. A class of hypothesis tests for one and two sample censored survival data. Communications in Statistics-Theory and Methods, 10(8):763–794. ↗
  6. Jing Huang, Diyi Yang, and Christopher Potts. 2024. Demystifying verbatim memorization in large language models. arXiv preprint arXiv:2407.17817. ↗
  7. Edward L Kaplan and Paul Meier. 1958. Nonparametric estimation from incomplete observations. Journal of the American statistical association, 53(282):457–481. ↗
  8. Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. 2023. Copyright violations and large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7403–7412. find
  9. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424–8445. ↗
  10. Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023. Scalable extraction of training data from (production) language models. CoRR. ↗ our card
  11. New York Times. 2023. One hundred examples of GPT-4 memorizing content from the New York Times. Document 1-68, Exhibit J. find
  12. Sebastian Pölsterl. 2020. scikit-survival: A library for time-to-event analysis built on top of scikit-learn. Journal of Machine Learning Research, 21(212):1–6. find
  13. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  14. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/veerbeek-2026-memorization-or-lucky-guesses-detecting/