Elamur Research⌕
☰
home / papers / 2026 / Investigating Memorization in Language Models Trained via Knowledge Distillation

Investigating Memorization in Language Models Trained via Knowledge Distillation

Summary

Knowledge distillation (KD) is a common way to build compact language models, but its effect on memorization of training data is poorly understood. This matters for both privacy and copyright. Mäcking and Regneri examine soft-label KD, where a student minimizes forward KL divergence to a teacher's output distribution, and compare it with ordinary hard-label training.

They continue pre-training Pythia-deduped models on the deduplicated Pile. They also insert 100 curated sequences, unseen in the Pile, into what the teacher, the student, or both are trained on. Memorization and forgetting of these sequences are tracked over training.

When teacher and student see different data, distillation strongly lowers memorization and speeds up forgetting of sequences the student had memorized. Privacy risk remains, however:

  • Teachers can leak memorized content to students that never saw it.
  • Students learn teacher-memorized sequences faster.

Teacher size creates a trade-off. Smaller teachers pass memorized content on more quickly, while larger teachers can eventually transfer more of it. These findings help practitioners weigh the usefulness of distilled models against their leakage risk.

Key points

  • Models: The teachers are Pythia-deduped 160M, 410M and 1B, and the student and hard-label baseline are 160M. All start from the 84K checkpoint with a fresh AdamW optimizer, use a KD temperature of 1, and train with batch size 128 and sequence length 128.
  • Injection set: There are 100 sequences of 129 tokens each, split evenly across five sources:
    • arXiv papers
    • BBC news
    • Mojo/Gleam code from GitHub
    • synthetic messages containing altered secrets (GPT-4o)
    • Wikipedia
  • Contamination check: Infini-gram and Data Portraits confirmed that each sequence overlaps the Pile by fewer than 10 consecutive tokens and fewer than 50 characters.
  • Training data: The memorization data (D plain) is a 2,048,000-example Pile slice from steps 85K–87K, equal to 16,000 update steps. Forgetting is measured on steps 87K–89K (D forget).
  • Teacher size and forgetting: In one setting, a larger teacher sped up the student's forgetting even though the teacher had itself memorized the sequences. Code is available at github.com/mcmaart/kd-memorization.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/39ec72wwr6ux

@inproceedings{macking2026investigating,
  title = {{Investigating Memorization in Language Models Trained via Knowledge Distillation}},
  author = {Mäcking, Maarten and Regneri, Michaela},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/39ec72wwr6ux},
  url = {https://doi.org/10.63317/39ec72wwr6ux}
}

References

48 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. 2023a. Emergent and predictable memorization in large language models. In Advances in Neural Information Processing Systems, volume 36, pages 28072–28090. Curran Associates, In… find
  2. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023b. Pythia: A suite for analyzing large language models acros… find
  3. Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy, pages 141–159. find
  4. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In International Conference on Learning Representations. find
  5. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267–284, Santa Clara, CA. USENIX Association. find our card
  6. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21)… find our card
  7. Harsh Chaudhari, Jamie Hayes, Matthew Jagielski, Ilia Shumailov, Milad Nasr, and Alina Oprea. 2025. Cascading adversarial bias from injection to distillation in language models. In ICML 2025 Workshop: Data in Generative Models (The Bad, the Ugly, and the Greats), Vancouver, Canada. find
  8. Jiaao Chen and Diyi Yang. 2023. Unlearn what you want to forget: Efficient unlearning for LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12041–12052, Singapore. Association for Computational Linguistics. find
  9. Jang Hyun Cho and Bharath Hariharan. 2019. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4794–4802. find
  10. A. Feder Cooper, Aaron Gokaslan, Amy B. Cyphert, Christopher De Sa, Mark Lemley, Daniel E. Ho, and Percy Liang. 2025. Extracting memorized pieces of (copyrighted) books from open-weight language models. In ICML 2025 Workshop on Reliable and Responsible Foundation Models. find
  11. Verna Dankers and Vikas Raunak. 2025. Memorization inheritance in sequence-level knowledge distillation for neural machine translation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 760–774, Vienna, Austria. Association for… find
  12. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography, pages 265–284. Springer Berlin Heidelberg. find
  13. Vitaly Feldman. 2020. Does learning require memorization? A short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954–959, Chicago, IL, USA. Association for Computing Machinery. find
  14. Vitaly Feldman and Chiyuan Zhang. 2020. What neural networks memorize and why: Discovering the long tail via influence estimation. In Advances in Neural Information Processing Systems, volume 33, pages 2881–2891. Curran Associates, Inc. find
  15. Joshua Freeman, Chloe Rippe, Edoardo Debenedetti, and Maksym Andriushchenko. 2024. Exploring memorization and copyright violation in frontier LLMs: A study of the New York Times v. OpenAI 2023 lawsuit. In NeurIPS Safe Generative AI Workshop 2024. find
  16. Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born again neural networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1607–1616. PMLR. find
  17. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. find
  18. Google. 2024. Moving from experimentation into production with Gemini models and Vertex AI. Accessed: 2025-08-02. find
  19. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. find our card
  20. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simo… find
  21. Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022a. Are large pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2038–2047. Association for Computational Linguistics. find
  22. Jing Huang, Diyi Yang, and Christopher Potts. 2024. Demystifying verbatim memorization in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10711–10732, Miami, Florida, USA. Association for Computational Linguistics. find
  23. Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. 2022b. Knowledge distillation from a stronger teacher. In Advances in Neural Information Processing Systems, volume 35, pages 33716–33727. Curran Associates, Inc. find
  24. Matthew Jagielski, Milad Nasr, Katherine Lee, Christopher A Choquette-Choo, Nicholas Carlini, and Florian Tramèr. 2023a. Students parrot their teachers: Membership inference on model distillation. In Advances in Neural Information Processing Systems, volume 36, pages 44382– 44397. Curran Associates,… find
  25. Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Guha Thakurta, Nicolas Papernot, and Chiyuan Zhang. 2023b. Measuring forgetting of memorized training examples. In International Conference on Learning Representation… find
  26. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174, Online. Association for Computational Linguis… find
  27. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. find
  28. Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. 2023. Copyright violations and large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7403–7412, Singapore. Association for Computational Linguistics. find
  29. Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas. Association for Computational Linguistics. find our card
  30. Yucheng Li, Frank Guerin, and Chenghua Lin. 2024. LatestEval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):18600–18607. find
  31. Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. 2024. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. In First Conference on Language Modeling. find
  32. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  33. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2025. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing, 33:3776–3786. find
  34. Marc Marone and Benjamin Van Durme. 2023. Data Portraits: Recording foundation model training data. In Advances in Neural Information Processing Systems, volume 36, pages 15121– 15135. Curran Associates, Inc. find
  35. Federico Mazzone, Leander van den Heuvel, Maximilian Huber, Cristian Verdecchia, Maarten Everts, Florian Hahn, and Andreas Peter. 2022. Repeated knowledge distillation with confidence masking to mitigate membership inference attacks. In Proceedings of the 15th ACM Workshop on Artificial Intelligence… find
  36. John X Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G Edward Suh, Alexander M Rush, Kamalika Chaudhuri, and Saeed Mahloujifar. 2025. How much do language models memorize? arXiv preprint arXiv:2505.24832. — [arXiv:2505.24832](https://arxiv.org/abs/2505.24832) find
  37. OpenAI. 2024. Model distillation in the API. Accessed: 2025-08-02. find
  38. Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, and Kunal Talwar. 2017. Semi-supervised knowledge transfer for deep learning from private training data. In International Conference on Learning Representations. find
  39. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. find
  40. Virat Shejwalkar and Amir Houmansadr. 2021. Membership privacy for machine learning models through knowledge transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9549–9557. find
  41. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy, pages 3–18. find our card
  42. Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization without overfitting: Analyzing the training dynamics of large language models. In Advances in Neural Information Processing Systems, volume 35, pages 38274–38290. Curran Associates, Inc. find our card
  43. Stefan Vasilev, Christian Herold, Baohao Liao, Seyyed Hadi Hashemi, Shahram Khadivi, and Christof Monz. 2025. Unilogit: Robust machine unlearning for LLMs using uniform-target self-distillation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22453–22472, Vienna, Austri… find
  44. Bichen Wang, Yuzhe Zi, Yixin Sun, Yanyan Zhao, and Bing Qin. 2025. Balancing forget quality and model utility: A reverse KL-divergence knowledge distillation approach for better unlearning in LLMs. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Co… find
  45. Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024. Large language model unlearning. In Advances in Neural Information Processing Systems, volume 37, pages 105425–105475. Curran Associates, Inc. find our card
  46. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling. find our card
  47. Ziqi Zhang, Ali Shahin Shamsabadi, Hanxiao Lu, Yifeng Cai, and Hamed Haddadi. 2025. Membership and memorization in LLM knowledge distillation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20074–20084, Suzhou, China. Association for Computational Li… find
  48. Language Resource References MacFarlane, John and Krewinkel, Albert and Rosenthal, Jesse. 2025. Pandoc. Zenodo. PID https://doi.org/10.5281/zenodo.15542972. Version 3.7.0.2. — [doi:10.5281/zenodo.15542972](https://doi.org/10.5281/zenodo.15542972) ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/macking-2026-investigating-memorization-language-models-trained/