Elamur Research
☰
home / papers / 2023 / Scalable Extraction of Training Data from (Production) Language Models

Scalable Extraction of Training Data from (Production) Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2311.17035

Summary

The paper asks how much training data an attacker can actually pull out of a language model without knowing what the model was trained on. The authors call this "extractable memorization." Earlier work relied either on slow manual verification or on prompting models with real training text. This paper closes that gap with an automated pipeline.

For open models, the authors sample short Wikipedia snippets as prompts. They then check the generated text against the published training corpora using suffix arrays. For ChatGPT, whose data is secret, they compare outputs with a roughly nine-terabyte merged collection of public web datasets.

Ordinary attacks are enough for unaligned models, and larger models leak more. The aligned gpt-3.5-turbo looks almost leak-free under these attacks. However, a new divergence attack based on token repetition pushes it out of assistant mode and makes it produce memorized text at a far higher rate.

The results show that real extraction risk has been greatly underestimated. They also show that alignment hides memorization rather than removing it.

Key points

  • Models tested: Pythia and GPT-Neo (open), LLaMA and Falcon (semi-open), and ChatGPT (gpt-3.5-turbo). In total, terabytes of output were generated from 17 open and semi-closed models.
  • Success criterion: An extraction counts only if the output contains a verbatim match of at least 50 tokens with the training data. Matches are found through a suffix array lookup. The threshold is double the longest accidental overlap (25 tokens) observed between news articles written after RedPajama was built.
  • ChatGPT results: Under standard attacks, ChatGPT appears about 50× more private than earlier models. The divergence attack makes it emit training data about 150× more often, and about 3× more often than the base model.
  • Cost and scale: Over 10,000 ChatGPT training examples were recovered for about $200 in queries. The authors' extrapolation suggests more than 10× that amount is reachable with more queries.
  • Memorization in open models: GPT-Neo 6B can emit nearly a gigabyte of training data, which suggests that about 10% of its capacity holds verbatim memorized text.
  • Disclosure: The flaw was found on July 11 and reported to OpenAI on August 30, followed by a 90-day disclosure window before publication.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

115 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 107 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.17035 · arXiv 2311.17035

@misc{nasr2023scalable,
  title = {{Scalable Extraction of Training Data from (Production) Language Models}},
  author = {Nasr, Milad and Carlini, Nicholas and Hayase, Jonathan and Jagielski, Matthew and Cooper, A. Feder and Ippolito, Daphne and Choquette-Choo, Christopher A. and Wallace, Eric and Tramer, Florian and Lee, Katherine},
  year = {2023},
  doi = {10.48550/arxiv.2311.17035},
  eprint = {2311.17035},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.17035}
}

References

55 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A NDERSSON , O. Sequential Good-Turing and the missing species problem. find
  2. A NIL , R., D AI , A. M., F IRAT , O., ET AL . PaLM 2 Technical Report, 2023. find
  3. B AI , Y., J ONES , A., N DOUSSE , K., A SKELL , A., C HEN , A., D AS S ARMA , N., D RAIN , D., F ORT , S., G ANGULI , D., H ENIGHAN , T., ET AL . Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022). find
  4. B ALLE , B., C HERUBIN , G., AND H AYES , J. Reconstructing training data with informed adversaries. In IEEE S&P (2022). find
  5. B IDERMAN , S., S CHOELKOPF , H., A NTHONY , Q., B RADLEY , H., O’B RIEN , K., H ALLAHAN , E., K HAN , M. A., P UROHIT , S., P RASHANTH , U. S., R AFF , E., S KOWRON , A., S UTAWIKA , L., AND VAN DER W AL , O. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling, 2023. find
  6. B LACK , S., G AO , L., W ANG , P., L EAHY , C., AND B I - DERMAN , S. GPT-Neo: Large scale autoregressive language modeling with Mesh-Tensorflow, 2021. find
  7. B ROWN , H., L EE , K., M IRESHGHALLAH , F., S HOKRI , R., AND T RAMÈR , F. What does it mean for a language model to preserve privacy? In ACM FAccT (2022). find
  8. B ROWN , T. B., M ANN , B., R YDER , N., S UBBIAH , M., K APLAN , J., D HARIWAL , P., N EELAKANTAN , A., S HYAM , P., ET AL . Language models are few-shot learners. In NeurIPS (2020). find
  9. C ARLINI , N., C HIEN , S., N ASR , M., S ONG , S., T ERZIS , A., AND T RAMER , F. Membership inference attacks from first principles. In IEEE Symposium on Security and Privacy (2022), IEEE. find
  10. C ARLINI , N., H AYES , J., N ASR , M., J AGIELSKI , M., S EHWAG , V., T RAMER , F., B ALLE , B., I PPOLITO , D., AND W ALLACE , E. Extracting training data from diffusion models. In USENIX Security Symposium (2023). find
  11. C ARLINI , N., I PPOLITO , D., J AGIELSKI , M., L EE , K., T RAMER , F., AND Z HANG , C. Quantifying memorization across neural language models. In ICLR (2023). find
  12. C ARLINI , N., L IU , C., E RLINGSSON , Ú., K OS , J., AND S ONG , D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium (2019). find our card
  13. C ARLINI , N., N ASR , M., C HOQUETTE -C HOO , C. A., J AGIELSKI , M., G AO , I., A WADALLA , A., K OH , P. W., I PPOLITO , D., L EE , K., T RAMER , F., ET AL . Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447 (2023). find
  14. C ARLINI , N., T RAMER , F., W ALLACE , E., J AGIEL - SKI , M., H ERBERT -V OSS , A., L EE , K., R OBERTS , A., B ROWN , T., S ONG , D., E RLINGSSON , U., ET AL . Extracting training data from large language models. In USENIX Security Symposium (2021). find our card
  15. C HAO , A. Nonparametric estimation of the number of classes in a population. Scandinavian Journal of statistics (1984), 265–270. find
  16. C HIU , C.-H., W ANG , Y.-T., W ALTHER , B. A., AND C HAO , A. An improved nonparametric lower bound of species richness via a modified good–turing frequency formula. Biometrics 70, 3 (2014), 671–682. find
  17. C HOQUETTE -C HOO , C. A., T RAMER , F., C ARLINI , N., AND P APERNOT , N. Label-only membership inference attacks. In International conference on machine learning (2021), PMLR, pp. 1964–1974. find
  18. C HRISTIANO , P. F., L EIKE , J., B ROWN , T., M ARTIC , M., L EGG , S., AND A MODEI , D. Deep reinforcement learning from human preferences. NeurIPS (2017). find
  19. C OMPUTER , T. RedPajama: An open source recipe to reproduce LLaMA training dataset, 2023. find
  20. C OMPUTER , T. Releasing 3B and 7B RedPajama-INCITE family of models including base, instruction-tuned & chat models, 2023. find
  21. F REDRIKSON , M., J HA , S., AND R ISTENPART , T. Model inversion attacks that exploit confidence information and basic countermeasures. In ACM Conference on Computer and Communications Security (CCS) (2015). find
  22. G ALE , W. A., AND S AMPSON , G. Good-Turing frequency estimation without tears. Journal of quantitative linguistics 2, 3 (1995), 217–237. find
  23. G AO , L., B IDERMAN , S., B LACK , S., G OLDING , L., H OPPE , T., F OSTER , C., P HANG , J., H E , H., T HITE , A., N ABESHIMA , N., ET AL . The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020). find
  24. G OOD , I. J. The population frequencies of species and the estimation of population parameters. Biometrika 40, 3-4 (1953), 237–264. find
  25. H OFFMANN , J., B ORGEAUD , S., M ENSCH , A., B UCHATSKAYA , E., C AI , T., R UTHERFORD , E., C ASAS , D. D . L., H ENDRICKS , L. A., W ELBL , J., C LARK , A., ET AL . Training compute-optimal large language models. In NeurIPS (2022). find
  26. H OFFMANN , J., B ORGEAUD , S., M ENSCH , A., B UCHATSKAYA , E., C AI , T., R UTHERFORD , E., DE L AS C ASAS , D., H ENDRICKS , L. A., W ELBL , J., C LARK , A., ET AL . An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems 35 (2022)… find
  27. I SHIHARA , S. Training data extraction from pre-trained language models: A survey, 2023. find
  28. J IANG , A. Q., S ABLAYROLLES , A., M ENSCH , A., B AMFORD , C., C HAPLOT , D. S., DE LAS C ASAS , D., B RESSAND , F., L ENGYEL , G., L AMPLE , G., S AULNIER , L., L AVAUD , L. R., L ACHAUX , M.-A., S TOCK , P., S CAO , T. L., L AVRIL , T., W ANG , T., L ACROIX , T., AND S AYED , W. E. Mistral 7b, 2… find
  29. K ANDPAL , N., W ALLACE , E., AND R AFFEL , C. Deduplicating training data mitigates privacy risks in language models. ICML (2022). find our card
  30. K UDUGUNTA , S., C ASWELL , I., Z HANG , B., G AR - CIA , X., C HOQUETTE -C HOO , C. A., L EE , K., X IN , D., K USUPATI , A., S TELLA , R., B APNA , A., ET AL . Madlad-400: A multilingual and document-level large audited dataset. arXiv preprint arXiv:2309.04662 (2023). find our card
  31. L EE , K., C OOPER , A. F., AND G RIMMELMANN , J. Talkin’ ’Bout AI Generation: Copyright and the Generative-AI Supply Chain, 2023. find
  32. L EE , K., C OOPER , A. F., G RIMMELMANN , J., AND I PPOLITO , D. AI and Law: The Next Generation, 2023. find
  33. L EE , K., I PPOLITO , D., N YSTROM , A., Z HANG , C., E CK , D., C ALLISON -B URCH , C., AND C ARLINI , N. Deduplicating training data makes language models better. In ACL (2022). find
  34. M UENNIGHOFF , N., R USH , A. M., B ARAK , B., S CAO , T. L., P IKTUS , A., T AZI , N., P YYSALO , S., W OLF , T., AND R AFFEL , C. Scaling data-constrained language models. arXiv preprint arXiv:2305.16264 (2023). find
  35. O PEN AI. ChatGPT: Optimizing Language Models for Dialogue, 2022. find
  36. O PEN AI. Custom instructions for ChatGPT, 2023. find
  37. O PEN AI. GPT-4 System Card. Tech. rep., Mar. 2023. find
  38. O PEN AI. GPT-4 technical report. arXiv:2303.08774 (2023). arXiv preprint find our card
  39. O UYANG , L., W U , J., J IANG , X., A LMEIDA , D., W AINWRIGHT , C., M ISHKIN , P., Z HANG , C., A GAR - WAL , S., S LAMA , K., R AY , A., ET AL . Training language models to follow instructions with human feedback. NeurIPS (2022). find
  40. P ENEDO , G., M ALARTIC , Q., H ESSLOW , D., C OJO - CARU , R., C APPELLI , A., A LOBEIDLI , H., P ANNIER , B., A LMAZROUEI , E., AND L AUNAY , J. The Refined-Web Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only, 2023. find
  41. P ROJECT Z ERO . Vulnerability disclosure policy. https://googleprojectzero.blogspot.com/p/vulnerability-disclosure-policy.html, 2021. ↗
  42. R ADFORD , A., W U , J., C HILD , R., L UAN , D., A MODEI , D., AND S UTSKEVER , I. Language Models are Unsupervised Multitask Learners. Tech. rep., OpenAI, 2019. find
  43. R AFFEL , C., S HAZEER , N., R OBERTS , A., L EE , K., N ARANG , S., M ATENA , M., Z HOU , Y., L I , W., AND L IU , P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR (2020). find
  44. S ANH , V., W EBSON , A., R AFFEL , C., B ACH , S. H., S UTAWIKA , L., A LYAFEAI , Z., C HAFFIN , A., S TIEGLER , A., S CAO , T. L., R AJA , A., ET AL . Multitask prompted training enables zero-shot task generalization. In ICLR (2021). find our card
  45. S HOKRI , R., S TRONATI , M., S ONG , C., AND S HMATIKOV , V. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy (2017). find our card
  46. S OLDAINI , L. AI2 Dolma: 3 trillion token open corpus for language model pretraining, 2023. find
  47. S OMEPALLI , G., S INGLA , V., G OLDBLUM , M., G EIP - ING , J., AND G OLDSTEIN , T. Diffusion art or digital forgery? Investigating data replication in diffusion models. In CVPR (2023). find
  48. S OUTHWOOD , T. R. E., AND H ENDERSON , P. A. Ecological methods. John Wiley & Sons, 2009. find
  49. T OUVRON , H., L AVRIL , T., I ZACARD , G., M AR - TINET , X., L ACHAUX , M.-A., L ACROIX , T., R OZ - IÈRE , B., G OYAL , N., H AMBRO , E., A ZHAR , F., R O - DRIGUEZ , A., J OULIN , A., G RAVE , E., AND L AMPLE , G. LLaMA: Open and Efficient Foundation Language Models, 2023. find
  50. T OUVRON , H., M ARTIN , L., S TONE , K., A LBERT , P., A LMAHAIRI , A., B ABAEI , Y., B ASHLYKOV , N., B A - TRA , S., B HARGAVA , P., B HOSALE , S., ET AL . LLaMA 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023). find
  51. Y EOM , S., G IACOMELLI , I., F REDRIKSON , M., AND J HA , S. Privacy risk in machine learning: Analyzing the connection to overfitting. In IEEE CSF (2018). find
  52. Z ELTERMAN , D. Smooth nonparametric estimation of the quantile function. Journal of statistical planning and inference 26, 3 (1990), 339–352. find
  53. Z HANG , S., R OLLER , S., G OYAL , N., A RTETXE , M., C HEN , M., C HEN , S., D EWAN , C., D IAB , M., L I , X., L IN , X. V., M IHAYLOV , T., O TT , M., S HLEIFER , S., S HUSTER , K., S IMIG , D., K OURA , P. S., S RIDHAR , A., W ANG , T., AND Z ETTLEMOYER , L. Opt: Open pre-trained transformer la… find
  54. Z IEGLER , A. Github Copilot research recitation, 2021. find
  55. Z OU , A., W ANG , Z., K OLTER , J. Z., AND F REDRIK - SON , M. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 (2023). find our card