Elamur Research⌕
☰
home / papers / 2022 / Scaling Laws and Interpretability of Learning from Repeated Data

Scaling Laws and Interpretability of Learning from Repeated Data

Summary

Large language models often see the same data several times. Sometimes this is done deliberately, to upweight high-quality sources, and sometimes it results from incomplete deduplication. The authors from Anthropic investigate why even a small fraction of repeated data harms models so severely. They train a family of transformer models on 100B tokens. Most of the data is unique, while a small subset is repeated many times. Across runs, they vary the model size, the fraction of repeated tokens, and the number of repetitions. The results are analyzed from two angles: through scaling laws and through mechanistic interpretability. The main finding is a pronounced double descent. The harm is greatest in a certain intermediate range of repetition frequency, roughly where the model begins to fully memorize the repeated subset. Copying and induction heads, that is, mechanisms associated with generalization, suffer the most. This leads to the hypothesis that memorization displaces generalization. In practice, this means that high-quality corpora such as Wikipedia or books should be duplicated with caution. The loss on the repeated subset can be monitored as a signal that repetition has already become harmful.

Key points

  • Data setup: 90% of tokens are unique, 10% are repetitions of a small subset. Model sizes range from roughly 1.5M to 805M parameters. Variants with 50% and 90% repeated data are also considered.
  • An 800M-parameter model trained on 0.1% of the data repeated 100 times degrades to the level of a model half its size (roughly 340–400M).
  • Extrapolation suggests that for GPT-3-scale models, noticeable harm may arise from repeating data as few as two times.
  • To test copying, the authors used the opening paragraph of Harry Potter, repeated 11 times. With 3% repeated data, the effective model size on this test dropped by up to a factor of 3. On the prefix matching score, the drop averaged 32%, while on ordinary test loss it was no more than 15%.
  • The effect replicates on Python code and on attention-only models with 1–2 layers. Pretraining on repeated data yields worse results than training from scratch.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Interpretability

Cite

Cite as Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan and Sam McCandlish. 2022. Scaling Laws and Interpretability of Learning from Repeated Data. arXiv:2205.10487.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2205.10487 · arXiv 2205.10487

@misc{hernandez2022scaling,
  title = {{Scaling Laws and Interpretability of Learning from Repeated Data}},
  author = {Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan and Sam McCandlish},
  year = {2022},
  doi = {10.48550/arxiv.2205.10487},
  eprint = {2205.10487},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.10487}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Advani, M. S. and Saxe, A. M. (2017). High-dimensional dynamics of generalization error in neural networks. find
  2. Amodei, D., Hernandez, D., Sastry, G., Clark, J., Brockman, G., and Sutskever, I. (2018). Ai and compute. Heruntergeladen von https://blog.openai. com/aiand-compute. ↗
  3. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. (2021). A general langua… find
  4. Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2018). Reconciling modern machine learning practice and the bias-variance trade-off. find
  5. Bi, B., Li, C., Wu, C., Yan, M., Wang, W., Huang, S., Huang, F., and Si, L. (2020). Palm: Pre-training an autoencoding and autoregressive language model for context-conditioned generation. find
  6. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… our card
  7. Cammarata, N., Carter, S., Goh, G., Olah, C., Petrov, M., Schubert, L., Voss, C., Egan, B., and Lim, S. K. (2020). Thread: Circuits. Distill. https://distill.pub/2020/circuits. ↗
  8. Droppo, J. and Elibol, O. (2021). Scaling laws for acoustic models. find
  9. Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., a… find
  10. Ganguli, D., Hernandez, D., Lovitt, L., DasSarma, N., Henighan, T., Jones, A., Joseph, N., Kernion, J., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Elhage, N., Showk, S. E., Fort, S., Hatfield-Dodds, Z., Johnston, S., Kravec, S., Nanda, N., Ndousse, K., Olsson, C., Amodei, D., A… find
  11. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. (2021). The pile: An 800gb dataset of diverse text for language modeling. find
  12. Geiger, M., Spigler, S., d’Ascoli, S., Sagun, L., Baity-Jesi, M., Biroli, G., and Wyart, M. (2019). Jamming transition as a paradigm to understand the loss landscape of deep neural networks. Physical Review E, 100(1):012115. find
  13. Goh, G., Nick, C., Chelsea, V., Carter, S., Petrov, M., Schubert, L., Radford, A., and Olah, C. (2021). Multimodal neurons in artificial neural networks. Distill. https://distill.pub/2021/multimodal-neurons. ↗
  14. Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and McCandlish, S. (2020). Scaling laws for autoregressive generative modeling. our card
  15. Hernandez, D. and Brown, T. B. (2020). Measuring the algorithmic efficiency of neural networks. CoRR, abs/2005.04305. — [arXiv:2005.04305](https://arxiv.org/abs/2005.04305) find
  16. Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. (2021). Scaling laws for transfer. arXiv preprint arXiv:2102.01293. our card
  17. Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. (2017). Deep learning scaling is predictable, empirically. our card
  18. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G. v. d., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. (2… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/