Elamur Research⌕
☰
home / papers / 2022 / Will we run out of data? Limits of LLM scaling based on human-generated data

Will we run out of data? Limits of LLM scaling based on human-generated data

Paper · 2022 · Pablo Villalobos, Anson Ho, Jaime Sevilla et al.

Summary

Scaling laws push LLM developers toward ever-larger training sets, which raises the question of whether public human-written text will eventually run out. The authors, from Epoch, address this with a quantitative model built from two projections. The first extrapolates dataset sizes from recent notable LLMs. The second estimates the stock of public human text. To get that stock, they use Google's index as a stand-in for the size of the indexed web. They then multiply by the average plain text per page, measured on Common Crawl, convert bytes to tokens, and apply an annual growth rate. They also adjust for text quality and for repeating data over several epochs. Uncertainty is modelled with log-normal distributions and Monte Carlo simulation. The two curves cross somewhere between 2026 and 2032, most likely in 2028. If frontier models are overtrained, this happens a year or two earlier. The authors do not expect progress to stop there. They point to synthetic data, transfer from domains with plenty of data, and gains in data efficiency as ways forward. The paper treats data, not just compute, as a real limit on scaling.

Key points

  • The effective stock of indexed-web text is about 4e14 tokens. Training on all of it would need about 5e28 FLOP for a model that is not overtrained. With 5x overtraining, the stock is used up sooner.
  • Raw stock estimates in tokens: Common Crawl about 130T, indexed web about 510T (95% CI 130T–2100T), whole web about 3100T. Images and video add roughly 300T and 1350T text-token equivalents.
  • Model inputs: Google's index holds about 250B pages (CI 100B–1200B). A page has about 7000 bytes of plain text on average, up from 6100 in 2013 to 8200 in 2021. One token is about 4 bytes under the cl100k_base tokenizer. Web growth is assumed to be 0–10% per year.
  • Data-selection methods such as pruning, domain mixing and curriculum learning are judged to give only small gains. One cited result found pruning helped less than tripling the dataset.
  • The code is available at epochai.org/code/data-stock. The impact statement raises privacy risks in training on social-media and messaging data, and argues that data creators should be paid.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States76 works · 32,207 citations🎓University of TübingenTübingen · Germany78 works · 41,280 citations🎓University of AberdeenAberdeen · United Kingdom23 works · 1,583 citations🤝Centre for the Governance of AIOxford · United Kingdom2 works · 302 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim and Marius Hobbhahn. 2022. Will we run out of data? Limits of LLM scaling based on human-generated data. arXiv:2211.04325.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2211.04325 · arXiv 2211.04325

@misc{villalobos2022will,
  title = {{Will we run out of data? Limits of LLM scaling based on human-generated data}},
  author = {Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim and Marius Hobbhahn},
  year = {2022},
  doi = {10.48550/arxiv.2211.04325},
  eprint = {2211.04325},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2211.04325}
}

References

118 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abnar, S., Dehghani, M., Neyshabur, B., and Sedghi, H. Exploring the limits of large scale pre-training, 2021. URL https://arxiv.org/abs/2110.02095. — [arXiv:2110.02095](https://arxiv.org/abs/2110.02095) ↗
  2. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. et al. GPT-4 Technical Report. Technical report, OpenAI, 2023. URL https://arxiv.org/ abs/2303.08774. ↗
  3. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023.
  4. Aghajanyan, A., Yu, L., Conneau, A., Hsu, W.-N., Hambardzumyan, K., Zhang, S., Roller, S., Goyal, N., Levy, O., and Zettlemoyer, L. Scaling laws for generative mixed-modal language models, 2023. URL https://arxiv.org/abs/2301.03728. ↗
  5. Campos, D. Curriculum learning for language modeling. CoRR, abs/2108.02170, 2021. URL https://arxiv.org/abs/2108.02170. — [arXiv:2108.02170](https://arxiv.org/abs/2108.02170) ↗
  6. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., and Gehrmann, S. et al. PaLM: Scaling Language Modeling with Pathways, 2022. URL https://arxiv.org/ abs/2204.02311. ↗
  7. Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. G. Self-consuming generative models go mad, 2023. find
  8. Cisco. Cisco Visual Networking Index: Forecast and Methodology, 2016–2021, 2017. find
  9. Alpert, J. and Hajaj, N. We knew the web was big... https://googleblog.blogspot.com/2008/07/we -knew-web-was-big.html, 2008. [Accessed 22-04-2024]. ↗
  10. Coffman, K. and Odlyzko, A. The size and growth rate of the internet, 1998. URL https://firstmonday. find
  11. Epoch. Key trends and figures in machine learning, 2023. URL https://epochai.org/trends. Accessed: 2024-01-27. org/ojs/index.php/fm/article/downloa d/620/541?inline=1. ↗
  12. Common Crawl. Common crawl, 2024. URL https://commoncrawl.org/. ↗
  13. Cottier, B. Trends in the dollar training cost of machine learning systems, 2023. URL https://epochai.org/blog/trends-in-the-dollar-train ing-cost-of-machine-learning-systems. Accessed: 2024-02-01. ↗
  14. Erdil, E. and Besiroglu, T. Algorithmic progress in computer vision, 2023. URL https://arxiv.org/abs/22 12.05153. ↗
  15. Fan, L., Chen, K., Krishnan, D., Katabi, D., Isola, P., and Tian, Y. Scaling laws of synthetic images for model training ... for now, 2023. find
  16. Delétang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., Hutter, M., and Veness, J. Language modeling is compression, 2023. URL https://arxiv.org/abs/2309.10668. ↗
  17. Fetterman, A. J., Kitanidis, E., Albrecht, J., Polizzi, Z., Fo-gelman, B., Knutins, M., Wróblewski, B., Simon, J. B., and Qiu, K. Tune as you scale: Hyperparameter optimization for compute efficient training, 2023. URL https://arxiv.org/abs/2306.08055. — [arXiv:2306.08055](https://arxiv.org/abs/2306.08055) ↗
  18. Dey, N., Gosal, G., Zhiming, Chen, Khachane, H., Marshall, W., Pathria, R., Tom, M., and Hestness, J. Cerebrasgpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023. URL https://arxiv.org/abs/2304.03208. — [arXiv:2304.03208](https://arxiv.org/abs/2304.03208) ↗
  19. Friel, N., McKeone, J. P., Oates, C. J., and Pettitt, A. N. Investigation of the widely applicable bayesian information criterion, 2016. URL https://arxiv.org/abs/ 1501.05447. ↗
  20. Gao, L. An empirical exploration in quality filtering of text data, 2021. URL https://arxiv.org/abs/2109 .00698. ↗
  21. Dohmatob, E., Feng, Y., Yang, P., Charton, F., and Kempe, J. A tale of tails: Model collapse as a change of scaling laws, 2024. find
  22. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling, 2021. URL https://arxiv.org/abs/2101.00027. ↗
  23. Domo. Data Never Sleeps 10.0. https://www.do mo.com/data-never-sleeps, 2022. [Accessed 12-04-2024]. ↗
  24. Dorner, F. E. Measuring progress in deep reinforcement learning sample efficiency, 2021. URL https://ar xiv.org/abs/2102.04881. — [arXiv:2102.04881](https://arxiv.org/abs/2102.04881) find
  25. GDELT. Visualizing Twitter’s Evolution 2012-2020 And How Tweeting Is Changing In The COVID-19 Era. https://blog.gdeltproject.org/visualiz ing-twitters-evolution-2012-2020-and -how-tweeting-is-changing-in-the-cov id-19-era/, 2020. [Accessed 12-04-2024]. ↗
  26. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/2010.11929. ↗
  27. Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., Hughes, J., Korbak, T., Agrawal, R., Pai, D., Gromov, A., Roberts, D. A., Yang, D., Donoho, D. L., and Koyejo, S. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data, 2024. find
  28. Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M., Zhou, Z., Wang, T., Wang, Y. E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, K., Duke, T., Dixon, L., Zhang, K., Le, Q. V., Wu, Y., Chen, Z., and Cui… — GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
  29. Griffin, A. Chatgpt creators openai are generating 100 billion words per day, ceo says, 2024. URL https://www.independent.co.uk/tech/chatgpt-o penai-words-sam-altman-b2494900.html. ↗
  30. Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Giorno, A. D., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y. Textbooks are all you need, 2023. find
  31. Epoch. Parameter, compute and data trends in machine learning, 2022. URL https://epochai.org/da ta/epochdb/visualization. Accessed: 2024-01-29. ↗
  32. Haluptzok, P., Bowers, M., and Kalai, A. T. Language models can teach themselves to program better, 2023. URL https://arxiv.org/abs/2207.14502. ↗
  33. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.0 8361. ↗
  34. Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and Mc-Candlish, S. Scaling laws for autoregressive generative modeling, 2020. URL…
  35. Kemp, S. Digital 2023: Global Overview Report — DataReportal. https://datareportal.com/reports/digital-2023-global-overview-report, 2023a. [Accessed 16-04-2024]. ↗
  36. Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer, 2021. URL https://ar xiv.org/abs/2102.01293.
  37. Kemp, S. Facebook Messenger Users, Stats, Data, Trends, and More — DataReportal. https://datareport al.com/essential-facebook-messenger-s tats, 2023b. [Accessed 15-04-2024]. find
  38. Ho, A., Erdil, E., and Besiroglu, T. Limits to the energy efficiency of cmos microprocessors. In 2023 IEEE International Conference on Rebooting Computing (ICRC), pp. 1–10. find
  39. Kemp, S. Facebook Users, Stats, Data, Trends, and More — DataReportal. https://datareportal.com/e ssential-facebook-stats, 2023c. [Accessed 12-04-2024]. ↗
  40. Ho, A., Besiroglu, T., Erdil, E., Owen, D., Rahman, R., Guo, Z. C., Atkinson, D., Thompson, N., and Sevilla, J. Algorithmic progress in language models. 2024. URL https://api.semanticscholar.org/Corp usID:268358466. ↗
  41. Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., Grefenstette, E., Bowman, S. R., Rocktäschel, T., and Perez, E. Debating with more persuasive llms leads to more truthful answers, 2024.
  42. Krebs, F., Lubascher, B., Moers, T., Schaap, P., and Spanakis, G. Social emotion mining techniques for facebook posts reaction prediction, 2017. URL https://arxiv.org/abs/1712.03249. — [arXiv:1712.03249](https://arxiv.org/abs/1712.03249) ↗
  43. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G. v. d., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. Tr… — Training Compute-Optimal Large Language Models
  44. Lee, E. 2021 worldwide image capture forecast: 2020 – 2025, 2021. URL https://riseaboveresearch.com/rar-reports/2021-worldwide-image -capture-forecast-2020-2025/. ↗
  45. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating training data makes language models better, 2022. URL https://arxiv.org/abs/2107.06499. ↗
  46. Howell, S. and Burtis, A. T. The continued problem of url decay: an updated analysis of health care management journal citations. Journal of the Medical Library Association : JMLA, 110:463 – 470, 2022. URL https://api.se manticscholar.org/CorpusID:257746935. ↗
  47. Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023.
  48. Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve, 2022. URL https://arxiv.org/abs/2210.11610. ↗
  49. Liu, B., Bubeck, S., Eldan, R., Kulkarni, J., Li, Y., Nguyen, A., Ward, R., and Zhang, Y. Tinygsm: achieving ¿80 Institute, N. H. G. R. Genomic data science (fact sheet), 2024. URL https://www.genome.gov/about -genomics/fact-sheets/Genomic-Data-S cience. ↗
  50. Loan, F. A. and Shah, U. Y. The decay and persistence of web references. Digit. Libr. Perspect., 36:157–166, 2020. URL https://api.semanticscholar.org/ CorpusID:219404788. ↗
  51. Irving, G., Christiano, P., and Amodei, D. Ai safety via debate, 2018.
  52. Loan, F. A., Khan, A. M., Andrabi, S. A. A., Sozia, S. R., and Parray, U. Y. Giving life to dead: role of WayBack Machine in recovery of dead URLs. Data Technologies and Applications, 2023. URL https://api.sema nticscholar.org/CorpusID:259585485. jerryspan. GitHub - jerryspan/FacebookR: Facebook Pos… ↗
  53. Lyman, P. and Varian, H. R. How much information, 2003. URL https://groups.ischool.berkeley.edu/archive/how-much-info-2003/. ↗
  54. Mahoney, M. Large Text Compression Benchmark, 2006/2024. URL http://www.mattmahoney.net/dc/text.html. [Accessed 29-01-2024]. ↗
  55. Nguyen, T., Ilharco, G., Wortsman, M., Oh, S., and Schmidt, L. Quality not quantity: On the interaction between dataset design and robustness of clip, 2022. URL https://arxiv.org/abs/2208.05516. — [arXiv:2208.05516](https://arxiv.org/abs/2208.05516) ↗
  56. Nostalgebraist. chinchilla’s wild implications, 2022. URL https://www.lesswrong.com/posts/6Fpv ch8RR29qLEWNH/chinchilla-s-wild-impli cations. ↗
  57. Marion, M., Üstün, A., Pozzobon, L., Wang, A., Fadaee, M., and Hooker, S. When less is more: Investigating data pruning for pretraining llms at scale, 2023. URL https://arxiv.org/abs/2309.04564. ↗
  58. OBDILCI. Indicators for the Presence of Languages in the Internet. https://www.obdilci.org/projec ts/main/, 2024. [Accessed 16-04-2024]. ↗
  59. Meta. Introducing Meta Llama 3: The most capable openly available LLM to date. https://ai.meta.com/bl og/meta-llama-3/, 2024. [Accessed 24-04-2024]. ↗
  60. Odlyzko, A. The growth rate and the nature of internet traffic. Transactions on Internet Research, pp. 39–42, 2016. find
  61. Michael, J., Mahdi, S., Rein, D., Petty, J., Dirani, J., Padmakumar, V., and Bowman, S. R. Debate helps supervise unreliable experts, 2023. find
  62. OECD. Gross domestic spending on R&D (indicator), 2022. URL https://data.oecd.org/rd/gross-d omestic-spending-on-r-d.htm. ↗
  63. Mitchell, M., Luccioni, A. S., Lambert, N., Gerchick, M., McMillan-Major, A., Ozoani, E., Rajani, N., Thrush, T., Jernite, Y., and Kiela, D. Measuring data, 2023. URL https://arxiv.org/abs/2212.05129. — [arXiv:2212.05129](https://arxiv.org/abs/2212.05129) ↗
  64. OpenAI. Gpt-4v(ision) system card. Technical report, OpenAI, 2023. Accessed: 2024-02-01. find
  65. Mosaic AI. Introducing DBRX: A New State-of-the-Art Open LLM — Databricks — databricks.com. https://www.databricks.com/blog/introduc ing-dbrx-new-state-art-open-llm, 2024. [Accessed 12-04-2024]. ↗
  66. OpenAI and Pilipiszyn, A. Gpt-3 powers the next generation of apps, 2021. URL https://openai.com/blog/ gpt-3-apps/. ↗
  67. OpenAI. tiktoken, 2024. URL https://github.com/openai/tiktoken. ↗
  68. Mosseri, A. and Chudnovsky, S. Say :wave: to Messenger: Introducing New Messaging Features for Instagram. https://about.fb.com/news/2020/09/new-m essaging-features-for-instagram/, 2020. [Accessed 15-04-2024]. ↗
  69. OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N., Tworek, J., Welinder, P., Weng, L., Yuan, Q., Zaremba, W., and Zhang, L. Solving rubik’s cube with a robot hand, 2019. find
  70. Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., and Raffel, C. Scaling data-constrained language models, 2023. URL https://arxiv.org/abs/2305.16264. ↗
  71. Ott, D. E. Reference hygiene and death on the internet– decay, rot, half-life, deterioration, and corruption. JSLS: Journal of the Society of Laparoscopic & Robotic Surgeons, 26(1), 2022. find
  72. Murray H., B. and Moore, A. Sizing the internet. Technical report, Cyveillance, 7 2000. URL http://www.cs.t oronto.edu/˜leehyun/papers/Sizing_the _Internet.pdf. ↗
  73. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L. E., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. J. Training language models to follow instruct… — Training language models to follow instructions with human feedback
  74. National Electrical Manufacturers Association (NEMA). Comments on innovative advanced transformers. RFI Response DE-FOA-0003021, U.S. Department of Energy, Washington, DC, May 2023. URL https://www.ne ma.org/docs/default-source/advocacy-d ocument-library/nema-comments-on-inn ovative-advanced-transfo… ↗
  75. Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., and Launay, J. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023. URL https://arxiv.org/abs/2306 .01116. ↗
  76. Pichai, S. and Hassabis, D. Introducing gemini: our largest and most capable ai model, 2023. URL https://bl og.google/technology/ai/google-gemin i-ai/. find
  77. Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J. Self-critiquing models for assisting human evaluators, 2022.
  78. Scao, T. L., Wang, T., Hesslow, D., Saulnier, L., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., Press, O., Raffel, C., Sanh, V., Shen, S., Sutawika, L., Tae, J., Yong, Z. X., Launay, J., and Beltagy, I. What language model to train if you have one million gpu hours?…
  79. Radicati. Email statistics report, 2020-2024. https://ww w.radicati.com/wp/wp-content/uploads /2020/01/Email_Statistics_Report,_20 20-2024_Executive_Summary.pdf. [Accessed 03-05-2024]. find
  80. Raffel, C., Shazeer, N. M., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2019. URL https://arxiv.org/abs/1910.10683. ↗
  81. Schuhmann, C. and Sun, Q. Strategic game datasets for enhancing ai planning: An invitation for collaborative research. LAION Projects Team Blog, Oct 2023. URL https://laion.ai/blog/strategic-gam e-dataset/. Accessed: [insert date here]. ↗
  82. Real Time Statistics Project. Internet Live Stats. URL https://www.internetlivestats.com/. [Accessed 02-02-2024]. ↗
  83. Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., and Villalobos, P. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN). find
  84. IEEE, 2022. doi: 10.1109/ ijcnn55064.2022.9891914. URL http://dx.doi.org/10.1109/IJCNN55064.2022.9891914. — [doi:10.1109/ijcnn55064.2022.9891914](https://doi.org/10.1109/ijcnn55064.2022.9891914) ↗
  85. Reinsel, D., Gantz, J., and Rydning, J. The digitization of the world from edge to core. Technical report, International Data Corporation, 11 2018. URL https://www.seagate.com/files/www-c ontent/our-story/trends/files/idc-sea gate-dataage-whitepaper.pdf. ↗
  86. Shepard, C. Google’s index size revealed: 400 billion docs (& changing), 2024. URL https://zyppy.com/se o/google-index-size/. ↗
  87. Ritchie, H. and Roser, M. Technology adoption. Our World in Data, 2017. URL https://ourworldindata.org/technology-adoption. ↗
  88. Shilov, A. Nvidia to reportedly triple output of compute gpus in 2024: Up to 2 million h100s, Aug 2023. URL https://www.tomshardware.com/news/nvid ia-to-reportedly-triple-output-of-c ompute-gpus-in-2024-up-to-2-million -h100s. ↗
  89. Ritchie, H. and Roser, M. Number of people using the internet. Our World in Data, 2020. URL https://ou rworldindata.org/grapher/number-of-i nternet-users. find
  90. Roose, K. and Newton, C. Dario amodei on the paradoxes of a.i. safety and netflix’s ’deep fake love’. Podcast Transcript, July 2023. URL https://www.nytimes.com/section/technology. Available at The New York Times. ↗
  91. Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., and Anderson, R. The curse of recursion: Training on generated data makes models forget, 2023. URL https://arxiv.org/abs/2305.17493. ↗
  92. Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., and Hassabis, D. Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017. URL https://arxiv.org/abs/1712.0… ↗
  93. Rosenfeld, A., Sina, S., Sarne, D., Avidov, O., and Kraus, S. WhatsApp usage patterns and prediction of demographic characteristics without access to message content. Demographic Research, 39(22):647–670, 2018. doi: 10.4054/DemRes.2018.39.22. URL https://www.demographic-research.org/volume s/vol39/… — [doi:10.4054/demres.2018.39.22](https://doi.org/10.4054/demres.2018.39.22) ↗
  94. Similarweb. Top Websites. https://www.similarw eb.com/top-websites/, 2024. [Accessed 16-04-2024]. ↗
  95. Sardana, N. and Frankle, J. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws, 2023. URL https://arxiv.org/abs/2401.0 0448. ↗
  96. Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A., Kumar, A., Alemi, A., Rizkowsky, A., Nova, A., Adlam, B., Bohnet, B., Elsayed, G., Sedghi, H., Mordatch, I., Simpson, I., Gur, I., Snoek, J., Pennington, J., Satyanarayan… find
  97. Hron, J., Kenealy, K., Swersky, K., Mahajan, K., Culp, L., Xiao, L., Bileschi, M. L., Constant, N., Novak, R., Liu, R., Warkentin, T., Qian, Y., Bansal, Y., Dyer, E., Neyshabur, B., Sohl-Dickstein, J., and Fiedel, N. Beyond human data: Scaling self-training for problem-solving with language models,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  98. Trinh, T. H., Wu, Y., Le, Q. V., He, H., and Luong, T. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476–482, January 2024. ISSN 1476-4687. doi: 10.1038/s41586-023-06747-5. URL http://dx.doi.org/10.1038/s41586-023 -06747-5. ↗
  99. Singh, M. WhatsApp is now delivering roughly 100 billion messages a day. https://techcrunch.com/2 020/10/29/whatsapp-is-now-deliverin g-roughly-100-billion-messages-a-day, 2020. [Accessed 15-04-2024]. ↗
  100. United Nations. World population prospects 2022, online edition, 2022. URL https://population.un.org/wpp/. ↗
  101. Size, W. W. W. World wide web size, 2024. URL https://www.worldwidewebsize.com/. ↗
  102. Stephens, Z. D., Lee, S., Faghri, F., Campbell, R. H., Zhai, C., Efron, M., Iyer, R. K., Schatz, M. C., Sinha, S., and Robinson, G. E. Big data: Astronomical or genomical? PLoS Biology, 13, 2015. URL https://journals.plos.org/plosbiology/article?id=10. 1371/journal.pbio.1002195. ↗
  103. Taniguchi, M., Ueda, Y., Taniguchi, T., and Ohkuma, T. A large-scale corpus of E-mail conversations with standard and two-level dialogue act annotations. In Scott, D., Bel, N., and Zong, C. (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp. 4969–4980, Barcelon…
  104. Villalobos, P. and Ho, A. Trends in training dataset sizes. https://epochai.org/blog/trends-in-t raining-dataset-sizes, 2022. Accessed: 2022-09-27. ↗
  105. Wei, T., Zhao, L., Zhang, L., Zhu, B., Wang, L., Yang, H., Li, B., Cheng, C., Lü, W., Hu, R., Li, C., Yang, L., Luo, X., Wu, X., Liu, L., Cheng, W., Cheng, P., Zhang, J., Zhang, X., Lin, L., Wang, X., Ma, Y., Dong, C., Sun, Y., Chen, Y., Peng, Y., Liang, X., Yan, S., Fang, H., and Zhou, Y. Skywork:… — Skywork: A More Open Bilingual Foundation Model
  106. White, K. Publications output: U.s. trends and international comparisons, 2019. URL https://ncses.nsf.gov/pubs/nsb20206/. ↗
  107. Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. Galactica: A large language model for science, 2022. URL https://arxiv.org/abs/2211.09085. ↗
  108. Thorgren, E., Mohammadinodooshan, A., and Carlsson, N. Temporal dynamics of user engagement on instagram: A comparative analysis of album, photo, and video interactions. URL — D4: Improving LLM Pretraining via Document De-Duplication and Diversification https://api.semanticscholar.org/CorpusID:268241757. Tirumala, K., Simig, D., Aghajanyan, A., and Morcos, A. S. D4: Improving… ↗
  109. Wikipedia. Wikipedia:Size of Wikipedia. https://en.wikipedia.org/wiki/Wikipedia:Size_o f_Wikipedia. [Accessed 22-04-2024]. ↗
  110. Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q. V., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pre-training, 2023. URL https://arxiv.org/abs/ 2305.10429. ↗
  111. Xue, F., Fu, Y., Zhou, W., Zheng, Z., and You, Y. To repeat or not to repeat: Insights from scaling llm under token-crisis, 2023. URL https://arxiv.org/abs/23 05.13230. ↗
  112. Together.ai. Redpajama-data-v2: An open dataset with 30 trillion tokens for training large language models, 2023. URL https://www.together.ai/blog/red pajama-data-v2. ↗
  113. XVERSE Technology Inc. Xverse-65b: A multilingual large language model, 2024. URL https://gith ub.com/xverse-ai/XVERSE-65B. Apache-2.0 License. find
  114. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., A. Theoretical growth model of the web Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs v: Tuning large… find
  115. We explain in more detail our theoretical model of data accumulation rates developed in Section 2.2.2 and check it on Reddit submission data. The model is explained in Equation 5. Yang, K., Swope, A. M., Gu, A., Chalamala, R., Song, P., Yu, S., Godil, S., Prenger, R., and Anandkumar, A. Leandojo: Th… — LeanDojo: Theorem Proving with Retrieval-Augmented Language Models
  116. A purely exponential model cannot reproduce the decrease in the growth rate of Reddit submissions over time, while a purely sigmoidal model plateaus at zero growth. The exponential times sigmoid model is able to better capture the deceleration in submission size growth (see Figure 7). Ye, W., Liu, S… — [arXiv:2111.00210](https://arxiv.org/abs/2111.00210) find
  117. In our actual model, the slowdown in population growth (which becomes subexponential) leads to additional deceleration, but the time period covered by the Reddit submission dataset seems too short for slowing population growth to be noticeable in the data. Yole Développement. Status of the CMOS Ima… find
  118. Projection Based on the Number of Internet Users Z y find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/villalobos-2022-will-run-out-data-limits/