Elamur Research⌕
☰
home / papers / 2024 / How to Train Data-Efficient LLMs

How to Train Data-Efficient LLMs

Paper · 2024 · Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang et al.

Summary

Pre-training large language models consumes enormous amounts of data and compute, and scaling laws mean that simply adding more of either yields shrinking gains. The authors ask whether cheap selection heuristics that maximize coverage are sufficient, or whether costly per-example quality judgments are worth paying for. They introduce two samplers that sit at opposite ends of this spectrum. ASK-LLM prompts an instruction-tuned proxy model (Flan-T5 variants) to judge whether each training example is useful. DENSITY estimates the data distribution in embedding space and draws a diverse subset that covers latent topics. The authors compare 19 sampling strategies by pre-training T5-Small and T5-Large and evaluating the resulting models on many downstream tasks. Coverage-oriented DENSITY roughly matches training on the full dataset. Models trained on ASK-LLM-filtered data regularly surpass full-data training and reach target quality considerably sooner. The work suggests that LLM-based quality rating is a practical way to improve the trade-off between data quantity and model quality. It also has implications for training cost and model self-improvement.

Key points

  • The benchmark covers 19 samplers used to pre-train T5-Small (60M) and T5-Large (800M) on 524B tokens. Evaluation spans 111 downstream tasks, for a total of about 170 pre-training runs and 2,500 fine-tuning runs.
  • ASK-LLM keeps beating full-data training even after discarding up to 90% of the examples. In one T5-Large run using Flan-T5-XL as the scorer and 60% of the data, training was up to 70% faster.
  • Even the smallest ASK-LLM proxy (60M parameters) outperforms most baseline curation methods, including perplexity filtering, SemDeDup and SSL prototypes.
  • Kendall's Tau analysis shows that ASK-LLM scores and perplexity scores are almost uncorrelated, meaning the two methods select very different data. DENSITY preserves in-distribution test perplexity better than any other sampler.
  • ASK-LLM requires one LLM inference call per example. The authors argue this cost is amortized across repeated pre-training runs and can be reduced by pre-filtering with cheaper scorers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H. Chi, James Caverlee, Julian McAuley and Derek Zhiyuan Cheng. 2024. How to Train Data-Efficient LLMs. arXiv:2402.09668.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.09668 · arXiv 2402.09668

@misc{sachdeva2024how,
  title = {{How to Train Data-Efficient LLMs}},
  author = {Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H. Chi, James Caverlee, Julian McAuley and Derek Zhiyuan Cheng},
  year = {2024},
  doi = {10.48550/arxiv.2402.09668},
  eprint = {2402.09668},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.09668}
}

References

86 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abbas, A., Tirumala, K., Simig, D., Ganguli, S., and Morcos, A. S. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540, 2023.
  2. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  3. Agarwal, R., Vieillard, N., Stanczyk, P., Ramos, S., Geist, M., and Bachem, O. Gkd: Generalized knowledge distillation for auto-regressive sequence models. arXiv preprint arXiv:2306.13649, 2023.
  4. Coleman, B. and Shrivastava, A. Sub-linear race sketches for approximate kernel density estimation on streaming data. In Proceedings of The Web Conference 2020, pp. 1739–1749, 2020. find
  5. Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. G. Self-consuming generative models go mad. arXiv preprint arXiv:2307.01850, 2023. — [arXiv:2307.01850](https://arxiv.org/abs/2307.01850) find
  6. Coleman, B., Geordie, B., Chou, L., Elworth, R. L., Trean-gen, T., and Shrivastava, A. One-pass diversified sampling with application to terabyte-scale genomic sequence streams. In International Conference on Machine Learning, pp. 4202–4218. PMLR, 2022. find
  7. Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., and et al., Z. C. Palm 2 technical report, 2023.
  8. Ayed, F. and Hayou, S. Data pruning and neural scaling laws: fundamental limitations of score-based algorithms. arXiv preprint arXiv:2302.06960, 2023a. — [arXiv:2302.06960](https://arxiv.org/abs/2302.06960) find
  9. Coleman, C., Yeh, C., Mussmann, S., Mirzasoleiman, B., Bailis, P., Liang, P., Leskovec, J., and Zaharia, M. Selection via proxy: Efficient data selection for deep learning. In ICLR, 2020. find
  10. Ayed, F. and Hayou, S. Data pruning and neural scaling laws: fundamental limitations of score-based algorithms. Transactions on Machine Learning Research, 2023b. ISSN 2835-8856. URL https://openreview.net/forum?id=iRTL4pDavo. ↗
  11. Datar, M., Immorlica, N., Indyk, P., and Mirrokni, V. S. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational geometry, pp. 253–262, 2004. find
  12. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022. find
  13. Borsos, Z., Mutny, M., and Krause, A. Coresets via bilevel optimization for continual learning and streaming. Advances in Neural Information Processing Systems, 33: 14879–14890, 2020. find
  14. Devroye, L. The equivalence of weak, strong and complete convergence in l1 for kernel density estimates. The Annals of Statistics, pp. 896–904, 1983. find
  15. Briesch, M., Sobania, D., and Rothlauf, F. Large language models suffer from their own output: An analysis of the self-consuming training loop. arXiv preprint arXiv:2311.16822, 2023. — [arXiv:2311.16822](https://arxiv.org/abs/2311.16822) find
  16. Engstrom, L., Feldmann, A., and Madry, A. Dsdm: Model-aware dataset selection with datamodels, 2024.
  17. Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., and Robinson, T. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013.
  18. Feldman, D., Schmidt, M., and Sohler, C. Turning big data into tiny data: Constant-size coresets for k-means, pca, and projective clustering. SIAM Journal on Computing, 49(3):601–657, 2020. find
  19. Feldman, V. and Zhang, C. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881–2891, 2020.
  20. Indyk, P., Mahabadi, S., Mahdian, M., and Mirrokni, V. S. Composable core-sets for diversity and coverage maximization. In Proceedings of the 33rd ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pp. 100–108, 2014. find
  21. Gemini, T., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  22. Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C. C. T., Chen, W., Del Giorno, A., Eldan, R., Gopi, S., et al. Phi-2: The surprising power of small language models, 2023. find
  23. Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9: 346–361, 2021. find
  24. Jiang, A. H., Wong, D. L.-K., Zhou, G., Andersen, D. G., Dean, J., Ganger, G. R., Joshi, G., Kaminksy, M., Kozuch, M., Lipton, Z. C., et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv:1910.00762, 2019. — [arXiv:1910.00762](https://arxiv.org/abs/1910.00762) find
  25. Guo, C., Zhao, B., and Bai, Y. Deepcore: A comprehensive library for coreset selection in deep learning. In International Conference on Database and Expert Systems Applications, pp. 181–195. Springer, 2022a. find
  26. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  27. Guo, C., Zhao, B., and Bai, Y. Deepcore: A comprehensive library for coreset selection in deep learning. In International Conference on Database and Expert Systems Applications, pp. 181–195. Springer, 2022b. find
  28. Karnin, Z. and Liberty, E. Discrepancy, coresets, and sketches in machine learning. In Conference on Learning Theory, pp. 1975–1993. find
  29. Katharopoulos, A. and Fleuret, F. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525– 2534. PMLR, 2018. find
  30. Guo, M., Dai, Z., Vrandečić, D., and Al-Rfou, R. Wiki-40b: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 2440–2452, 2020. find
  31. Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., and Hajishirzi, H. Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700, 2020.
  32. Guu, K., Webson, A., Pavlick, E., Dixon, L., Tenney, I., and Bolukbasi, T. Simfluence: Modeling the influence of individual training examples by simulating training runs. arXiv preprint arXiv:2303.08114, 2023.
  33. Hastings, W. K. Monte carlo sampling methods using markov chains and their applications. 1970. find
  34. Kirsch, A. and Mitzenmacher, M. Distance-sensitive bloom filters. In 2006 Proceedings of the Eighth Workshop on Algorithm Engineering and Experiments (ALENEX), pp. 41–50. find
  35. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  36. Lee, A., Miranda, B., and Koyejo, S. Beyond scale: the diversity coefficient as a data quality metric demonstrates llms are pre-trained on formally diverse data. arXiv preprint arXiv:2306.13840, 2023. — [arXiv:2306.13840](https://arxiv.org/abs/2306.13840) find
  37. Hermann, K. M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
  38. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424– 8445, 2022. find
  39. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35: 30016–30030, 2022. find
  40. Liu, Z., Xu, Z., Coleman, B., and Shrivastava, A. One-pass distribution sketch for measuring data heterogeneity in federated learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. find
  41. Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688, 2023a.
  42. Patel, A., Bhattamishra, S., and Goyal, N. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191, 2021.
  43. Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34: 20596–20607, 2021. find
  44. Longpre, S., Yauney, G., Reif, E., Lee, K., Roberts, A., Zoph, B., Zhou, D., Wei, J., Robinson, K., Mimno, D., et al. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint arXiv:2305.13169, 2023b.
  45. Phillips, J. M. Coresets and sketches. In Handbook of discrete and computational geometry, pp. 1269–1288. Chapman and Hall/CRC, 2017. find
  46. Luccioni, A. S., Viguier, S., and Ligozat, A.-L. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24(253): 1–15, 2023. find
  47. Phuong, M. and Hutter, M. Formal algorithms for transformers. arXiv preprint arXiv:2207.09238, 2022. — [arXiv:2207.09238](https://arxiv.org/abs/2207.09238) find
  48. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  49. Mai, T., Musco, C., and Rao, A. Coresets for classification– simplified and strengthened. Advances in Neural Information Processing Systems, 34:11643–11654, 2021. find
  50. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  51. Maini, P., Seto, S., Bai, H., Grangier, D., Zhang, Y., and Jaitly, N. Rephrasing the web: A recipe for compute and data-efficient language modeling, 2024.
  52. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.
  53. Marion, M., Üstün, A., Pozzobon, L., Wang, A., Fadaee, M., and Hooker, S. When less is more: Investigating data pruning for pretraining llms at scale. arXiv preprint arXiv:2309.04564, 2023. find
  54. Rosenbaum, P. R. and Rubin, D. B. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41–55, 1983. find
  55. Meding, K., Buschoff, L. M. S., Geirhos, R., and Wichmann, F. A. Trivial or impossible–dichotomous data difficulty masks model differences (on imagenet and beyond). arXiv preprint arXiv:2110.05922, 2021. — [arXiv:2110.05922](https://arxiv.org/abs/2110.05922) find
  56. Rosenblatt, M. Remarks on some nonparametric estimates of a density function. The annals of mathematical statistics, pp. 832–837, 1956. find
  57. Miao, S.-Y., Liang, C.-C., and Su, K.-Y. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprint arXiv:2106.15772, 2021. find
  58. Sachdeva, N. and McAuley, J. Data distillation: A survey. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. Survey Certification.
  59. Mindermann, S., Brauner, J. M., Razzak, M. T., Sharma, M., Kirsch, A., Xu, W., Höltgen, B., Gomez, A. N., Morisot, A., Farquhar, S., et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pp. 15630– 15649. PMLR,… find
  60. Sachdeva, N., Wu, C.-J., and McAuley, J. Svp-cf: Selection via proxy for collaborative filtering data. arXiv preprint arXiv:2107.04984, 2021. — [arXiv:2107.04984](https://arxiv.org/abs/2107.04984) find
  61. Sachdeva, N., He, Z., Kang, W.-C., Ni, J., Cheng, D. Z., and McAuley, J. Farzi data: Autoregressive data distillation. arXiv preprint arXiv:2310.09983, 2023. — [arXiv:2310.09983](https://arxiv.org/abs/2310.09983) find
  62. Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., and Raffel, C. Scaling data-constrained language models. arXiv preprint arXiv:2305.16264, 2023. find
  63. Schubert, E., Zimek, A., and Kriegel, H.-P. Generalized outlier detection with flexible kernel density estimates. In Proceedings of the 2014 SIAM International Conference on Data Mining, pp. 542–550. find
  64. Ni, J., Ábrego, G. H., Constant, N., Ma, J., Hall, K. B., Cer, D., and Yang, Y. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108.08877, 2021.
  65. Shen, S., Hou, L., Zhou, Y., Du, N., Longpre, S., Wei, J., Chung, H. W., Zoph, B., Fedus, W., Chen, X., et al. Mixture-of-experts meets instruction tuning: A winning combination for large language models. arXiv preprint arXiv:2305.14705, 2023.
  66. OpenAI. Gpt-4 technical report, 2023.
  67. Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., and Anderson, R. The curse of recursion: Training on generated data makes models forget.(5 2023). URl: https://arxiv.org/abs/2305.17493, 2023. understanding systems. Advances in neural information processing systems, 32, 2019. ↗
  68. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824–24837, 2022.
  69. Siminelakis, P., Rong, K., Bailis, P., Charikar, M., and Levis, P. Rehashing kernel evaluation in high dimensions. In International Conference on Machine Learning, pp. 5789–5798. PMLR, 2019. find
  70. Weng, L. Large transformer model inference optimization. Lil’Log, Jan 2023. URL https://lilianweng.github.io/posts/ 2023-01-10-inference-optimization/. ↗
  71. Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523–19536, 2022.
  72. Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019.
  73. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
  74. Wied, D. and Weißbach, R. Consistency of the kernel density estimator: a survey. Statistical Papers, 53(1):1–21, 2012. find
  75. Strubell, E., Ganesh, A., and McCallum, A. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.
  76. Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q. V., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pre-training. arXiv preprint arXiv:2305.10429, 2023a. find
  77. Tirumala, K., Simig, D., Aghajanyan, A., and Morcos, A. S. D4: Improving llm pretraining via document de-duplication and diversification. arXiv preprint arXiv:2308.12284, 2023.
  78. Xie, S. M., Santurkar, S., Ma, T., and Liang, P. Data selection for language models via importance resampling. arXiv preprint arXiv:2302.03169, 2023b. find
  79. Toneva, M., Sordoni, A., Combes, R., Trischler, A., Bengio, Y., and Gordon, G. An empirical study of example forgetting during deep neural network learning. In ICLR, 2019. find
  80. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
  81. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
  82. Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023. find
  83. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  84. Tukan, M., Baykal, C., Feldman, D., and Rus, D. On coresets for support vector machines. Theoretical Computer Science, 890:171–191, 2021. find
  85. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.
  86. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Superglue: A stickier benchmark for general-purpose language find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sachdeva-2024-train-data-efficient-llms/