Elamur Research⌕
☰
home / papers / 2023 / The MiniPile Challenge for Data-Efficient Language Models

The MiniPile Challenge for Data-Efficient Language Models

Paper · 2023 · Jean Kaddour

Summary

New ideas in Transformer design, optimizers and training recipes are usually tested on small, single-domain corpora such as enwik8 or WikiText103. Diverse collections like the 825GB The Pile need more compute than most academic labs have. This paper proposes the MiniPile Challenge: pre-train a language model on a varied corpus limited to one million documents. The accompanying resource, MiniPile, is a 6GB slice of the deduplicated Pile. To build it, every document was embedded with E5-Large and the embeddings were grouped into 220 clusters with k-means. The author then manually reviewed samples from each cluster and removed those judged harmful or low-value, and documents were drawn from the remaining clusters. BERT-Base and T5v1.1-Base models were pre-trained on MiniPile with a single RTX 3090 GPU. On GLUE and SNI they scored only about two to three points below public checkpoints trained on 2.6x and 745x more data. MiniPile therefore offers a cheap but diverse testbed for data-efficient pre-training research.

Key points

  • Resource: MiniPile is released on the Hugging Face hub as jeankaddour/minipile.
    • It has 1M training, 500 validation and 10k test documents.
    • It takes about 6GB uncompressed and 3GB compressed, with a median document length of 294.
  • Curation pipeline:
    • Documents were embedded with E5-Large.
    • Embeddings were clustered with batched k-means using cosine distance (k = 220, i.e. 10 per Pile subset).
    • For each cluster, the author inspected the 5 documents nearest to the centroid and the 5 farthest from it.
    • 38 clusters were dropped: near-duplicates, pornography, navigation bars, product specifications and long lists of named entities.
  • Sampling choice: In preliminary BERT runs, keeping only the documents closest to each centroid gave worse GLUE results than random sampling within clusters.
  • BERT results: BERT-Base, trained with the Cramming recipe, reached a GLUE-dev average of 79.0, compared with 80.9 for the baseline. The largest drop was on CoLA (45.0 vs 56.5).
  • T5 results: T5v1.1-Base reached 38.47 on the SNI test set, compared with 41.0 for the baseline.
  • Compute cost: All runs used a single RTX 3090.
    • Pre-training took 54h for BERT and 21h for T5.
    • Fine-tuning took 3h for BERT and 2h for T5.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University College LondonLondon · United Kingdom88 works · 19,432 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Jean Kaddour. 2023. The MiniPile Challenge for Data-Efficient Language Models. arXiv:2304.08442.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.08442 · arXiv 2304.08442

@misc{kaddour2023minipile,
  title = {{The MiniPile Challenge for Data-Efficient Language Models}},
  author = {Jean Kaddour},
  year = {2023},
  doi = {10.48550/arxiv.2304.08442},
  eprint = {2304.08442},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.08442}
}

References

53 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [1] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540, 2023.
  2. [2] Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differentially private bert. arXiv preprint arXiv:2108.01624, 2021. find
  3. [3] Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning, 2021. find
  4. Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR, 2022. find
  5. [11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  6. [12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  7. [13] Nolan Dey, Gurpreet Gosal, Zhiming, Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, and Joel Hestness. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023. find
  8. [14] Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758, 2021. find
  9. [15] N Elhage, N Nanda, C Olsson, T Henighan, N Joseph, B Mann, A Askell, Y Bai, A Chen, T Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. find
  10. [16] Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference on Learning Representations, 2023. find
  11. [17] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. find
  12. [18] Jonas Geiping and Tom Goldstein. Cramming: Training a language model on a single gpu in one day. arXiv preprint arXiv:2212.14034, 2022. find
  13. [19] Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus, 2019. ↗
  14. [20] Peter Henderson, Mark Simon Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel E. Ho. Pile of law: Learning responsible data filtering from the law and a 256GB open-source legal dataset. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and… find
  15. [21] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205.10487, 2022.
  16. [22] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
  17. [23] Peter Izsak, Moshe Berchansky, and Omer Levy. How to train bert with an academic budget. arXiv preprint arXiv:2104.07705, 2021. find
  18. [24] Jean Kaddour. Stop wasting my time! saving days of imagenet and bert training with latest weight averaging. arXiv preprint arXiv:2209.14981, 2022. find
  19. [25] Jean Kaddour, Steindór Sæmundsson, et al. Probabilistic active meta-learning. Advances in Neural Information Processing Systems, 33:20813–20822, 2020. find
  20. [26] Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt Kusner. When do flat minima optimizers work? In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/ forum?id=vDeh2yxTvuh. ↗
  21. [27] Jean Kaddour, Aengus Lynch, Qi Liu, Matt J. Kusner, and Ricardo Silva. Causal machine learning: A survey and open problems. arXiv preprint arXiv:2206.15475, 2022. URL https://arxiv.org/abs/2206.15475. — [arXiv:2206.15475](https://arxiv.org/abs/2206.15475) ↗
  22. [28] Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. arXiv preprint arXiv:2211.08411, 2022.
  23. [29] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  24. [30] Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Lingu…
  25. [31] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018.
  26. [32] Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian-Ling Mao. Copy is all you need. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id= CROlOA9Nd8C. ↗
  27. [33] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021. find
  28. [34] Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-1: Technical details and evaluation. White Paper. AI21 Labs, 1, 2021. find
  29. [35] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. find
  30. [47] Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, 2(11):100336, 2021. find
  31. [36] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representa- [48] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, tions, 2017. URL https://openreview. John Aslanides, Sarah… find
  32. [39] Matt Mahoney. mark, 2011. find
  33. Large text compression bench-[40] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
  34. [41] Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. Nonparametric masked language modeling. arXiv preprint arXiv:2212.01349, 2022. find
  35. [42] Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. MTEB Leaderboard. https://huggingface.co/spaces/ mteb/leaderboard, 2023. Accessed: 17th April 2023. ↗
  36. [43] Koichi Nagatsuka, Clifford Broni-Bediako, and Masayasu Atsumi. Length-based curriculum learning for efficient pre-training of language models. New Generation Computing, pages 1–26, 2022. find
  37. [44] Piotr Nawrot. nanoT5, 3 2023. URL https://github.com/PiotrNawrot/nanoT5. ↗
  38. [45] Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. arXiv preprint arXiv:2208.05516, 2022. — [arXiv:2208.05516](https://arxiv.org/abs/2208.05516) find
  39. [46] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario… — In-context Learning and Induction Heads
  40. [50] Phillip Rust, Jonas F Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. Language modelling with pixels. arXiv preprint arXiv:2207.06991, 2022. — [arXiv:2207.06991](https://arxiv.org/abs/2207.06991) find
  41. [51] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.
  42. [52] Noam Shazeer. Glu variants improve transformer, 2020. find
  43. [53] Leslie N. Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates, 2018. find
  44. [54] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206.14486, 2022.
  45. [55] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.
  46. [56] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022.
  47. [57] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Dosh… find
  48. [58] Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, and Chengxu Zhuang. Call for papers – the babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus, 2023. find
  49. [59] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021.
  50. [60] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling, 2023. URL https://arxiv.org/abs/2302.03169. ↗
  51. [61] Xingcheng Yao, Yanan Zheng, Xiaocong Yang, and Zhilin Yang. Nlp from scratch without large-scale pretraining: A simple and efficient framework. In International Conference on Machine Learning, pages 25438–25451. PMLR, 2022. find
  52. [62] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
  53. [63] Rui-Jie Zhu, Qihang Zhao, and Jason K Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint arXiv:2302.13939, 2023. — [arXiv:2302.13939](https://arxiv.org/abs/2302.13939) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kaddour-2023-minipile-challenge-data-efficient-language/