Elamur Research⌕
☰
home / papers / 2024 / MiniPLM: Knowledge Distillation for Pre-Training Language Models

MiniPLM: Knowledge Distillation for Pre-Training Language Models

Paper · 2024 · Yuxian Gu, Hao Zhou, Fandong Meng et al.

Summary

The paper tackles how to use knowledge distillation (KD) when pre-training small language models, not only when fine-tuning them. Existing options each have a cost. Online KD runs the teacher during training, which adds compute and usually requires a shared tokenizer. Offline KD on teacher-generated text tends to yield easy, repetitive data. The authors propose MiniPLM, which treats KD as reward maximization and applies the teacher's knowledge by reshaping the pre-training corpus itself. Its core step, Difference Sampling, scores each document by how much more likely a large teacher finds it than a tiny reference LM does. Top-scoring documents are kept. This keeps hard, diverse text, reduces easy boilerplate, and drops noise. Teacher inference is done once offline, so the refined corpus can train many students, including ones from other model families. Students of 200M–1.2B parameters, distilled from a 1.8B Qwen teacher, outperformed baselines on downstream tasks and language modeling while using less compute. This matters for building cheap, deployable models and for stretching limited web data.

Key points

  • The teacher has 1.8B parameters (Qwen family). Students of 200M, 500M and 1.2B are trained from scratch and evaluated zero-shot on 9 downstream NLP tasks.
  • The selection score is the log-ratio of teacher to reference-model probability. A theoretical proposition explains why selecting from the real corpus can stand in for sampling from the student.
  • Online KD would need about 30PB to pre-store teacher probabilities for 50B tokens with a 150K vocabulary. MiniPLM avoids this storage cost and any extra training-time cost.
  • Scaling-law extrapolation suggests the gains hold up to about 10T training tokens. The computation-scaling plot marks roughly a 2.2x saving, and pre-training data demand drops by 2.4x.
  • Cross-family distillation works: the Qwen teacher improves Llama3.1 and Mamba students. Code, data and models are available at github.com/thu-coai/MiniPLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🏢TencentShenzhen · China134 works · 19,743 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Yuxian Gu, Hao Zhou, Fandong Meng, Jie Zhou and Minlie Huang. 2024. MiniPLM: Knowledge Distillation for Pre-Training Language Models. arXiv:2410.17215.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.17215 · arXiv 2410.17215

@misc{gu2024miniplm,
  title = {{MiniPLM: Knowledge Distillation for Pre-Training Language Models}},
  author = {Yuxian Gu, Hao Zhou, Fandong Meng, Jie Zhou and Minlie Huang},
  year = {2024},
  doi = {10.48550/arxiv.2410.17215},
  eprint = {2410.17215},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.17215}
}

References

91 references, 41 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In Proceedings of ICLR, 2024. URL https://openreview.net/ forum?id=3zKtaqxLhW. ↗
  2. Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. In Proceedings of ICML, 2024. URL https://openreview.net/forum?id= 5x788rqbcj. ↗
  3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. URL https://arxiv.org/pdf/2309.16609. ↗
  4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. URL https://arxiv.org/abs/ 2212.08073. ↗
  5. Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable LM 2 1.6B technical report. arXiv preprint arXiv:2402.17834, 2024. URL https://arxiv.org/abs/ 2402.17834. ↗
  6. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of AAAI, 2020. URL https://ojs.aaai.org/index.php/AAAI/article/view/6239/6095. ↗
  7. Nicolas Boizard, Kevin El-Haddad, Céline Hudelot, and Pierre Colombo. Towards cross-tokenizer distillation: the universal logit distillation loss for llms. arXiv preprint arXiv:2402.12030, 2024. URL https://arxiv.org/abs/2402.12030. ↗
  8. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. URL https://arxiv.org/abs/2108.07258. ↗
  9. David Brandfonbrener, Hanlin Zhang, Andreas Kirsch, Jonathan Richard Schwarz, and Sham Kakade. CoLoR-Filter: Conditional loss reduction filtering for targeted language model pre-training. arXiv preprint arXiv:2406.10670, 2024. URL https://arxiv.org/abs/2406.10670. — [arXiv:2406.10670](https://arxiv.org/abs/2406.10670) ↗
  10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. Language models are few-shot learners. In Proceedings of NeurIPS, 2020. URL https://papers.nips.cc/paper/ 2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html. ↗
  11. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023. URL https://opena…
  12. Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, et al. Stealing part of a production language model. In Proceedings of ICML, 2024. URL https://arxiv.org/ abs/2403.06634. — [arXiv:2403.06634](https://arxiv.org/abs/2403.06634) ↗
  13. Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. Instruction pre-training: Language models are supervised multitask learners. In Proceedings of EMNLP, 2024. URL https://arxiv.org/abs/2406.14491. ↗
  14. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna… ↗
  15. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. URL https://arxiv.org/abs/2204.02311. ↗
  16. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018. URL https://arxiv.org/abs/1803.05457. ↗
  17. Wojciech M Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszcz, and Max Jaderberg. Distilling policy distillation. In Proceedings of AISTATS, 2019. URL http://proceedings.mlr.press/v89/czarnecki19a/czarnecki19a.pdf. ↗
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, 2019. URL https://aclanthology.org/N19-1423.pdf. ↗
  19. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783. ↗
  20. Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401.12926, 2024. URL https://arxiv.org/abs/ 2401.12926. ↗
  21. Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with v-usable information. In Proceedings of ICML, 2022. URL https://proceedings.mlr.press/v162/ethayarajh22a/ethayarajh22a.pdf. ↗
  22. Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=g97OHbQyk1. ↗
  23. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. URL https://arxiv.org/ abs/2101.00027. ↗
  24. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, et al. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo.org/records/12608602. ↗
  25. Nathan Godey, Roman Castagné, Éric de la Clergerie, and Benoît Sagot. MANTa: Efficient gradient-based tokenization for end-to-end robust language modeling. In Findings of EMNLP, 2022. URL https://aclanthology.org/2022.findings-emnlp.207. ↗
  26. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. OLMo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024. URL https://arxiv.org/abs/2402.00838. ↗
  27. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. URL https://arxiv.org/abs/2312.00752. ↗
  28. Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. PPT: Pre-trained prompt tuning for few-shor learning. In Proceedings of ACL, 2022. URL https://arxiv.org/abs/2109.04332. — [arXiv:2109.04332](https://arxiv.org/abs/2109.04332) ↗
  29. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Pre-training to learn in context. In Proceedings of ACL, 2023. URL https://aclanthology.org/2023.acl-long.267/. ↗
  30. Yuxian Gu, Li Dong, Yaru Hao, Qingxiu Dong, Minlie Huang, and Furu Wei. Towards optimal learning of language models. arXiv preprint arXiv:2402.17759, 2024a. URL https://arxiv.org/abs/2402.17759. — [arXiv:2402.17759](https://arxiv.org/abs/2402.17759) ↗
  31. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In Proceedings of ICLR, 2024b. URL https://openreview.net/forum? id=5h0qf7IBZZ. ↗
  32. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. find
  33. Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of ICML, 2017. URL http://proceedings.mlr.press/v70/haarnoja17a.html?ref=https://githubhelp.com. ↗
  34. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, et al. Pre-trained models: Past, present and future. AI Open, 2021. URL https://www.sciencedirect.com/science/article/pii/ S26666510210002319. ↗
  35. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. URL https://arxiv.org/pdf/1503.02531.pdf. ↗
  36. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, et al. Training compute-optimal large language models. In Proceedings of NeurIPS, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ c1e2fa… ↗
  37. Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the ACL, 2023. URL https://aclantho…
  38. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In Proceedings of COLM, 2024. URL https://openreview. net/forum?id=3X2L2TFr0f&noteI…
  39. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. URL https://arxiv.org/abs/2001.08361. ↗
  40. Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of EMNLP, 2016. URL https://aclanthology.org/D16-1139.pdf. ↗
  41. Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models. In Proceedings of ICML, 2024. URL https://openreview.net/forum?id=lsHZNNoC7r. ↗
  42. Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Proceedings of KR, 2012. URL https://dl.acm.org/doi/10.5555/3031843.3031909. ↗
  43. Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. URL https://arxiv.org/pdf/2005.01643.pdf. ↗
  44. Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. DataComp-LM: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794, 2024a. URL https://arxiv.org/abs/2406.11794… ↗
  45. Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, and Furu Wei. Direct preference knowledge distillation for large language models. arXiv preprint arXiv:2406.19774, 2024b. URL https://arxiv.org/abs/2406.19774. — [arXiv:2406.19774](https://arxiv.org/abs/2406.19774) ↗
  46. Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024. URL https://arxiv.org/ abs/2403.19887. ↗
  47. Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In Proceedings of ICLR, 2024. URL https://openreview.net/forum?id=3xHDeA8Noi. ↗
  48. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019. URL https://arxiv.org/abs/1907. 11692. ↗
  49. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of ICLR, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. ↗
  50. Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D Lane, and Mengwei Xu. Small language models: Survey, measurements, and insights. arXiv preprint arXiv:2409.15790, 2024. URL https://arxiv.org/abs/2409.15790. ↗
  51. Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al. OpenELM: An efficient language model family with open-source training and inference framework. arXiv preprint arXiv:2404.14619, 2024. U… — [arXiv:2404.14619](https://arxiv.org/abs/2404.14619) find
  52. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In CProceedings of EMNLP, 2018. URL https://api.semanticscholar.org/CorpusID:52183757. ↗
  53. Tom Minka et al. Divergence measures and message passing. Technical report, Citeseer, 2005. URL https://www.microsoft.com/en-us/research/wp-content/ uploads/2016/02/tr-2005-173.pdf. ↗
  54. Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of AAAI, 2020. URL https://ojs.aaai.org/index.php/AAAI/article/view/5963/5819. ↗
  55. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of NAACL-HLT, 2016. URL https://aclanthology.org/N16-1098/. ↗
  56. Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Proceedings of NeurIPS, 2023. URL https://openreview.net/forum?id= j5BuTrEj35. ↗
  57. Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. arXiv preprint arXiv:2407.14679, 2024. URL https://www.arxiv.org/ab… ↗
  58. OpenAI. OpenAI: Introducing ChatGPT, 2022. URL https://openai.com/blog/chatgpt. ↗
  59. OpenAI. GPT-4 technical report, 2023. URL https://cdn.openai.com/papers/gpt-4. pdf. ↗
  60. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of ACL, 2016. URL https://aclanthology.org/P16-1144.p… ↗
  61. Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of CVPR, 2019. URL https://openaccess.thecvf.com/content_ CVPR_2019/papers/Park_Relational_Knowledge_Distillation_CVPR_ 2019_paper.pdf. ↗
  62. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. arXiv preprint arXiv:2304.03277, 2023. URL https://arxiv.org/abs/2304. 03277. ↗
  63. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory optimizations toward training trillion parameter models. In Proceedings of SC20, 2020. URL https://arxiv.org/pdf/1910.02054.pdf. ↗
  64. Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022. URL https://openreview.net/pdf/ 3b30857a628099896b6123e85d6cf04c59abe77b.pdf. ↗
  65. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. URL https://arxiv.org/pdf/1910.01108.pdf. ↗
  66. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Proceedings of EMNLP, 2019. URL https://aclanthology.org/D19-1454. ↗
  67. Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. In Proceedings of ICML, 2024. URL https://openreview.net/forum?id=0bmXrtTDUu. ↗
  68. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. URL https://arxiv.org/ pdf/1707.06347.pdf. ↗
  69. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of ICML, 2018. URL https://proceedings.mlr.press/v80/ shazeer18a/shazeer18a.pdf. ↗
  70. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 631(8022):755–759, 2024. URL https://www.nature.com/articles/s41586-024-07566-y. ↗
  71. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Proceedings of NeurIPS, 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 1f89885d556929e98d3ef9b86448f951-Paper.… ↗
  72. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression. In Proceedings EMNLP, 2019. URL https://aclanthology.org/D19-1441. ↗
  73. Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023. URL https://arxiv.org/abs/2307.08621. ↗
  74. Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. In Proceedings of ICLR, 2022. URL https://openreview.net/forum?id=JtBRnrlOEFN. ↗
  75. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. Ul2: Unifying language learning paradigms. In Proceedings of ICLR, 2023. URL https://openreview.net/pdf?id=6ruVLB727MC. ↗
  76. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. URL https://arxiv.org/abs/2312.11805. ↗
  77. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. URL https://arxiv.org/abs/2403.082… ↗
  78. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint…
  79. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. URL https://arxiv.org/abs/2307.09288. ↗
  80. Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. Will we run out of data? an analysis of the limits of scaling datasets in machine learning. arXiv preprint arXiv:2211.04325, 2022. URL https://arxiv.org/abs/2211.04325. ↗
  81. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Proceedings of NeurIPS, 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. ↗
  82. Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. MiniLMv2: Multi-head self-attention relation distillation for compressing pretrained transformers. In Findings of ACL, 2021. URL https://aclanthology.org/2021.findings-acl.188. ↗
  83. Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation. In Proceedings of ACL, 2023. URL https://aclanthology.org/ 2023.acl-long.605.pdf. ↗
  84. Taiqiang Wu, Chaofan Tao, Jiahao Wang, Zhe Zhao, and Ngai Wong. Rethinking kullback-leibler divergence in knowledge distillation for large language models. arXiv preprint arXiv:2404.02657, 2024. URL https://arxiv.org/abs/2404.02657. ↗
  85. Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. In Proceedings of ICLR, 2024. URL https://openreview.net/pdf?id=6s77hjBNfS. ↗
  86. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. DoReMi: Optimizing data mixtures speeds up language model pretraining. In Proceedings of NeurIPS, 2024. URL https://openreview.net/forum?id=lXuByUeHhd. ↗
  87. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In Proceedings of ICML, 2020. URL https://openreview.net/forum?id= B1x8anVFPr. ↗
  88. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024. URL https://arxiv.org/abs/2402.13116. — [arXiv:2402.13116](https://arxiv.org/abs/2402.13116) ↗
  89. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952, 2024. URL https://arxiv.org/abs/2403.16952. ↗
  90. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of ACL, 2019. URL https://aclanthology.org/P19-1472/. ↗
  91. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024. URL https://arxiv.org/abs/ 2401.02385. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gu-2024-miniplm-knowledge-distillation-pre-training/