Elamur Research⌕
☰
home / papers / 2025 / Distillation Scaling Laws

Distillation Scaling Laws

Summary

Small language models are cheap to serve, but making them strong through heavy overtraining needs trillions of tokens. Distillation, where a larger teacher supplies training targets for a smaller student, is a possible shortcut. Practitioners still lack guidance on how to split compute between teacher and student, or whether distillation beats ordinary supervised pretraining at the same budget. The authors, from Apple and the University of Oxford, ran a large controlled study. Transformer students and teachers ranged from 143M to 12.6B parameters and were trained on a few billion up to 512B tokens. From these runs they fitted a distillation scaling law that predicts student cross-entropy from student size, distillation tokens and teacher quality. The law also holds when extrapolated beyond the fitted range. They then derived compute-optimal recipes for two cases: a teacher is already available, or one must be trained first. The work matters because it lets teams plan distillation runs ahead of time and avoid wasting compute. It also shows clearly when distillation is the wrong choice.

Key points

  • The teacher's size N_T and its training tokens D_T affect the student only through the teacher's cross-entropy L_T. This removes two dimensions from hyperparameter search.
  • The law captures the "capacity gap", where a stronger teacher can produce a worse student. The authors explain it as a gap in learning capacity, covering both hypothesis space and optimization ability, rather than in size alone. A student can sometimes outperform its teacher.
  • Distillation beats supervised learning only under two conditions:
    • the student's compute or token budget stays below a threshold that grows with student size;
    • the teacher already exists or will be reused for more than one student.
  • If a single student is needed and the teacher must be trained from scratch, supervised pretraining is usually the better option. With unlimited data or compute, distillation never achieves lower cross-entropy than supervised learning.
  • The law was fitted on students with L_S > 2.3 and extrapolated to stronger students. The paper was published at ICML 2025 (arXiv:2502.08606).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Topics Efficient models & compression

Cite

Cite as Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin and Russ Webb. 2025. Distillation Scaling Laws. arXiv:2502.08606.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.08606 · arXiv 2502.08606

@misc{busbridge2025distillation,
  title = {{Distillation Scaling Laws}},
  author = {Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin and Russ Webb},
  year = {2025},
  doi = {10.48550/arxiv.2502.08606},
  eprint = {2502.08606},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.08606}
}

References

156 references, 37 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdin, M. I., Aneja, J., Behl, H. S., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., Lee, J. R., Lee, Y. T., Li, Y., Liu, W., Mendes, C. C. T., Nguyen, A., Price, E., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Wang, X., Ward, R., Wu, Y., Yu, D… our card
  2. Abdin, M. I., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H. S., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Mendes, C. C. T., Chen, W., Chaudhary, V., Chopra, P., Giorno, A. D., de Rosa, G., Dixon, M., Eldan, R., Iter… our card
  3. Abnar, S., Shah, H., Busbridge, D., Ali, A. M. E., Susskind, J., and Thilak, V. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models, 2025. URL https://arxiv.org/abs/2501.12370. — [arXiv:2501.12370](https://arxiv.org/abs/2501.12370) ↗
  4. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce… our card
  5. Aitchison, L. Why you don’t overfit, and don’t need bayes if you only train for one epoch. CoRR, abs/2411.14478, 2024. doi: 10.48550/ARXIV.2411.14478. URL https://doi.org/10.48550/arXiv.2411.14478. — [doi:10.48550/arxiv.2411.14478](https://doi.org/10.48550/arxiv.2411.14478) ↗
  6. Amara, I., Sepahvand, N. M., Meyer, B. H., Gross, W. J., and Clark, J. J. BD-KD: balancing the divergences for online knowledge distillation. CoRR, abs/2212.12965, 2022. doi: 10.48550/ARXIV.2212.12965. URL https://doi.org/10.48550/arXiv.2212.12965. — [doi:10.48550/arxiv.2212.12965](https://doi.org/10.48550/arxiv.2212.12965) ↗
  7. Apple. The axlearn library for deep learning., 2023. URL https://github.com/apple/axlearn. Accessed: 2025-02-11. ↗
  8. Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Explaining neural scaling laws. CoRR, abs/2102.06701, 2021. URL https://arxiv.org/abs/2102.06701. — [arXiv:2102.06701](https://arxiv.org/abs/2102.06701) ↗
  9. Barnett, M. An empirical study of scaling laws for transfer. CoRR, abs/2408.16947, 2024. doi: 10.48550/ARXIV.2408.16947. URL https://doi.org/10.48550/arXiv.2408.16947. — [doi:10.48550/arxiv.2408.16947](https://doi.org/10.48550/arxiv.2408.16947) ↗
  10. Berant, J., Chou, A., Frostig, R., and Liang, P. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a S… our card
  11. ACL, 2013. URL https://aclanthology.org/D13-1160/. ↗ our card
  12. Besiroglu, T., Erdil, E., Barnett, M., and You, J. Chinchilla scaling: A replication attempt. CoRR, abs/2404.10102, 2024. doi: 10.48550/ARXIV.2404.10102. URL https://doi.org/10.48550/arXiv.2404.10102. ↗ our card
  13. Beyer, L., Zhai, X., Royer, A., Markeeva, L., Anil, R., and Kolesnikov, A. Knowledge distillation: A good teacher is patient and consistent. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10915–10924. find
  14. IEEE, 2022. doi: 10.1109/CVPR52688.2022.01065. URL https://doi.org/10.1109/CVPR52688.2022.01065. — [doi:10.1109/cvpr52688.2022.01065](https://doi.org/10.1109/cvpr52688.2022.01065) ↗
  15. Bhakthavatsalam, S., Khashabi, D., Khot, T., Mishra, B. D., Richardson, K., Sabharwal, A., Schoenick, C., Tafjord, O., and Clark, P. Think you have solved direct-answer question answering? try arc-da, the direct-answer AI2 reasoning challenge. CoRR, abs/2102.03315, 2021. URL https://arxiv.org/abs/21… — [arXiv:2102.03315](https://arxiv.org/abs/2102.03315) ↗
  16. Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., Gao, H., Gao, K., Gao, W., Ge, R., Guan, K., Guo, D., Guo, J., Hao, G., Hao, Z., He, Y., Hu, W., Huang, P., Li, E., Li, G., Li, J., Li, Y., Li, Y. K., Liang, W., Lin, F., Liu, A. X., Liu, B., Liu, W., Liu, X… find
  17. Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth… our card
  18. AAAI Press, 2020. doi: 10.1609/AAAI.V34I05.6239. URL https://doi.org/10.1609/aaai. v34i05.6239. — [doi:10.1609/aaai.v34i05.6239](https://doi.org/10.1609/aaai.v34i05.6239) ↗
  19. Blasiok, J., Gopalan, P., Hu, L., and Nakkiran, P. When does optimizing a proper loss yield calibration? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Syste… find
  20. Blondel, M. and Roulet, V. The elements of differentiable programming. CoRR, abs/2403.14606, 2024. doi: 10.48550/ARXIV.2403.14606. URL https://doi.org/10.48550/arXiv.2403.14606. — [doi:10.48550/arxiv.2403.14606](https://doi.org/10.48550/arxiv.2403.14606) ↗
  21. Brown, B. C. A., Juravsky, J., Ehrlich, R. S., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. CoRR, abs/2407.21787, 2024. doi: 10.48550/ARXIV.2407.21787. URL https://doi.org/10.48550/arXiv.2407.21787. ↗ our card
  22. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… our card
  23. Bucila, C., Caruana, R., and Niculescu-Mizil, A. Model compression. In Eliassi-Rad, T., Ungar, L. H., Craven, M., and Gunopulos, D. (eds.), Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, August 20-23, 2006, pp. 535–541. find
  24. ACM, 2006. doi: 10.1145/1150402.1150464. URL https://doi.org/10.1145/1150402.1150464. — [doi:10.1145/1150402.1150464](https://doi.org/10.1145/1150402.1150464) ↗
  25. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Forty-first International Conference on Machine Learning,… our card
  26. OpenReview.net, 2024. URL https://openreview.net/forum?id=ghNRg2mEgN. ↗
  27. Caballero, E., Gupta, K., Rish, I., and Krueger, D. Broken neural scaling laws. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. find
  28. OpenReview.net, 2023. URL https://openreview.net/forum?id=sckjveqlCZ. ↗
  29. Carrell, A. M., Mallinar, N., Lucas, J., and Nakkiran, P. The calibration generalization gap. CoRR, abs/2210.01964, 2022. doi: 10.48550/ARXIV.2210.01964. URL https://doi.org/10.48550/arXiv.2210.01964. — [doi:10.48550/arxiv.2210.01964](https://doi.org/10.48550/arxiv.2210.01964) ↗
  30. CERN. Cern data centre: Key information, March 2018. URL http://information-technology.web.cern.ch/sites/information-technology.web.cern.ch/files/CERNDataCentre_KeyInformation_02March2018V1.pdf. Accessed: 2025-01-29. ↗
  31. Chien, A. A., Lin, L., Nguyen, H., Rao, V., Sharma, T., and Wijayawardana, R. Reducing the carbon impact of generative AI inference (today and in 2035). In Porter, G., Anderson, T., Chien, A. A., Eilam, T., Josephson, C., and Park, J. (eds.), Proceedings of the 2nd Workshop on Sustainable Computer S… find
  32. ACM, 2023. doi: 10.1145/3604930.3605705. URL https://doi.org/10.1145/3604930.3605705. — [doi:10.1145/3604930.3605705](https://doi.org/10.1145/3604930.3605705) ↗
  33. Cho, J. H. and Hariharan, B. On the efficacy of knowledge distillation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pp. 4793–4801. find
  34. IEEE, 2019. doi: 10.1109/ICCV.2019.00489. URL https://doi.org/10.1109/ICCV.2019.00489. — [doi:10.1109/iccv.2019.00489](https://doi.org/10.1109/iccv.2019.00489) ↗
  35. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Aust… our card
  36. Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B. A., Cai, T., Borgeaud, S., van den Driessche, G., Rutherford, E., Hennigan, T., Johnson, M. J., Cassirer, A., Jones, C., Buchatskaya, E., Budden, D., Sifre, L., Osindero, S., Vinyals, O., Ranzato, M… find
  37. PMLR, 2022. URL https://proceedings.mlr.press/v162/clark22a.html. ↗
  38. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. ↗ our card
  39. Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. find
  40. OpenReview.net, 2024. URL https://openreview.net/forum?id=mZn2Xyh9Ec. ↗
  41. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Infor… find
  42. DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H.,… find
  43. Dehghani, M., Arnab, A., Beyer, L., Vaswani, A., and Tay, Y. The efficiency misnomer. CoRR, abs/2110.12894, 2021. URL https://arxiv.org/abs/2110.12894. — [arXiv:2110.12894](https://arxiv.org/abs/2110.12894) ↗
  44. Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248–255. find
  45. IEEE Computer Society, 2009. doi: 10.1109/CVPR.2009.5206848. URL https://doi.org/10.1109/CVPR.2009.5206848. — [doi:10.1109/cvpr.2009.5206848](https://doi.org/10.1109/cvpr.2009.5206848) ↗
  46. Dey, N., Gosal, G., Chen, Z., Khachane, H., Marshall, W., Pathria, R., Tom, M., and Hestness, J. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster. CoRR, abs/2304.03208, 2023. doi: 10.48550/ARXIV.2304.03208. URL https://doi.org/10.48550/arXiv.2304.03208. — [doi:10.48550/arxiv.2304.03208](https://doi.org/10.48550/arxiv.2304.03208) ↗
  47. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B., Biron, B., Tang… find
  48. Epoch AI. Key trends and figures in machine learning, 2023. URL https://epoch.ai/trends. Accessed: 2025-02-11. ↗
  49. Fan, W., Lu, S., Li, X., Zhan, D., and Gan, L. Revisit the essence of distilling knowledge through calibration. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. find
  50. OpenReview.net, 2024. URL https://openreview.net/forum?id=NZgbwzaOIx. ↗
  51. Furlanello, T., Lipton, Z. C., Tschannen, M., Itti, L., and Anandkumar, A. Born-again neural networks. In Dy, J. G. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings… find
  52. PMLR, 2018. URL http://proceedings.mlr.press/v80/furlanello18a.html. ↗
  53. Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., Xin, R., Nezhurina, M., Vasiljevic, I., Jitsev, J., Dimakis, A. G., Ilharco, G., Song, S., Kollar, T., Carmon, Y., Dave, A., Heckel, R., Muennighoff, N., and Schmidt, L. Language mo… find
  54. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework… find
  55. Gunter, T., Wang, Z., Wang, C., Pang, R., Narayanan, A., Zhang, A., Zhang, B., Chen, C., Chiu, C., Qiu, D., Gopinath, D., Yap, D. A., Yin, D., Nan, F., Weers, F., Yin, G., Huang, H., Wang, J., Lu, J., Peebles, J., Ye, K., Lee, M., Du, N., Chen, Q., Keunebroek, Q., Wiseman, S., Evans, S., Lei, T., Ra… — [doi:10.48550/arxiv.2407.21075](https://doi.org/10.48550/arxiv.2407.21075) find
  56. Harutyunyan, H., Rawat, A. S., Menon, A. K., Kim, S., and Kumar, S. Supervision complexity and its role in knowledge distillation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. find
  57. OpenReview.net, 2023. URL https://openreview.net/forum?id=8jU7wy7N7mA. ↗
  58. Havrilla, A. and Liao, W. Understanding scaling laws with statistical and approximation theory for transformer neural networks on intrinsically low-dimensional data. CoRR, abs/2411.06646, 2024. doi: 10.48550/ARXIV.2411.06646. URL https://doi.org/10.48550/arXiv.2411.06646. — [doi:10.48550/arxiv.2411.06646](https://doi.org/10.48550/arxiv.2411.06646) ↗
  59. Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning AI with shared human values. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. our card
  60. OpenReview.net, 2021a. URL https://openreview.net/forum?id=dNy_RKzJacY. ↗
  61. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. our card
  62. Open-Review.net, 2021b. URL https://openreview.net/forum?id=d7KBjmI3GmQ. ↗
  63. Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and McCandlish, S. Scaling laws for autoregressive generative modeling. CoRR, abs/… our card
  64. Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer. CoRR, abs/2102.01293, 2021. URL https://arxiv.org/abs/2102.01293. ↗ our card
  65. Hestness, J., Narang, S., Ardalani, N., Diamos, G. F., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. CoRR, abs/1712.00409, 2017. URL http://arxiv.org/abs/1712.00409. ↗ our card
  66. Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. CoRR, abs/1503.02531, 2015. URL http://arxiv.org/abs/1503.02531. ↗ our card
  67. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L.… our card
  68. Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., Zhang, X., Thai, Z. L., Zhang, K., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng, G., Li, D., Liu, Z., and Sun, M. Minicpm: Unveiling the potential of small language mode… our card
  69. Ildiz, M. E., Gozeten, H. A., Taga, E. O., Mondelli, M., and Oymak, S. High-dimensional analysis of knowledge distillation: Weak-to-strong generalization and scaling laws. CoRR, abs/2410.18837, 2024. doi: 10.48550/ARXIV.2410.18837. URL https://doi.org/10.48550/arXiv.2410.18837. — [doi:10.48550/arxiv.2410.18837](https://doi.org/10.48550/arxiv.2410.18837) ↗
  70. Jain, A., Montanari, A., and Sasoglu, E. Scaling laws for learning with real and surrogate data. CoRR, abs/2402.04376, 2024. doi: 10.48550/ARXIV.2402.04376. URL https://doi.org/10.48550/arXiv.2402.04376. — [doi:10.48550/arxiv.2402.04376](https://doi.org/10.48550/arxiv.2402.04376) ↗
  71. Jelassi, S., Mohri, C., Brandfonbrener, D., Gu, A., Vyas, N., Anand, N., Alvarez-Melis, D., Li, Y., Kakade, S. M., and Malach, E. Mixture of parrots: Experts improve memorization more than reasoning. CoRR, abs/2410.19034, 2024. doi: 10.48550/ARXIV.2410.19034. URL https://doi.org/10.48550/arXiv.2410.… — [doi:10.48550/arxiv.2410.19034](https://doi.org/10.48550/arxiv.2410.19034) ↗
  72. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. CoRR, abs/2310.06825, 2023. doi: 10.4855… find
  73. Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Barzilay, R. and Kan, M. (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, Ju… our card
  74. Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., Das-Sarma, N., Tran-Johnson, E., Johnston, S., Showk, S. E., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., K… find
  75. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020. URL https://arxiv.org/abs/2001.08361. ↗ our card
  76. Kim, Y. and Rush, A. M. Sequence-level knowledge distillation. In Su, J., Carreras, X., and Duh, K. (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1317–1327. The Association for Computational Lin… our card
  77. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980. ↗
  78. Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J. (eds.), Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020,… find
  79. Springer, 2020. doi: 10.1007/978-3-030-58558-7\\_29. URL https://doi.org/10.1007/978-3-030-58558-7_29. — [doi:10.1007/978-3-030-58558-7\\_29](https://doi.org/10.1007/978-3-030-58558-7_29) ↗
  80. Kraft, D. A Software Package for Sequential Quadratic Programming. Deutsche Forschungs- und Versuch-sanstalt für Luft- und Raumfahrt Köln: Forschungs-bericht. Wiss. Berichtswesen d. DFVLR, 1988. URL https://books.google.co.uk/books?id=4rKaGwAACAAJ. ↗
  81. Lee, D., Tian, Z., Zhao, Y., Cheung, K. C., and Zhang, N. L. Hard gate knowledge distillation - leverage calibration for robust and reliable language model. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EM… find
  82. Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. Textbooks are all you need II: phi-1.5 technical report. CoRR, abs/2309.05463, 2023. doi: 10.48550/ARXIV.2309.05463. URL https://doi.org/10.48550/arXiv.2309.05463. ↗ our card
  83. Liu, Z., Zhao, C., Iandola, F. N., Lai, C., Tian, Y., Fedorov, I., Xiong, Y., Chang, E., Shi, Y., Krishnamoorthi, R., Lai, L., and Chandra, V. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. In Forty-first International Conference on Machine Learning, ICML 2024,… find
  84. OpenReview.net, 2024. URL https://openreview.net/forum?id=EIGbXbxcUQ. ↗
  85. Lopez-Paz, D., Bottou, L., Schölkopf, B., and Vapnik, V. Unifying distillation and privileged information. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arx… — [arXiv:1511.03643](https://arxiv.org/abs/1511.03643) find
  86. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. find
  87. OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. ↗
  88. Ludziejewski, J., Krajewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygózdz, T., Sankowski, P., Cygan, M., and Jaszczur, S. Scaling laws for fine-grained mixture of experts. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna,… find
  89. OpenReview.net, 2024. URL https://openreview.net/forum?id=yoqdlynCRs. ↗
  90. Lukasik, M., Bhojanapalli, S., Menon, A. K., and Kumar, S. Teacher’s pet: understanding and mitigating biases in distillation. Trans. Mach. Learn. Res., 2022, 2022. URL https://openreview.net/forum?id=ph3AYXpwEb. ↗
  91. Menon, A. K., Rawat, A. S., Reddi, S. J., Kim, S., and Kumar, S. Why distillation helps: a statistical perspective. CoRR, abs/2005.10419, 2020. URL https://arxiv.org/abs/2005.10419. — [arXiv:2005.10419](https://arxiv.org/abs/2005.10419) ↗
  92. Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., Hussenot, L., Chowdhery, A., Roberts, A., Barua, A., Botev, A., Castro-Ros, A., Slone, A., Héliou, A., Tacchetti, A., Bulanova, A., Paterson, A., Tsai, B., Shahriari, B., Lan,… find
  93. Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D., and Lucic, M. Revisiting the calibration of modern neural networks. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Ann… find
  94. Mirzadeh, S., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., and Ghasemzadeh, H. Improved knowledge distillation via teacher assistant. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAA… our card
  95. AAAI Press, 2020. doi: 10.1609/AAAI.V34I04.5963. URL https://doi.org/10.1609/aaai.v34i04.5963. — [doi:10.1609/aaai.v34i04.5963](https://doi.org/10.1609/aaai.v34i04.5963) ↗
  96. Mobahi, H., Farajtabar, M., and Bartlett, P. L. Self-distillation amplifies regularization in hilbert space. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2… find
  97. Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., and Raffel, C. Scaling data-constrained language models. CoRR, abs/2305.16264, 2023a. doi: 10.48550/ARXIV.2305.16264. URL https://doi.org/10.48550/arXiv.2305.16264. ↗
  98. Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: An… find
  99. Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P. H. S., and Dokania, P. K. Calibrating deep neural networks using focal loss. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural In… find
  100. Muralidharan, S., Sreenivas, S. T., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Compact language models via pruning and knowledge distillation. CoRR, abs/2407.14679, 2024. doi: 10.48550/ARXIV.2407.14679. URL https://doi.org/10.48550/arXiv.2407.146… ↗
  101. Nagarajan, V., Menon, A. K., Bhojanapalli, S., Mobahi, H., and Kumar, S. On student-teacher deviations in distillation: does it pay to disobey? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference… find
  102. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Zaharia, M. Efficient large-scale language model training on GPU clusters using megatron-lm. In de Supin-ski, B. R., Hall, M. W., and… find
  103. ACM, 2021. doi: 10.1145/3458817.3476209. URL https://doi.org/10.1145/3458817.3476209. — [doi:10.1145/3458817.3476209](https://doi.org/10.1145/3458817.3476209) ↗
  104. Nguyen, T. Q. and Salazar, J. Transformers without tears: Improving the normalization of self-attention. In Niehues, J., Cattoni, R., Stüker, S., Negri, M., Turchi, M., Ha, T., Salesky, E., Sanabria, R., Barrault, L., Specia, L., and Federico, M. (eds.), Proceedings of the 16th International Confere… find
  105. Nilsback, M. and Zisserman, A. Automated flower classification over a large number of classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Processing, ICVGIP 2008, Bhubaneswar, India, 16-19 December 2008, pp. 722–729. find
  106. IEEE Computer Society, 2008. doi: 10.1109/ICVGIP.2008.47. URL https://doi.org/10.1109/ICVGIP.2008.47. — [doi:10.1109/icvgip.2008.47](https://doi.org/10.1109/icvgip.2008.47) ↗
  107. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774. ↗ our card
  108. OpenAI and Pilipiszyn, A. Gpt-3 powers the next generation of apps, 2021. URL http://website-url.com. Accessed on Jan 19, 2025. ↗
  109. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 201… our card
  110. Paquette, E., Paquette, C., Xiao, L., and Pennington, J. 4+3 phases of compute-optimal neural scaling laws. CoRR, abs/2405.15074, 2024. doi: 10.48550/ARXIV.2405.15074. URL https://doi.org/10.48550/arXiv.2405.15074. — [doi:10.48550/arxiv.2405.15074](https://doi.org/10.48550/arxiv.2405.15074) ↗
  111. Pareek, D., Du, S. S., and Oh, S. Understanding the gains from repeated self-distillation. CoRR, abs/2407.04600, 2024. doi: 10.48550/ARXIV.2407.04600. URL https://doi.org/10.48550/arXiv.2407.04600. — [doi:10.48550/arxiv.2407.04600](https://doi.org/10.48550/arxiv.2407.04600) ↗
  112. Pearce, T. and Song, J. Reconciling kaplan and chinchilla scaling laws. CoRR, abs/2406.12907, 2024. doi: 10.48550/ARXIV.2406.12907. URL https://doi.org/10.48550/arXiv.2406.12907. — [doi:10.48550/arxiv.2406.12907](https://doi.org/10.48550/arxiv.2406.12907) ↗
  113. Peng, H., Lv, X., Bai, Y., Yao, Z., Zhang, J., Hou, L., and Li, J. Pre-training distillation for large language models: A design space exploration. CoRR, abs/2410.16215, 2024. doi: 10.48550/ARXIV.2410.16215. URL https://doi.org/10.48550/arXiv.2410.16215. ↗
  114. Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Carmon, Y. Resolving discrepancies in compute-optimal scaling of language models. CoRR, abs/2406.19146, 2024. doi: 10.48550/ARXIV.2406.19146. URL https://doi.org/10.48550/arXiv.2406.19146. ↗ our card
  115. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020. URL https://jmlr.org/papers/v21/20-074.html. ↗ our card
  116. Rawat, A. S., Sadhanala, V., Rostamizadeh, A., Chakrabarti, A., Jitkrittum, W., Feinberg, V., Kim, S., Harutyunyan, H., Saunshi, N., Nado, Z., Shivanna, R., Reddi, S. J., Menon, A. K., Anil, R., and Kumar, S. A little help goes a long way: Efficient LLM training by leveraging small lms. CoRR, abs/24… — [doi:10.48550/arxiv.2410.18779](https://doi.org/10.48550/arxiv.2410.18779) find
  117. Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T. P., Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., Antonoglou, I., Anil, R., Borgeaud, S., Dai, A. M., Millican, K., Dyer, E., Glaese, M., Sottiaux, T., Lee, B., Viola, F., Reynolds, M., Xu, Y., Molloy, J., Ch… our card
  118. Rivière, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman,… find
  119. Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y. Fitnets: Hints for thin deep nets. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://… find
  120. Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. find
  121. OpenReview.net, 2020. URL https://openreview.net/forum?id=ryenvpEKDr. ↗
  122. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106, 2021. doi: 10.1145/3474381. URL https://doi.org/10.1145/3474381. ↗
  123. Sardana, N., Portes, J. P., Doubov, S., and Frankle, J. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. our card
  124. OpenReview.net, 2024. URL https://openreview.net/forum?id=0bmXrtTDUu. ↗
  125. Shazeer, N. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL https://arxiv.org/abs/2002.05202. ↗
  126. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. E., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. CoRR, abs/1701.06538, 2017. URL http://arxiv.org/abs/1701.06538. ↗
  127. Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.03314, 2024. doi: 10.48550/ARXIV.2408.03314. URL https://doi.org/10.48550/arXiv.2408.03314. ↗ our card
  128. Sreenivas, S. T., Muralidharan, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. LLM pruning and distillation in practice: The minitron approach. CoRR, abs/2408.11796, 2024. doi: 10.48550/ARXIV.2408.11796. URL https://doi.org/10.48550/arXiv.2408.11… ↗
  129. Stanton, S., Izmailov, P., Kirichenko, P., Alemi, A. A., and Wilson, A. G. Does knowledge distillation really work? In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Pro… find
  130. Stein, C. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 1:197–206, 1956. find
  131. Su, J., Ahmed, M. H. M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. doi: 10.1016/J.NEUCOM.2023.127063. URL https://doi.org/10.1016/j.neucom.2023.127063. ↗ our card
  132. Tian, Y., Krishnan, D., and Isola, P. Contrastive representation distillation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. find
  133. OpenReview.net, 2020. URL https://openreview.net/forum?id=SkgpBJrtvS. ↗
  134. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/ARXIV.2302.13971. URL htt… our card
  135. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Ho… find
  136. Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C., Polat, I., Feng, Y., Moore,… our card
  137. Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Derczynski, L., Xu, W., Ritter, A., and Baldwin, T. (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pp. 94–106. Association for Com… our card
  138. Wortsman, M., Liu, P. J., Xiao, L., Everett, K., Alemi, A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabilities. CoRR, abs/2309.14322, 2023… find
  139. Wortsman, M., Liu, P. J., Xiao, L., Everett, K. E., Alemi, A. A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabilities. In The Twelfth Inter… find
  140. OpenReview.net, 2024. URL https://openreview.net/forum?id=d8w0pmvXbZ. ↗
  141. Wu, C., Acun, B., Raghavendra, R., and Hazelwood, K. M. Beyond efficiency: Scaling AI sustainably. IEEE Micro, 44(5):37–46, 2024a. doi: 10.1109/MM.2024.3409275. URL https://doi.org/10.1109/MM.2024.3409275. — [doi:10.1109/mm.2024.3409275](https://doi.org/10.1109/mm.2024.3409275) ↗
  142. Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. An empirical analysis of compute-optimal inference for problem-solving with language models. CoRR, abs/2408.00724, 2024b. doi: 10.48550/ARXIV.2408.00724. URL https://doi.org/10.48550/arXiv.2408.00724. ↗ our card
  143. Yang, G. and Hu, E. J. Tensor programs IV: feature learning in infinite-width neural networks. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, p… find
  144. PMLR, 2021. URL http://proceedings.mlr.press/v139/yang21c.html. ↗
  145. Yang, G. and Littwin, E. Tensor programs ivb: Adaptive optimization in the infinite-width limit. CoRR, abs/2308.01814, 2023. doi: 10.48550/ARXIV.2308.01814. URL https://doi.org/10.48550/arXiv.2308.01814. — [doi:10.48550/arxiv.2308.01814](https://doi.org/10.48550/arxiv.2308.01814) ↗
  146. Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs V: tuning large neural networks via zero-shot hyperparameter transfer. CoRR, abs/2203.03466, 2022. doi: 10.48550/ARXIV.2203.03466. URL https://doi.org/10.48550/arXiv.220… ↗
  147. Yang, G., Simon, J. B., and Bernstein, J. A spectral condition for feature learning. CoRR, abs/2310.17813, 2023. doi: 10.48550/ARXIV.2310.17813. URL https://doi.org/10.48550/arXiv.2310.17813. ↗
  148. Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor programs VI: feature learning in infinite depth neural networks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. find
  149. OpenReview.net, 2024. URL https://openreview.net/forum?id=17pVDnpwwl. ↗
  150. Yuan, M., Lang, B., and Quan, F. Student-friendly knowledge distillation. Knowl. Based Syst., 296: 111915, 2024. doi: 10.1016/J.KNOSYS.2024.111915. URL https://doi.org/10.1016/j.knosys.2024.111915. — [doi:10.1016/j.knosys.2024.111915](https://doi.org/10.1016/j.knosys.2024.111915) ↗
  151. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Korhonen, A., Traum, D. R., and Màrquez, L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August… find
  152. Zhang, B. and Sennrich, R. Root mean square layer normalization. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 201… find
  153. Zhang, C., Raghu, M., Kleinberg, J. M., and Bengio, S. Pointer value retrieval: A new benchmark for understanding the limits of neural network generalization. CoRR, abs/2107.12580, 2021. URL https://arxiv.org/abs/2107.12580. ↗
  154. Zhang, C., Song, D., Ye, Z., and Gao, Y. Towards the law of capacity gap in distilling language models. CoRR, abs/2311.07052, 2023a. doi: 10.48550/ARXIV.2311.07052. URL https://doi.org/10.48550/arXiv.2311.07052. ↗
  155. Zhang, C., Yang, Y., Liu, J., Wang, J., Xian, Y., Wang, B., and Song, D. Lifting the curse of capacity gap in distilling language models. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  156. Zhu, C., Xu, B., Wang, Q., Zhang, Y., and Mao, Z. On the calibration of large language models and alignment. CoRR, abs/2311.13240, 2023. doi: 10.48550/ARXIV.2311.13240. URL https://doi.org/10.48550/arXiv.2311.13240. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/busbridge-2025-distillation-scaling-laws/