Distilling the Knowledge in a Neural Network
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1503.02531
Summary
Ensembles and very large regularized networks generalize well, but they are often too slow and expensive to serve to many users. Hinton, Vinyals and Dean propose "distillation", in which a compact student network learns to reproduce the class probabilities of a big "cumbersome" teacher. Raising the softmax temperature exposes the small probabilities the teacher gives to wrong classes, which reveal how it generalizes. The student's loss combines cross-entropy on these softened targets, scaled by T², with a lower-weighted cross-entropy on true labels. The authors show that Caruana-style logit matching is the high-temperature limit of this scheme. On MNIST, a small net recovers most of a dropout-trained large net's accuracy and can even recognize digits missing from its transfer set. For an acoustic model related to Android voice search, most of an ensemble's gain fits into one normal-sized network, and soft targets strongly regularize training on scarce data. The paper also introduces specialist networks for easily confused classes on JFT, which train in parallel, unlike mixtures of experts.
Key points
- MNIST baseline result:
- The teacher (2×1200 ReLU units, dropout, two-pixel jitter) made 67 test errors.
- An unregularized 2×800 net made 146 errors.
- The same 2×800 net trained on soft targets at T=20 made only 74 errors.
- Temperature choice: Students with at least 300 units per layer performed similarly at any T above 8. Very small students (30 units per layer) worked best at T between 2.5 and 4.
- Unseen classes:
- With every 3 removed from the transfer set, the student made 206 errors, 133 of them on the 1010 test threes.
- Raising the 3-class bias by 3.5 cut this to 109 errors, with 98.6% of threes correct.
- Training only on 7s and 8s gave 47.3% error, which fell to 13.2% after bias adjustment.
- Soft targets as regularizer: For the 85M-parameter speech model trained on 3% of the data (about 20M examples):
- Hard targets overfit and peaked at 44.5% test frame accuracy.
- Soft targets reached 57.0%, without early stopping.
- The full-data baseline reached 58.9%.
- Specialists on JFT: Class subsets are chosen from the generalist's confusion matrix, and other classes go into a single dustbin class. Accuracy gains grew as more specialists covered a class. Distilling the specialists back into one net is left for future work.
Cited by
- Towards a Human-like Open-Domain Chatbot
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
- On the Opportunities and Risks of Foundation Models
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
- Emerging Properties in Self-Supervised Vision Transformers
- UNITER: UNiversal Image-TExt Representation Learning
- A Survey of Domain Adaptation for Neural Machine Translation
- Rethinking embedding coupling in pre-trained language models
- Semi-Supervised Sequence Modeling with Cross-View Training
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
- Learning to Parse and Translate Improves Neural Machine Translation
- IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
- Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving
- MiniLLM: On-Policy Distillation of Large Language Models
- The False Promise of Imitating Proprietary LLMs
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
- Large Language Models Can Self-Improve
- DistiLLM: Towards Streamlined Distillation for Large Language Models
- Symphonym: Universal Phonetic Embeddings for Cross-Script Name Matching
- Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- Teaching Small Language Models to Reason
- Merging Models with Fisher-Weighted Averaging
- Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
- Improved Knowledge Distillation via Teacher Assistant
- DINOv2: Learning Robust Visual Features without Supervision
- Continual Lifelong Learning with Neural Networks: A Review
- Carbon Emissions and Large Neural Network Training
- Regularizing Neural Networks by Penalizing Confident Output Distributions
- Discovering Language Model Behaviors with Model-Written Evaluations
- Using the Output Embedding to Improve Language Models
- Learning Compact Metrics for MT
- Pre-trained Models for Natural Language Processing: A Survey
- RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
- DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
- Normalizing without Modernizing: Keeping Historical Wordforms of Middle French while Reducing Spelling Variants
- Masked Language Model Scoring
- Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
- It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
- Membership Inference Attacks against Machine Learning Models
- Distilling Reasoning Capabilities into Smaller Language Models
- Patient Knowledge Distillation for BERT Model Compression
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
- Efficient Transformers: A Survey
- Training data-efficient image transformers & distillation through attention
- Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
- Parsing Tweets into Universal Dependencies
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning
- Symbolic Knowledge Distillation: from General Language Models to Commonsense Models
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
- Pay Less Attention with Lightweight and Dynamic Convolutions
- Scalable Zero-shot Entity Linking with Dense Entity Retrieval
- Unsupervised Data Augmentation for Consistency Training
- ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
- ZeroGen: Efficient Zero-shot Learning via Dataset Generation
- GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation
- Gradient Surgery for Multi-Task Learning
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
- A Survey on Model Compression for Large Language Models
Cite
@misc{hinton2015distilling,
title = {{Distilling the Knowledge in a Neural Network}},
author = {Hinton, Geoffrey and Vinyals, Oriol and Dean, Jeff},
year = {2015},
doi = {10.48550/arxiv.1503.02531},
eprint = {1503.02531},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.1503.02531}
}References
- C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’06, pages 535–541, New York, NY, USA, 2006. ACM. find
- J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng. Large scale distributed deep networks. In NIPS, 2012. find
- T. G. Dietterich. Ensemble methods in machine learning. In Multiple classifier systems, pages 1–15. Springer, 2000. find
- G. E. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEEE, 29(6):82–97, 2012. find
- G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012. find
- R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991. find
- A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097–1105, 2012. find
- J. Li, R. Zhao, J. Huang, and Y. Gong. Learning small-size dnn with output-distribution-based criteria. In Proceedings Interspeech 2014, pages 1910–1914, 2014. find
- N. Srivastava, G.E. Hinton, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014. find