Elamur Research
☰
home / papers / 2019 / Improved Knowledge Distillation via Teacher Assistant

Improved Knowledge Distillation via Teacher Assistant

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1902.03393

Summary

Knowledge distillation trains a compact "student" network to imitate the softened outputs of a large pre-trained "teacher." It is a common way to get models small enough for phones and embedded hardware. Mirzadeh et al. find that this pairing breaks down when the two networks differ too much in capacity. Beyond a certain point, a bigger and more accurate teacher actually produces a worse student.

To close that gap, they propose Teacher Assistant Knowledge Distillation (TAKD). An intermediate-sized network is first distilled from the teacher, and the student then learns only from this assistant. The idea extends to longer chains of several TAs.

The authors evaluate plain CNNs and ResNet models on CIFAR-10, CIFAR-100 and ImageNet. They compare TAKD against standard distillation (BLKD) and training from scratch (NOKD). They also give a learning-theory argument based on error upper bounds and inspect loss landscapes. TAKD consistently improves the student, especially under aggressive compression.

The work matters because it shows that the biggest available teacher is not automatically the best choice. It also offers a simple, architecture-agnostic fix that can be combined with other distillation methods.

Key points

  • Capacity-gap experiment: a CNN student with 2 convolutional layers was distilled from teachers with 4, 6, 8 and 10 layers on CIFAR-10 and CIFAR-100. The largest teacher did not yield the best student. In the reverse setup, a fixed 10-layer teacher gave diminishing benefit as the student shrank too far.
  • Baseline objective (BLKD): the loss is a λ-weighted mix of cross-entropy on true labels and a temperature-scaled (τ) KL term toward the teacher's softened outputs.
  • Theoretical justification: the argument builds on Lopez-Paz et al. (2015). Replacing one large student–teacher gap with two smaller ones (student–TA and TA–teacher) gives faster learning rates and a tighter asymptotic error bound than BLKD.
  • Example chains: plain CNN 10 → 4 → 2 and ResNet 110 → 20 → 8. An additional comparison against other distillation alternatives used ResNet8 and ResNet14 students with a ResNet26 teacher.
  • Loss landscape and future work: visualizations with the method of Li et al. (2018) on CIFAR-100 show the TAKD student sitting in flatter minima than NOKD and BLKD, which the authors link to better generalization. Automatic, data-driven TA selection is left as future work. The paper was published at AAAI 2020.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

136 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 133 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1902.03393 · arXiv 1902.03393

@misc{mirzadeh2019improved,
  title = {{Improved Knowledge Distillation via Teacher Assistant}},
  author = {Mirzadeh, Seyed-Iman and Farajtabar, Mehrdad and Li, Ang and Levine, Nir and Matsukawa, Akihiro and Ghasemzadeh, Hassan},
  year = {2019},
  doi = {10.48550/arxiv.1902.03393},
  eprint = {1902.03393},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1902.03393}
}

References

39 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anil, R.; Pereyra, G.; Passos, A.; Ormándi, R.; Dahl, G.; and Hinton, G. 2018. Large scale distributed neural network training through online distillation. CoRR abs/1804.03235. — [arXiv:1804.03235](https://arxiv.org/abs/1804.03235) find
  2. Ba, J., and Caruana, R. 2014. Do deep nets really need to be deep? In NIPS, 2654–2662. find
  3. Bergstra, J.; Bardenet, R.; Bengio, Y.; and Kégl, B. 2011. Algorithms for hyper-parameter optimization. In NIPS. find
  4. Bertsekas, D. 2005. Dynamic programming and optimal control, volume 1. Athena scientific Belmont, MA. find
  5. Bucila, C.; Caruana, R.; and Niculescu-Mizil, A. 2006. Model compression. In SIGKDD, 535–541. ACM. find
  6. Chen, G.; Choi, W.; Yu, X.; Han, T.; and Chandraker, M. 2017. Learning efficient object detection models with knowledge distillation. In NIPS, 742–751. find
  7. Czarnecki, W.; Osindero, S.; Jaderberg, M.; Swirszcz, G.; and Pascanu, R. 2017. Sobolev training for neural networks. In NIPS, 4278–4287. find
  8. Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805. find
  9. Furlanello, T.; Lipton, Z.; Tschannen, M.; Itti, L.; and Anandkumar, A. 2018. Born again neural networks. arXiv preprint arXiv:1805.04770. — [arXiv:1805.04770](https://arxiv.org/abs/1805.04770) find
  10. Han, K. J.; Chandrashekaran, A.; Kim, J.; and Lane, I. 2017. The capio 2017 conversational speech recognition system. CoRR abs/1801.00059. — [arXiv:1801.00059](https://arxiv.org/abs/1801.00059) find
  11. Han, S.; Mao, H.; and Dally, W. J. 2016. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In 4th International Conference on Learning Representations, ICLR 2016. find
  12. He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In CVPR, 770–778. find
  13. Heo, B.; Lee, M.; Yun, S.; and Choi, J. 2018. Improving knowledge distillation with supporting adversarial samples. arXiv preprint arXiv:1805.05532. — [arXiv:1805.05532](https://arxiv.org/abs/1805.05532) find
  14. Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop. find our card
  15. Hu, J.; Shen, L.; and Sun, G. 2018. Squeeze-and-excitation networks. In CVPR. find
  16. Huang, G.; Liu, Z.; Maaten, L.; and Weinberger, K. 2017. Densely connected convolutional networks. CVPR 2261–2269. find
  17. Li, H.; Kadav, A.; Durdanovic, I.; Samet, H.; and Graf, H. P. 2016. Pruning filters for efficient convnets. CoRR abs/1608.08710. — [arXiv:1608.08710](https://arxiv.org/abs/1608.08710) find
  18. Li, H.; Xu, Z.; Taylor, G.; Studer, C.; and Goldstein, T. 2018. Visualizing the loss landscape of neural nets. In NIPS, 6391–6401. find
  19. Lopez-Paz, D.; Bottou, L.; Schölkopf, B.; and Vapnik, V. 2015. Unifying distillation and privileged information. arXiv preprint arXiv:1511.03643. — [arXiv:1511.03643](https://arxiv.org/abs/1511.03643) find
  20. Microsoft-Research. 2018. Neural network intelligence toolkit. find
  21. Paszke, A.; Gross, S.; Chintala, S.; and et. al. 2017. Automatic differentiation in pytorch. In NIPS Autodiff Workshop. find
  22. Polino, A.; Pascanu, R.; and Alistarh, D. 2018. Model compression via distillation and quantization. In ICLR. find
  23. Romero, A.; Ballas, N.; Kahou, S.; Chassang, A.; Gatta, C.; and Bengio, Y. 2014. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550. find
  24. Sau, B., and Balasubramanian, V. 2016. Deep model compression: Distilling knowledge from noisy teachers. arXiv preprint arXiv:1610.09650. — [arXiv:1610.09650](https://arxiv.org/abs/1610.09650) find
  25. Schmitt, S.; Hudson, J.; Zidek, A.; and et. al. 2018. Kickstarting deep reinforcement learning. CoRR abs/1803.03835. — [arXiv:1803.03835](https://arxiv.org/abs/1803.03835) find
  26. Tai, C.; Xiao, T.; Wang, X.; and Weinan, E. 2015. Convolutional neural networks with low-rank regularization. CoRR abs/1511.06067. — [arXiv:1511.06067](https://arxiv.org/abs/1511.06067) find
  27. Tarvainen, A., and Valpola, H. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NIPS. find
  28. Urban, G.; Geras, K. J.; Kahou, S. E.; and et. al. 2017. Do deep convolutional nets really need to be deep and convolutional? In 5th International Conference on Learning Representations, ICLR 2017, Conference Track Proceedings. find
  29. Vapnik, V. 1998. Statistical learning theory. 1998, volume 3. Wiley, New York. find
  30. Wang, J.; Bao, W.; Sun, L.; Zhu, X.; Cao, B.; and Yu, P. S. 2018a. Private model compression via knowledge distillation. CoRR abs/1811.05072. — [arXiv:1811.05072](https://arxiv.org/abs/1811.05072) find
  31. Wang, X.; Zhang, R.; Sun, Y.; and Qi, J. 2018b. Kdgan: Knowledge distillation with generative adversarial networks. In NIPS, 783–794. find
  32. Wang, Y.; Xu, C.; Xu, C.; and Tao, D. 2018c. Adversarial learning of portable student networks. In AAAI. find
  33. Xu, Z.; Hsu, Y.; and Huang, J. 2018. Training shallow and thin networks for acceleration via knowledge distillation with conditional adversarial networks. In 6th International Conference on Learning Representations, ICLR 2018. find
  34. Yim, J.; Joo, D.; Bae, J.; and Kim, J. 2017. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 7130– 7138. find
  35. You, S.; Xu, C.; Xu, C.; and Tao, D. 2017. Learning from multiple teacher networks. In SIGKDD, 1285–1294. ACM. find
  36. Yu, R.; Li, A.; Morariu, V. I.; and Davis, L. 2017. Visual relationship detection with internal and external linguistic knowledge distillation. In ICCV. find
  37. Yu, R.; Li, A.; Chen, C.; Lai, J.; Morariu, V.; Han, X.; Gao, M.; Lin, C.; and Davis, L. 2018. Nisp: Pruning networks using neuron importance score propagation. In CVPR. find
  38. Z, S., and K, N. 2016. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928. find
  39. Zhang, Y.; Xiang, T.; Hospedales, T.; and Lu, H. 2017. Deep mutual learning. CoRR abs/1706.00384. Network Architectures — [arXiv:1706.00384](https://arxiv.org/abs/1706.00384) find