Elamur Research⌕
☰
home / papers / 2024 / Quantifying Knowledge Distillation Using Partial Information Decomposition

Quantifying Knowledge Distillation Using Partial Information Decomposition

Paper · 2024 · Pasan Dissanayake, Faisal Hamman, Barproda Halder et al.

Summary

Knowledge distillation trains a compact student network with help from a larger teacher. However, the teacher's internal representations may carry information that is useless for the target task, and forcing a small student to copy it can hurt performance. The authors study the information-theoretic limits of distillation using Partial Information Decomposition (PID). They treat the teacher representation, the student representation and the task label as random variables. Knowledge still left to distill is defined as the teacher's unique information about the task, and knowledge already transferred as the redundant information shared by both models. Their theory shows that methods maximizing teacher–student mutual information, such as VID, align the two models without regard to task relevance. Redundant information is hard to optimize directly, so they prove a lower bound via intersection information and build a multi-level optimization called Redundant Information Distillation (RID). In experiments, RID matches VID when the teacher is trained. With an untrained teacher, RID stays close to the no-distillation baseline, while VID degrades sharply. This makes compression more robust when teachers are unreliable.

Key points

  • PID splits I(Y; T, S) into unique, redundant and synergistic parts. In this framework, Uni(Y : T\S) measures knowledge left to distill and Red(Y : T, S) measures knowledge already transferred.
  • Theorem 3.1 identifies a basic flaw of mutual-information-based distillation such as VID for capacity-limited students. Theorem 4.1 shows that intersection information lower-bounds redundant information without Gaussian or other distributional assumptions.
  • Experiments cover CIFAR-10, CIFAR-100 and ImageNet → CUB-200-2011 transfer learning, using WRN-(28,10)/WRN-(16,8) and ResNet-34/ResNet-18 teacher–student pairs. RID is compared with VID, TED and a baseline student without distillation (BAS), averaged over three runs.
  • PID estimates during training confirm the definitions. With a trained teacher, unique information falls and redundant information rises over epochs. With an untrained teacher, both stay near zero, and VID's I(Y; S) remains low.
  • The paper was accepted at AISTATS 2025, and code is available at github.com/pasandissanayake/kd-rid. A stated limitation is that computing exact redundant information during training is costly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States533 works · 443,885 citations🎓Princeton UniversityPrinceton · United States137 works · 53,849 citations🎓University of Maryland, College ParkCollege Park · United States149 works · 29,191 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputer Vision and Pattern Recognition · 3,621 works

Cite

Cite as Pasan Dissanayake, Faisal Hamman, Barproda Halder, Ilia Sucholutsky, Qiuyi Zhang and Sanghamitra Dutta. 2024. Quantifying Knowledge Distillation Using Partial Information Decomposition. arXiv:2411.07483.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2411.07483 · arXiv 2411.07483

@misc{dissanayake2024quantifying,
  title = {{Quantifying Knowledge Distillation Using Partial Information Decomposition}},
  author = {Pasan Dissanayake, Faisal Hamman, Barproda Halder, Ilia Sucholutsky, Qiuyi Zhang and Sanghamitra Dutta},
  year = {2024},
  doi = {10.48550/arxiv.2411.07483},
  eprint = {2411.07483},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.07483}
}

References

52 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G. Hinton. “Distilling the Knowledge in a Neural Network”. In: arXiv preprint arXiv:1503.02531 (2015).
  2. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. “Fitnets: Hints for thin deep nets”. In: ICLR (2015). find
  3. S. Ahn, S. X. Hu, A. Damianou, N. D. Lawrence, and Z. Dai. “Variational information distillation for knowledge transfer”. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, pp. 9163–9171. find
  4. Y. Tian, D. Krishnan, and P. Isola. “Contrastive representation distillation”. In: ICLR (2020). find
  5. C. Liang, S. Zuo, Q. Zhang, P. He, W. Chen, and T. Zhao. “Less is more: Task-aware layer-wise distillation for language model compression”. In: International Conference on Machine Learning. PMLR. 2023, pp. 20852–20867. find
  6. J. Gou, B. Yu, S. J. Maybank, and D. Tao. “Knowledge distillation: A survey”. In: International Journal of Computer Vision 129.6 (2021), pp. 1789–1819. find
  7. I. Sucholutsky et al. “Getting aligned on representational alignment”. In: arXiv preprint arXiv:2310.13018 (2023). — [arXiv:2310.13018](https://arxiv.org/abs/2310.13018) find
  8. Q. Zhang, X. Cheng, Y. Chen, and Z. Rao. “Quantifying the knowledge in a DNN to explain knowledge distillation for classification”. In: IEEE Transactions on Pattern Analysis and Machine Intelligence 45.4 (2022), pp. 5099–5113. find
  9. C. Wang, Q. Yang, R. Huang, S. Song, and G. Huang. “Efficient knowledge distillation from model checkpoints”. In: Advances in Neural Information Processing Systems 35 (2022), pp. 607–619. find
  10. P. L. Williams and R. D. Beer. “Nonnegative decomposition of multivariate information”. In: arXiv preprint arXiv:1004.2515 (2010). — [arXiv:1004.2515](https://arxiv.org/abs/1004.2515) find
  11. V. Griffith, E. K. P. Chong, R. G. James, C. J. Ellison, and J. P. Crutchfield. “Intersection Information Based on Common Randomness”. In: Entropy 16.4 (2014), pp. 1985–2000. issn: 1099-4300. find
  12. N. Bertschinger, J. Rauh, E. Olbrich, J. Jost, and N. Ay. “Quantifying unique information”. In: Entropy 16.4 (2014), pp. 2161–2183. find
  13. V. Griffith and T. Ho. “Quantifying redundant information in predicting a target random variable”. In: Entropy 17.7 (2015), pp. 4644–4653. find
  14. L. Chen, D. Wang, Z. Gan, J. Liu, R. Henao, and L. Carin. “Wasserstein contrastive representation distillation”. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021, pp. 16296–16305. find
  15. R. Miles, A. L. Rodriguez, and K. Mikolajczyk. “Information theoretic representation distillation”. In: arXiv preprint arXiv:2112.00459 (2021). — [arXiv:2112.00459](https://arxiv.org/abs/2112.00459) find
  16. Y. Zhu, N. Liu, Z. Xu, X. Liu, W. Meng, L. Wang, Z. Ou, and J. Tang. “Teach less, learn more: On the undistillable classes in knowledge distillation”. In: Advances in Neural Information Processing Systems 35 (2022), pp. 32011–32024. find
  17. S. Kundu, Q. Sun, Y. Fu, M. Pedram, and P. Beerel. “Analyzing the confidentiality of undistillable teachers in knowledge distillation”. In: Advances in Neural Information Processing Systems 34 (2021), pp. 9181–9192. find
  18. D. Y. Park, M.-H. Cha, D. Kim, B. Han, et al. “Learning student-friendly teacher networks for knowledge distillation”. In: Advances in neural information processing systems 34 (2021), pp. 13292–13303. find
  19. E. Ben-Baruch, M. Karklinsky, Y. Biton, A. Ben-Cohen, H. Lawen, and N. Zamir. “It’s all in the head: Representation knowledge distillation through classifier sharing”. In: arXiv preprint arXiv:2201.06945 (2022). — [arXiv:2201.06945](https://arxiv.org/abs/2201.06945) find
  20. X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu. “Tinybert: Distilling bert for natural language understanding”. In: arXiv preprint arXiv:1909.10351 (2019).
  21. W. Huang, Z. Peng, L. Dong, F. Wei, J. Jiao, and Q. Ye. “Generic-to-specific distillation of masked autoencoders”. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023, pp. 15996–16005. find
  22. N. Tishby, F. C. Pereira, and W. Bialek. “The information bottleneck method”. In: arXiv preprint physics/0004057 (2000). find
  23. N. Tishby and N. Zaslavsky. “Deep learning and the information bottleneck principle”. In: 2015 ieee information theory workshop (itw). IEEE. 2015, pp. 1–5. find
  24. S. Dutta, P. Venkatesh, P. Mardziel, A. Datta, and P. Grover. “An information-theoretic quantification of discrimination with exempt features”. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2020, pp. 3825–3833. find
  25. S. Dutta, P. Venkatesh, P. Mardziel, A. Datta, and P. Grover. “Fairness under feature exemptions: Counterfactual and observational measures”. In: IEEE Transactions on Information Theory 67.10 (2021), pp. 6675–6710. find
  26. S. Dutta and F. Hamman. “A review of partial information decomposition in algorithmic fairness and explainability”. In: Entropy 25.5 (2023), p. 795. find
  27. F. Hamman and S. Dutta. “Demystifying Local and Global Fairness Trade-offs in Federated Learning Using Partial Information Decomposition”. In: International Conference on Learning Representations. 2024. find
  28. P. P. Liang et al. “Quantifying & modeling multimodal interactions: An information decomposition framework”. In: Advances in Neural Information Processing Systems 36 (2024). find
  29. P. P. Liang et al. “Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications”. In: The Twelfth International Conference on Learning Representations. 2024. find
  30. F. Hamman and S. Dutta. “A Unified View of Group Fairness Tradeoffs Using Partial Information Decomposition”. In: IEEE International Symposium on Information Theory (ISIT). 2024, pp. 214–219. find
  31. T. Tax, P. Mediano, and M. Shanahan. “The partial information decomposition of generative neural network models”. In: Entropy 19.9 (2017), p. 474. find
  32. D. A. Ehrlich, A. C. Schneider, M. Wibral, V. Priesemann, and A. Makkeh. “Partial information decomposition reveals the structure of neural representations”. In: arXiv preprint arXiv:2209.10438 (2022). — [arXiv:2209.10438](https://arxiv.org/abs/2209.10438) find
  33. P. Wollstadt, S. Schmitt, and M. Wibral. “A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition.” In: J. Mach. Learn. Res. 24 (2023), pp. 131–1. find
  34. S. Mohamadi, G. Doretto, and D. A. Adjeroh. “More Synergy, Less Redundancy: Exploiting Joint Mutual Information for Self-Supervised Learning”. In: arXiv preprint arXiv:2307.00651 (2023). — [arXiv:2307.00651](https://arxiv.org/abs/2307.00651) find
  35. P. Venkatesh, C. Bennett, S. Gale, T. Ramirez, G. Heller, S. Durand, S. Olsen, and S. Mihalas. “Gaussian partial information decomposition: Bias correction and application to high-dimensional data”. In: Advances in Neural Information Processing Systems 36 (2024). find
  36. B. Halder, F. Hamman, P. Dissanayake, Q. Zhang, I. Sucholutsky, and S. Dutta. “Quantifying Spuriousness of Biased Datasets Using Partial Information Decomposition”. In: ICML Workshop on Data-centric Machine Learning Research (DMLR): Datasets for Foundation Models (2024). find
  37. S. Dewan, R. Zawar, P. Saxena, Y. Chang, A. Luo, and Y. Bisk. “Diffusion PID: Interpreting Diffusion via Partial Information Decomposition”. In: Advances in Neural Information Processing Systems 37 (2024), pp. 2045–2079. find
  38. P. Venkatesh and G. Schamberg. “Partial information decomposition via deficiency for multivariate gaussians”. In: 2022 IEEE International Symposium on Information Theory (ISIT). IEEE. 2022, pp. 2892–2897. find
  39. C. Goswami and A. Merkley. “Analytically deriving Partial Information Decomposition for affine systems of stable and convolution-closed distributions”. In: Advances in Neural Information Processing Systems 37 (2024), pp. 86749–86835. find
  40. M. Kleinman, A. Achille, S. Soatto, and J. C. Kao. “Redundant information neural estimation”. In: Entropy 23.7 (2021), p. 922. find
  41. A. Lyu, A. Clark, and N. Raviv. “Explicit Formula for Partial Information Decomposition”. In: arXiv preprint arXiv:2402.03554 (2024). — [arXiv:2402.03554](https://arxiv.org/abs/2402.03554) find
  42. A. Pakman, A. Nejatbakhsh, D. Gilboa, A. Makkeh, L. Mazzucato, M. Wibral, and E. Schneidman. “Estimating the unique information of continuous variables”. In: Advances in neural information processing systems 34 (2021), pp. 20295–20307. find
  43. T. M. Cover and J. A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). USA: Wiley-Interscience, 2006. isbn: 0471241954. find
  44. A. Achille and S. Soatto. “Emergence of invariance and disentanglement in deep representations”. In: Journal of Machine Learning Research 19.50 (2018), pp. 1–34. find
  45. A. Krizhevsky, V. Nair, and G. Hinton. CIFAR-10 (Canadian Institute for Advanced Research). 2009. url: http://www.cs.toronto.edu/~kriz/cifar.html. ↗
  46. A. Krizhevsky, V. Nair, and G. Hinton. CIFAR-100 (Canadian Institute for Advanced Research). 2009. url: http://www.cs.toronto.edu/~kriz/cifar.html. ↗
  47. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. “Imagenet: A large-scale hierarchical image database”. In: 2009 IEEE conference on computer vision and pattern recognition. IEEE. 2009, pp. 248–255. find
  48. C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. 2011. find
  49. A. Malinin, B. Mlodozeniec, and M. Gales. “Ensemble Distribution Distillation”. In: International Conference on Learning Representations. 2020. find
  50. I. Sucholutsky and M. Schonlau. “Soft-Label Dataset Distillation and Text Dataset Distillation”. In: 2021 International Joint Conference on Neural Networks (IJCNN). 2021, pp. 1–8. find
  51. P. Dissanayake and S. Dutta. “Model Reconstruction Using Counterfactual Explanations: A Perspective From Polytope Theory”. In: Advances in Neural Information Processing Systems (NeurIPS). 2024. find
  52. P. K. Banerjee, E. Olbrich, J. Jost, and J. Rauh. “Unique informations and deficiencies”. In: 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton). IEEE. 2018, pp. 32–38. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dissanayake-2024-quantifying-knowledge-distillation-using-partial/