Elamur Research⌕
☰
home / papers / 2023 / Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework

Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework

Paper · 2023 · Lirong Wu; Haitao Lin; Yufei Huang; Tianyu Fan; and Stan Z. Li

Summary

Graph Neural Networks (GNNs) classify nodes well, but their reliance on neighbourhood message passing makes inference slow. MLPs are fast but less accurate. GNN-to-MLP distillation tries to bridge this gap, and the authors ask which parts of a teacher GNN's knowledge actually reach the student MLP.

Using graph signal processing, they split GNN knowledge into a low-frequency part (smoothness among neighbours) and a high-frequency part (differences between connected nodes). They then link these spectral components to spatial interpretations. Their analysis reveals an "information drowning" effect: in standard node-to-node distillation such as GLNN, the dominant low-frequency signal tends to swamp the high-frequency signal.

To counter this, they propose FF-G2M (Full-Frequency GNN-to-MLP). It pairs a topology-aware low-frequency distillation loss with a high-frequency loss that keeps teacher-student pairwise node relations consistent.

Across six benchmark graphs and three GNN backbones, the distilled MLPs exceed both plain MLPs and their own teachers. The result shows that fast, graph-free inference need not cost accuracy.

Key points

  • Average gains: +12.6% over vanilla MLPs and +2.6% over the corresponding teacher GNNs, computed across six datasets and three GNN architectures.
  • Datasets: Cora, Citeseer, Pubmed, Amazon-Photo, Coauthor-CS, Coauthor-Phy.
  • With a GCN teacher on Cora, accuracy is 84.3% for FF-G2M, compared with 82.8% for GLNN, 82.2% for GCN and 59.7% for the vanilla MLP.
  • In ablations, low-frequency distillation (L_LFD) carries most of the benefit and high-frequency distillation (L_HFD) plays a supporting role. Combining the two works better than using either alone.
  • The paper appeared at AAAI 2023, and the code is released at github.com/LirongWu/FF-G2M.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China236 works · 14,072 citations🎓Westlake UniversityHangzhou · China60 works · 8,829 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 328 works

Cite

Cite as Lirong Wu; Haitao Lin; Yufei Huang; Tianyu Fan; and Stan Z. Li. 2023. Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework. arXiv:2305.10758.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.10758 · arXiv 2305.10758

@misc{fan2023extracting,
  title = {{Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework}},
  author = {Lirong Wu; Haitao Lin; Yufei Huang; Tianyu Fan; and Stan Z. Li},
  year = {2023},
  doi = {10.48550/arxiv.2305.10758},
  eprint = {2305.10758},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.10758}
}

References

47 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bo, D.; Wang, X.; Shi, C.; and Shen, H. 2021a. Beyond Low-frequency Information in Graph Convolutional Networks. arXiv preprint arXiv:2101.00797. find
  2. Bo, D.; Wang, X.; Shi, C.; and Shen, H. 2021b. Beyond Low-frequency Information in Graph Convolutional Networks. Association for the Advance of Artificial Intelligence (AAAI). find
  3. Bruna, J.; Zaremba, W.; Szlam, A.; and LeCun, Y. 2013. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203. find
  4. Chien, E.; Peng, J.; Li, P.; and Milenkovic, O. 2021. Adaptive Universal Generalized PageRank Graph Neural Network. International Conference on Learning Representations (ICLR). find
  5. Chung, F. R.; and Graham, F. C. 1997. Spectral graph theory. find
  6. American Mathematical Soc. Defferrard, M.; Bresson, X.; and Vandergheynst, P. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. arXiv preprint arXiv:1606.09375.
  7. Errica, F.; Podda, M.; Bacciu, D.; and Micheli, A. 2019. A fair comparison of graph neural networks for graph classification. arXiv preprint arXiv:1912.09893. — [arXiv:1912.09893](https://arxiv.org/abs/1912.09893) find
  8. Fan, W.; Ma, Y.; Li, Q.; He, Y.; Zhao, E.; Tang, J.; and Yin, D. 2019. Graph neural networks for social recommendation. In The world wide web conference, 417–426. find
  9. Ghorbani, M.; Bahrami, M.; Kazi, A.; Soleymani Baghshah, M.; Rabiee, H. R.; and Navab, N. 2021. GKD: Semi-supervised Graph Knowledge Distillation for Graph-Independent Inference. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 709–718. find
  10. Springer. Giles, C. L.; Bollacker, K. D.; and Lawrence, S. 1998. Cite-Seer: An automatic citation indexing system. In Proceedings of the third ACM conference on Digital libraries, 89–98. find
  11. Gou, J.; Yu, B.; Maybank, S. J.; and Tao, D. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6): 1789–1819. find
  12. Hamilton, W.; Ying, Z.; and Leskovec, J. 2017. Inductive representation learning on large graphs. In Advances in neural information processing systems, 1024–1034.
  13. Hinton, G.; Vinyals, O.; Dean, J.; et al. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7).
  14. Iscen, A.; Tolias, G.; Avrithis, Y.; and Chum, O. 2019. Label propagation for deep semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5070–5079. find
  15. Jia, Z.; Lin, S.; Ying, R.; You, J.; Leskovec, J.; and Aiken, A. 2020. Redundancy-Free Computation for Graph Neural Networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 997–1005. find
  16. Joshi, C. K.; Liu, F.; Xun, X.; Lin, J.; and Foo, C.-S. 2021. On Representation Knowledge Distillation for Graph Neural Networks. arXiv preprint arXiv:2111.04964. — [arXiv:2111.04964](https://arxiv.org/abs/2111.04964) find
  17. Kim, T.; Oh, J.; Kim, N.; Cho, S.; and Yun, S.-Y. 2021. Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation. arXiv preprint arXiv:2105.08919. — [arXiv:2105.08919](https://arxiv.org/abs/2105.08919) find
  18. Kipf, T. N.; and Welling, M. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
  19. Lassance, C.; Bontonou, M.; Hacene, G. B.; Gripon, V.; Tang, J.; and Ortega, A. 2020. Deep geometric knowledge distillation with graphs. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 8484–8488. IEEE. find
  20. Liu, M.; Gao, H.; and Ji, S. 2020. Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 338–348. find
  21. Liu, Z.; Luo, Y.; Wu, L.; Li, S.; Liu, Z.; and Li, S. Z. 2022. Are Gradients on Graph Structure Reliable in Gray-box Attacks? arXiv preprint arXiv:2208.05514. — [arXiv:2208.05514](https://arxiv.org/abs/2208.05514) find
  22. Luo, Y.; Chen, A.; Yan, K.; and Tian, L. 2021. Distilling Self-Knowledge From Contrastive Links to Classify Graph Nodes Without Passing Messages. arXiv preprint arXiv:2106.08541. — [arXiv:2106.08541](https://arxiv.org/abs/2106.08541) find
  23. McCallum, A. K.; Nigam, K.; Rennie, J.; and Seymore, K. 2000. Automating the construction of internet portals with machine learning. Information Retrieval, 3(2): 127–163. find
  24. McInnes, L.; Healy, J.; and Melville, J. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426.
  25. Pei, H.; Wei, B.; Chang, K. C.-C.; Lei, Y.; and Yang, B. 2020. Geom-gcn: Geometric graph convolutional networks. arXiv preprint arXiv:2002.05287. — [arXiv:2002.05287](https://arxiv.org/abs/2002.05287) find
  26. Ren, Y.; Ji, J.; Niu, L.; and Lei, M. 2021. Multi-task Self-distillation for Graph-based Semi-Supervised Learning. arXiv preprint arXiv:2112.01174. — [arXiv:2112.01174](https://arxiv.org/abs/2112.01174) find
  27. Ricaud, B.; Borgnat, P.; Tremblay, N.; Gonçalves, P.; and Vandergheynst, P. 2019. Fourier could be a data scientist: From graph Fourier transform to signal processing on graphs. Comptes Rendus Physique, 20(5): 474–488. find
  28. Sandryhaila, A.; and Moura, J. M. 2013. Discrete signal processing on graphs: Graph fourier transform. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, 6167–6170. IEEE. find
  29. Sen, P.; Namata, G.; Bilgic, M.; Getoor, L.; Galligher, B.; and Eliassi-Rad, T. 2008. Collective classification in network data. AI magazine, 29(3): 93–93.
  30. Shchur, O.; Mumme, M.; Bojchevski, A.; and Günnemann, S. 2018. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868. — [arXiv:1811.05868](https://arxiv.org/abs/1811.05868) find
  31. Shuman, D. I.; Narang, S. K.; Frossard, P.; Ortega, A.; and Vandergheynst, P. 2013. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains. IEEE signal processing magazine, 30(3): 83–98. Veličković, P.; Cucurull, G.; Casa…
  32. Wang, M.; Zheng, D.; Ye, Z.; Gan, Q.; Li, M.; Song, X.; Zhou, J.; Ma, C.; Yu, L.; Gai, Y.; He, T.; Karypis, G.; Li, J.; and Zhang, Z. 2019. Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks. arXiv preprint arXiv:1909.01315. — [arXiv:1909.01315](https://arxiv.org/abs/1909.01315) find
  33. Wu, F.; Souza, A.; Zhang, T.; Fifty, C.; Yu, T.; and Weinberger, K. 2019. Simplifying graph convolutional networks. In International conference on machine learning, 6861– 6871. find
  34. PMLR. Wu, L.; Lin, H.; Gao, Z.; Tan, C.; Li, S.; et al. 2021a. GraphMixup: Improving Class-Imbalanced Node Classification on Graphs by Self-supervised Context Prediction. arXiv preprint arXiv:2106.11133. — [arXiv:2106.11133](https://arxiv.org/abs/2106.11133) find
  35. Wu, L.; Lin, H.; Gao, Z.; Tan, C.; Li, S.; et al. 2021b. Self-supervised on Graphs: Contrastive, Generative, or Predictive. arXiv preprint arXiv:2105.07342. — [arXiv:2105.07342](https://arxiv.org/abs/2105.07342) find
  36. Wu, L.; Lin, H.; Huang, Y.; and Li, S. Z. 2022a. Knowledge Distillation Improves Graph Structure Augmentation for Graph Neural Networks. In Oh, A. H.; Agarwal, A.; Belgrave, D.; and Cho, K., eds., Advances in Neural Information Processing Systems. find
  37. Wu, L.; Lin, H.; Xia, J.; Tan, C.; and Li, S. Z. 2022b. Multi-level disentanglement graph neural network. Neural Computing and Applications, 1–15. find
  38. Wu, T.; Zhao, Z.; Wang, J.; Bai, X.; Wang, L.; Wong, N.; and Yang, Y. 2023. Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs. arXiv preprint arXiv:2303.13763. find
  39. Wu, Z.; Pan, S.; Chen, F.; Long, G.; Zhang, C.; and Philip, S. Y. 2020. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems.
  40. Xia, J.; Wu, L.; Chen, J.; Hu, B.; and Li, S. Z. 2022. SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation. In Proceedings of the ACM Web Conference 2022, 1070–1079. find
  41. Yan, B.; Wang, C.; Guo, G.; and Lou, Y. 2020. TinyGNN: Learning Efficient Graph Neural Networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 1848–1856. find
  42. Yang, C.; Liu, J.; and Shi, C. 2021. Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework. In Proceedings of the Web Conference 2021, 1227–1237. find
  43. Zhang, M.; and Chen, Y. 2018. Link prediction based on graph neural networks. arXiv preprint arXiv:1802.09691. — [arXiv:1802.09691](https://arxiv.org/abs/1802.09691) find
  44. Zhang, S.; Liu, Y.; Sun, Y.; and Shah, N. 2021. Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation. arXiv preprint arXiv:2110.08727. — [arXiv:2110.08727](https://arxiv.org/abs/2110.08727) find
  45. Zhang, W.; Miao, X.; Shao, Y.; Jiang, J.; Chen, L.; Ruas, O.; and Cui, B. 2020. Reliable data distillation on graph convolutional network. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, 1399– 1414. find
  46. Zhou, J.; Cui, G.; Hu, S.; Zhang, Z.; Yang, C.; Liu, Z.; Wang, L.; Li, C.; and Sun, M. 2020. Graph neural networks: A review of methods and applications. AI Open, 1: 57–81. find
  47. Zhu, J.; Yan, Y.; Zhao, L.; Heimann, M.; Akoglu, L.; and Koutra, D. 2020. Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs. Advances in Neural Information Processing Systems, 33. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2023-extracting-low-high-frequency-knowledge/