Elamur Research
☰
home / papers / 2019 / Fast Graph Representation Learning with PyTorch Geometric

Fast Graph Representation Learning with PyTorch Geometric

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1903.02428

Summary

Graph neural networks are difficult to run efficiently: inputs are sparse, irregular and vary in size, which makes high GPU throughput hard to achieve. The authors present PyTorch Geometric (PyG), an open-source extension of PyTorch under the MIT license for deep learning on graphs, point clouds and manifolds. Its core is a generic MessagePassing abstraction. A user defines a message function, an update function and an aggregation (sum, mean or max), and the library runs them through custom CUDA gather/scatter kernels. Graphs of different sizes are batched by stacking them into one block-diagonal sparse adjacency matrix. PyG bundles many published convolution and pooling operators, along with loaders for common benchmark datasets. To check the implementations, the authors re-ran these methods under uniform settings on node, graph and point cloud classification. Most published results were reproduced, and PyG trained faster than DGL. This matters because researchers get one consistent and fast toolkit for both prototyping and fair comparison of GNN methods.

Key points

  • Operators included: GCN, SGC, ChebNet, ARMA, GraphSAGE, GAT, AGNN, GIN, APPNP, RGCN, PointNet++, PointCNN, MoNet, SplineCNN and EdgeCNN, plus pooling methods such as DiffPool, top-k, Graclus, Set2Set and SortPool.
  • Datasets: automatic loaders cover over 60 graph kernel benchmarks (e.g., PROTEINS, IMDB-BINARY), Cora/CiteSeer/PubMed, QM9, ModelNet10/40, ShapeNet, FAUST, ICEWS and GDELT, among others.
  • Accuracy results:
    • On semi-supervised node classification, APPNP scored best (Cora: 83.3% with the fixed split, 82.2% with random splits). Random splits lowered accuracy for every model.
    • DiffPool reached 92.1% on REDDIT-BINARY.
    • PointCNN reached 93.28% on ModelNet10.
  • Training speed vs. DGL:
    • GAT on Cora took 0.80s in PyG, compared with 5.36s and 6.31s for the two DGL variants.
    • GCN on PubMed took 0.32s in PyG, compared with up to 12.91s in DGL.
  • Gather/scatter vs. sparse-matrix multiplication: gather/scatter stays faster until the average node degree reaches roughly 128, and it does not require coalesced input. However, GPU scatter is non-deterministic, so training runs can differ even with the same seed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

281 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1903.02428 · arXiv 1903.02428

@misc{fey2019fast,
  title = {{Fast Graph Representation Learning with PyTorch Geometric}},
  author = {Fey, Matthias and Lenssen, Jan Eric},
  year = {2019},
  doi = {10.48550/arxiv.1903.02428},
  eprint = {1903.02428},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1903.02428}
}

References

53 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. F. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, Ç. Gülçehre, F. Song, A. J. Ballard, J. Gilmer, G. E. Dahl, A. Vaswani, K. Allen, C. Nash, V. Langston, C. Dyer, N. Heess, D. Wierstra, P. Kohli, M. Botvini… find our card
  2. F. M. Bianchi, D. Grattarola, L. Livi, and C. Alippi. Graph neural networks with convolutional ARMA filters. CoRR, abs/1901.01343, 2019. — [arXiv:1901.01343](https://arxiv.org/abs/1901.01343) find
  3. F. Bogo, J. Romero, M. Loper, and M. J. Black. FAUST: Dataset and evaluation for 3D mesh registration. In CVPR, 2014. find
  4. A. Bojchevski and S. Günnemann. Deep gaussian embedding of attributed graphs: Unsupervised inductive learning via ranking. In ICLR, 2018. find
  5. E. Boschee, J. Lautenschlager, S. O’Brien, S. Shellman, J. Starz, and M. Ward. ICEWS coded event data. Harvard Dataverse, 2015. find
  6. M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst. Geometric deep learning: Going beyond euclidean data. In Signal Processing Magazine, 2017. find
  7. C. Cai and Y. Wang. A simple yet effective baseline for non-attribute graph classification. CoRR, abs/1811.03508, 2018. — [arXiv:1811.03508](https://arxiv.org/abs/1811.03508) find
  8. C. Cangea, P. Veličković, N. Jovanović, T. N. Kipf, and P. Liò. Towards sparse hierarchical graph classifiers. In NeurIPS-W, 2018. find
  9. A. X. Chang, T. Funkhouser, L. J. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An information-rich 3D model repository. CoRR, abs/1512.03012, 2015. — [arXiv:1512.03012](https://arxiv.org/abs/1512.03012) find
  10. M. Defferrard, X. Bresson, and P. Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In NIPS, 2016. find our card
  11. T. Derr, Y. Ma, and J. Tang. Signed graph convolutional networks. In ICDM, 2018. find
  12. I. S. Dhillon, Y. Guan, and B. Kulis. Weighted graph cuts without eigenvectors: A multilevel approach. In TPAMI, 2007. find
  13. B. O. Fagginger Auer and R. H. Bisseling. A GPU algorithm for greedy graph matching. In Facing the Multicore - Challenge II - Aspects of New Paradigms and Technologies in Parallel Computing, 2011. find
  14. M. Fey. Just jump: Dynamic neighborhood aggregation in graph neural networks. In ICLR-W, 2019. find
  15. M. Fey, J. E. Lenssen, F. Weichert, and H. Müller. SplineCNN: Fast geometric deep learning with continuous B-spline kernels. In CVPR, 2018. find
  16. H. Gao and S. Ji. Graph U-Net. https://openreview.net/forum?id=HJePRoAct7, 2018. Submitted to ICLR. ↗
  17. J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In ICML, 2017. find our card
  18. P. Guerrero, Y. Kleiman, M. Ovsjanikov, and N. J. Mitra. PCPNet: Learning local shape properties from raw point clouds. Computer Graphics Forum, 37, 2018. find
  19. W. L. Hamilton, R. Ying, and J. Leskovec. Inductive representation learning on large graphs. In NIPS, 2017. find our card
  20. W. Jin, C. Zhang, P. Szekely, and X. Ren. Recurrent event network for reasoning over temporal knowledge graphs. In ICLR-W, 2019. find
  21. K. Kersting, N. M. Kriege, C. Morris, P. Mutzel, and M. Neumann. Benchmark data sets for graph kernels. http://graphkernels.cs.tu-dortmund.de, 2016. ↗
  22. T. N. Kipf and M. Welling. Variational graph auto-encoders. In NIPS-W, 2016. find
  23. T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017. find our card
  24. J. Klicpera, A. Bojchevski, and S. Günnemann. Predict then propagate: Graph neural networks meet personalized PageRank. In ICLR, 2019. find
  25. S. Kumar, F. Spezzano, V. Subrahmanian, and C. Faloutsos. Edge weight prediction in weighted signed networks. In ICDM, 2016. find
  26. K. Leetaru and P. A. Schrodt. GDELT: Global data on events, location, and tone. ISA Annual Convention, 2013. find
  27. Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel. Gated graph sequence neural networks. In ICLR, 2016. find
  28. Y. Li, R. Bu, M. Sun, W. Wu, X. Di, and B. Chen. PointCNN: Convolution on -transformed points. In NeurIPS, 2018. find
  29. G. Montavon, M. Rupp, V. Gobre, A. Vazquez-Mayagoitia, K. Hansen, A. Tkatchenko, K. Müller, and O. A. von Lilienfeld. Machine learning of molecular electronic properties in chemical compound space. New Journal of Physics, 2013. find
  30. F. Monti, D. Boscaini, J. Masci, E. Rodolà, J. Svoboda, and M. M. Bronstein. Geometric deep learning on graphs and manifolds using mixture model CNNs. In CVPR, 2017. find
  31. C. Morris, M. Ritzert, M. Fey, W. L. Hamilton, J. E. Lenssen, G. Rattan, and M. Grohe. Weisfeiler and Leman go neural: Higher-order graph neural networks. In AAAI, 2019. find
  32. S. Pan, R. Hu, G. Long, J. Jiang, L. Yao, and C. Zhang. Adversarially regularized graph autoencoder for graph embedding. In IJCAI, 2018. find
  33. A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in PyTorch. In NIPS-W, 2017. find
  34. C. R. Qi, L. Yi, H. Su, and L. J. Guibas. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In NIPS, 2017. find
  35. R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 2014. find
  36. A. Ranjan, T. Bolkart, S. Sanyal, and M. J. Black. Generating 3D faces using convolutional mesh autoencoders. In ECCV, 2018. find
  37. M. S. Schlichtkrull, T. N. Kipf, P. Bloem, R. van den Berg, I. Titov, and M. Welling. Modeling relational data with graph convolutional networks. In ESWC, 2018. find
  38. G. Sen, G. Namata, M. Bilgic, and L. Getoor. Collective classification in network data. AI Magazine, 29, 2008. find
  39. O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann. Pitfalls of graph neural network evaluation. In NeurIPS-W, 2018. find
  40. M. Simonovsky and N. Komodakis. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In CVPR, 2017. find
  41. K. K. Thekumparampil, C. Wang, S. Oh, and L. Li. Attention-based graph neural network for semi-supervised learning. CoRR, abs/1803.03735, 2018. — [arXiv:1803.03735](https://arxiv.org/abs/1803.03735) find
  42. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph attention networks. In ICLR, 2018. find
  43. P. Veličković, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjeml. Deep graph infomax. In ICLR, 2019. find
  44. O. Vinyals, S. Bengio, and M. Kudlur. Order matters: Sequence to sequence for sets. In ICLR, 2016. find
  45. M. Wang, L. Yu, A. Gan, D. Zheng, Y. Gai, Z. Ye, M. Li, J. Zhou, Q. Huang, J. Zhao, H. Lin, C. Ma, D. Deng, Q. Guo, H. Zhang, J. Li, A. J. Smola, and Z. Zhang. Deep graph library. http: //dgl.ai, 2018a. find
  46. Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph CNN for learning on point clouds. CoRR, abs/1801.07829, 2018b. — [arXiv:1801.07829](https://arxiv.org/abs/1801.07829) find
  47. F. Wu, T. Zhang, A. H. de Souza Jr., C. Fifty, T. Yu, and K. Q. Weinberger. Simplifying graph convolutional networks. CoRR, abs/1902.07153, 2019. — [arXiv:1902.07153](https://arxiv.org/abs/1902.07153) find
  48. Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3D ShapeNets: A deep representation for volumetric shapes. In CVPR, 2015. find
  49. K. Xu, C. Li, Y. Tian, T. Sonobe, K. Kawarabayashi, and S. Jegelka. Representation learning on graphs with jumping knowledge networks. In ICML, 2018. find
  50. K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In ICLR, 2019. find our card
  51. R. Ying, J. You, C. Morris, X. Ren, W. Hamilton, and J. Leskovec. Hierarchical graph representation learning with differentiable pooling. In NeurIPS, 2018. find
  52. M. Zhang, Z. Cui, M. Neumann, and Y. Chen. An end-to-end deep learning architecture for graph classification. In AAAI, 2018. find
  53. Coalesced GS Non-coalesced GS Forward Backward Seconds for 1 000 runs 1.5 0.5 8 16 32 Average degree of 10 000 nodes find