Elamur Research⌕
☰
home / papers / 2023 / GraphGPT: Graph Instruction Tuning for Large Language Models

GraphGPT: Graph Instruction Tuning for Large Language Models

Paper · 2023 · Jiabin Tang, Yuhao Yang, Wei Wei et al.

Summary

Graph neural networks usually still need labels from the target task, even when pre-trained with self-supervised objectives such as contrastive (DGI, GCA) or generative (GraphMAE, S2GAE) learning. This weakens them when labels are scarce, as in cold-start recommendation or traffic forecasting for a new city. The authors ask whether a large language model can learn to read graph structure well enough to generalize to unseen datasets and tasks without downstream graph supervision.

GraphGPT first grounds graph encodings in language space by contrastively matching them with node text. A lightweight projector then feeds graph tokens into the LLM. Tuning has two stages: self-supervised instructions built from a graph-matching task on unlabeled graphs, then task-specific graph instructions. Chain-of-Thought distillation is added to improve step-by-step reasoning under distribution shift.

The authors report that GraphGPT beats existing baselines in both supervised and zero-shot settings and avoids catastrophic forgetting. The work matters because it offers a compact, scalable alternative to describing graphs as long text prompts.

Key points

  • The paper targets three challenges: aligning graph structure with the language space, guiding the LLM to understand that structure, and enabling stepwise reasoning on complex graph tasks.
  • A motivating arXiv paper-classification example (TiM-DNN) compares prompt sizes and predictions:
    • ChatGPT with node text only used 615 tokens and wrongly predicted cs.AR.
    • ChatGPT with the citation neighbourhood written out as text (103 papers) used 4,649 tokens and again predicted cs.AR.
    • GraphGPT, using a <graph> placeholder, needed 750 tokens and correctly predicted cs.LG.
  • The paper was published at SIGIR'24 by authors from the University of Hong Kong and Baidu Inc. Code is at github.com/HKUDS/GraphGPT.
  • The authors name pruning of redundant LLM parameters as future work to shrink the model.
  • The provided excerpt omits the experimental sections, so dataset names and benchmark scores are not given here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢BaiduBeijing · China84 works · 17,681 citations🎓University of Hong KongPok Fu Lam · China118 works · 13,496 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin and Chao Huang. 2023. GraphGPT: Graph Instruction Tuning for Large Language Models. arXiv:2310.13023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.13023 · arXiv 2310.13023

@misc{tang2023graphgpt,
  title = {{GraphGPT: Graph Instruction Tuning for Large Language Models}},
  author = {Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin and Chao Huang},
  year = {2023},
  doi = {10.48550/arxiv.2310.13023},
  eprint = {2310.13023},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.13023}
}

References

67 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, et al. 2022. Constitutional AI: Harmlessness from AI Feedback. CoRR abs/2212.08073 (2022).
  2. Zhikai Chen, Haitao Mao, Hang Li, et al. 2023. Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs. CoRR abs/2307.03393 (2023). — [arXiv:2307.03393](https://arxiv.org/abs/2307.03393) find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT (1). Association for Computational Linguistics, 4171–4186.
  4. Yushun Dong, Ninghao Liu, Brian Jalaian, et al. 2022. EDITS: Modeling and Mitigating Data Bias for Graph Neural Networks. In WWW. ACM, 1259–1269. find
  5. Jiayan Guo, Lun Du, and Hengyu Liu. 2023. GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarking. CoRR abs/2305.15066 (2023).
  6. Zhichun Guo, Kehan Guo, Bozhao Nan, Yijun Tian, Roshni G. Iyer, et al. 2023. Graph-based Molecular Representation Learning. In IJCAI. 6638–6646. find
  7. William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive Representation Learning on Large Graphs. In NeurIPS. 1024–1034.
  8. Xiaoxin He, Xavier Bresson, et al. 2023. Explanations as Features: LLM-Based Features for Text-Attributed Graphs. CoRR abs/2305.19523 (2023).
  9. Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong-Dong Zhang, and Meng Wang. 2020. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In SIGIR. ACM, 639–648. find
  10. Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, et al. 2023. GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner. In WWW. 737–746. find
  11. Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Jie Tang, et al. 2022. Graphmae: Self-supervised masked graph autoencoders. In KDD. 594–604. find
  12. Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, et al. 2020. Open Graph Benchmark: Datasets for Machine Learning on Graphs. In NeurIPS. find
  13. Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. 2020. Gpt-gnn: Generative pre-training of graph neural networks. In KDD. 1857–1867. find
  14. Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020. Heterogeneous Graph Transformer. In WWW. ACM / IW3C2, 2704–2710. find
  15. Yangqin Jiang, Chao Huang, and Lianghao Huang. 2023. Adaptive graph contrastive learning for recommendation. In KDD. 4252–4261. find
  16. Baoyu Jing, Chanyoung Park, and Hanghang Tong. 2021. Hdmi: High-order deep multiplex infomax. In WWW. 2414–2424. find
  17. Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR (Poster). OpenReview.net.
  18. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In NeurIPS.
  19. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, et al. 2023. RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. CoRR abs/2309.00267 (2023). find
  20. Bolian Li, Baoyu Jing, and Hanghang Tong. 2022. Graph communal contrastive learning. In WWW. 1203–1213. find
  21. Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. 2021. Training Graph Neural Networks with 1000 Layers. In ICML. 6437–6449. find
  22. Mingkai Lin, Wenzhong Li, Ding Li, Yizhou Chen, and Sanglu Lu. 2022. Resource-Efficient Training for Large Graph Convolutional Networks with Label-Centric Cumulative Sampling. In WWW. ACM, 1170–1180. find
  23. Haotian Liu, Chunyuan Li, et al. 2023. Visual Instruction Tuning.
  24. Yixin Liu, Ming Jin, Shirui Pan, Chuan Zhou, Yu Zheng, Feng Xia, and S Yu Philip. 2022. Graph self-supervised learning: A survey. TKDE 35, 6 (2022), 5879–5900. find
  25. Yunchao Liu, Yu Wang, Oanh Vu, Rocco Moretti, et al. 2023. Interpretable Chirality-Aware Graph Neural Network for Quantitative Structure Activity Relationship Modeling in Drug Discovery. In AAAI. 14356–14364. find
  26. Zemin Liu, Xingtong Yu, et al. 2023. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In WWW. 417–428. find
  27. Xiaojun Ma, Qin Chen, et al. 2022. Meta-Weight Graph Neural Network: Push the Limits Beyond Global Homophily. In WWW. ACM, 1270–1280. find
  28. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. In EMNLP. 11048–11064.
  29. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, et al. 2022. Training language models to follow instructions with human feedback. In NeurIPS.
  30. Alec Radford, Jong Wook Kim, Chris Hallacy, et al. 2021. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML). PMLR, 8748–8763.
  31. Zezhi Shao et al. 2022. Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting. In KDD. ACM, 1567–1577. find
  32. Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. Distilling Reasoning Capabilities into Smaller Language Models. In ACL. 7059–7073.
  33. Mingchen Sun, Kaixiong Zhou, et al. 2022. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In KDD. 1717–1727. find
  34. Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. 2023. All in One: Multi-Task Prompting for Graph Neural Networks. In KDD. find
  35. Qiaoyu Tan, Ninghao Liu, Xiao Huang, Soo-Hyun Choi, Li Li, Rui Chen, and Xia Hu. 2023. S2GAE: Self-Supervised Graph Autoencoders are Generalizable Learners with Graph Masking. In WSDM. 787–795. find
  36. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. CoRR abs/2302.13971 (2023).
  37. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. CoRR abs/2307.09288 (2023).
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, et al. 2017. Attention is all you need. In NeurIPS, Vol. 30.
  39. Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, et al. 2018. Graph Attention Networks. In ICLR (Poster). OpenReview.net. find
  40. Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, et al. 2019. Deep Graph Infomax. In ICLR (Poster). OpenReview.net. find
  41. Kuansan Wang, Zhihong Shen, et al. 2020. Microsoft Academic Graph: When experts are not enough. Quant. Sci. Stud. 1, 1 (2020), 396–413. find
  42. Xiang Wang, Tinglin Huang, Dingxian Wang, et al. 2021. Learning Intents behind Interactions with Knowledge Graph for Recommendation. In WWW. 878–887. find
  43. Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, et al. 2019. Heterogeneous Graph Attention Network. In WWW. ACM, 2022–2032. find
  44. Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, et al. 2023. How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. CoRR abs/2306.04751 (2023).
  45. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In ACL. 13484–13508. find
  46. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Jeff Dean, William Fedus, et al. 2022. Emergent Abilities of Large Language Models. Trans. Mach. Learn. Res. 2022 (2022).
  47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS.
  48. Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. 2023. LLMRec: Large Language Models with Graph Augmentation for Recommendation. CoRR abs/2311.00423 (2023). — [arXiv:2311.00423](https://arxiv.org/abs/2311.00423) find
  49. Zhihao Wen and Yuan Fang. 2023. Augmenting Low-Resource Text Classification with Graph-Grounded Pre-training and Prompting. In SIGIR. find
  50. Qitian Wu, Chenxiao Yang, et al. 2023. DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion. In ICLR. find
  51. Qitian Wu, Wentao Zhao, et al. 2023. NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification. CoRR abs/2306.08385 (2023). — [arXiv:2306.08385](https://arxiv.org/abs/2306.08385) find
  52. Jun Xia, Lirong Wu, Jintao Chen, et al. 2022. Simgrace: A simple framework for graph contrastive learning without data augmentation. In WWW. 1070–1079. find
  53. Lianghao Xia, Chao Huang, Tao Yu, Ben Kao, et al. 2023. Automated Self-Supervised Learning for Recommendation. In WWW. 992–1002. find
  54. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, et al. 2023. Baichuan 2: Open Large-scale Language Models. CoRR abs/2309.10305 (2023).
  55. Chenxiao Yang, Qitian Wu, and Junchi Yan. 2022. Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks. In NeurIPS. find
  56. Haoran Yang, Hongxu Chen, Shirui Pan, Lin Li, Philip S Yu, and Guandong Xu. 2022. Dual space graph contrastive learning. In WWW. 1238–1247. find
  57. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. CoRR abs/2305.10601 (2023). find
  58. Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. 2021. Graph contrastive learning automated. In ICML. PMLR, 12121–12132. find
  59. Yuning You, Tianlong Chen, Yongduo Sui, et al. 2020. Graph contrastive learning with augmentations. In NeurIPS, Vol. 33. 5812–5823. find
  60. Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J. Kim. 2019. Graph Transformer Networks. In NeurIPS. 11960–11970. find
  61. Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. 2019. Graph transformer networks. In NeurIPS, Vol. 32. find
  62. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, et al. 2023. GLM-130B: An Open Bilingual Pre-trained Model. In ICLR.
  63. Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. 2022. Graph-less Neural Networks: Teaching Old MLPs New Tricks Via Distillation. In ICLR. find
  64. Wen Zhang, Yushan Zhu, Mingyang Chen, et al. 2023. Structure Pretraining and Prompt Tuning for Knowledge Graph Transfer. In WWW. 2581–2590. find
  65. Yanfu Zhang et al. 2022. Robust Self-Supervised Structural Graph Neural Network for Social Network Prediction. In WWW. ACM, 1352–1361. find
  66. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv preprint arXiv:2304.10592 (2023).
  67. Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2021. Graph contrastive learning with adaptive augmentation. In WWW. 2069–2080. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2023-graphgpt-graph-instruction-tuning-large/