Elamur Research⌕
☰
home / papers / 2025 / Taming Language Models for Text-attributed Graph Learning with Decoupled Aggregation

Taming Language Models for Text-attributed Graph Learning with Decoupled Aggregation

Paper · 2025 · Chuang Zhou, Zhu Wang, Shengyuan Chen et al.

Summary

Text-attributed graphs (TAGs) are graphs whose nodes carry text and whose edges encode relations. Learning on them usually follows a two-stage cascade: a language model produces frozen node embeddings, and a GNN is then trained on those embeddings. The authors argue that this split ignores how text and topology interact, while full end-to-end LM+GNN training is too expensive. Their framework, SKETCH (Semantic Knowledge and Structure Enrichment), drops the GNN entirely and moves node aggregation into the input of a long-context LM. For each anchor node it gathers two kinds of extra text:

  • Semantic context: similar nodes from anywhere in the graph, found with sentence-transformer embeddings and FAISS cosine search.
  • Structural context: multi-hop neighbors ranked by Jaccard overlap of their neighborhoods, estimated cheaply with MinHash.

The enriched, weighted context is used to fine-tune the LM for node classification. The authors report that SKETCH outperforms state-of-the-art TAG methods while using less time and memory. This suggests a lightweight, GNN-free way to make text classifiers graph-aware.

Key points

  • Semantic aggregation: node texts are embedded with a sentence-transformer and indexed in FAISS. The authors state that retrieval over hundreds of thousands of nodes takes only a few minutes.
  • Structural aggregation: relatedness is defined as the Jaccard coefficient over k-hop neighbor sets. MinHash signatures and matrix intersection approximate this coefficient, so cost does not grow exponentially as more hops are added.
  • Hyperparameter analysis: context lengths from 1k to 8k tokens and structural ranking weights from 0 to 2.0 were tested. Very long inputs can add noise and force smaller batch sizes, while giving more weight to structure-based ranking generally helped. The authors describe the model as not overly sensitive to these settings.
  • Running example: a citation network in which an anchor paper on Kalman Filters, labeled DATABASE, is classified correctly thanks to retrieved related abstracts.
  • Limitations: the number of retrieved nodes is capped by the LM token limit (8k max), and larger models increase inference time. Planned future work covers long-sequence models such as T5 and quantization.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong Polytechnic UniversityHong Kong · China84 works · 4,956 citations🎓Stevens Institute of TechnologyHoboken · United States10 works · 95 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Chuang Zhou, Zhu Wang, Shengyuan Chen, Jiahe Du, Qiyuan Zheng, Zhaozhuo Xu and Xiao Huang. 2025. Taming Language Models for Text-attributed Graph Learning with Decoupled Aggregation. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.173

@inproceedings{zhou2025taming,
  title = {{Taming Language Models for Text-attributed Graph Learning with Decoupled Aggregation}},
  author = {Chuang Zhou, Zhu Wang, Shengyuan Chen, Jiahe Du, Qiyuan Zheng, Zhaozhuo Xu and Xiao Huang},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.173},
  url = {https://doi.org/10.18653/v1/2025.acl-long.173}
}

References

58 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexandr Andoni, Piotr Indyk, Huy L Nguyen, and Ilya Razenshteyn. 2014. Beyond locality-sensitive hashing. In Proceedings of the twenty-fifth annual ACM-SIAM symposium on Discrete algorithms, pages 1018–1028. SIAM. https://www.aclweb.org/portal/content/ acl-code-ethics Alexandr Andoni, Thijs Laarhov… ↗
  2. Alexandr Andoni and Ilya Razenshteyn. 2015. Optimal data-dependent hashing for approximate near neighbors. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing (STOC), pages 793–801. find
  3. Shaked Brody, Uri Alon, and Eran Yahav. 2022. How attentive are graph attention networks? In International Conference on Learning Representations. find
  4. J Lawrence Carter and Mark N Wegman. 1977. Universal classes of hash functions. In Proceedings of the ninth annual ACM symposium on Theory of computing, pages 106–112. find
  5. Moses S Charikar. 2002. Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing, pages 380–388. find
  6. Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. 2024a. Llaga: Large language and graph assistant. arXiv preprint arXiv:2402.08170. — [arXiv:2402.08170](https://arxiv.org/abs/2402.08170) find
  7. Shengyuan Chen, Qinggang Zhang, Junnan Dong, Wen Hua, Jiannong Cao, and Xiao Huang. 2024b. Neuro-symbolic entity alignment via variational inference. arXiv preprint arXiv:2410.04153. — [arXiv:2410.04153](https://arxiv.org/abs/2410.04153) find
  8. Shengyuan Chen, Qinggang Zhang, Junnan Dong, Wen Hua, Qing Li, and Xiao Huang. 2024c. Entity alignment with noisy annotations from large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  9. Shengyuan Chen, Qinggang Zhang, Junnan Dong, Wen Hua, Qing Li, and Xiao Huang. 2024d. Entity alignment with noisy annotations from large language models. arXiv preprint arXiv:2405.16806. — [arXiv:2405.16806](https://arxiv.org/abs/2405.16806) find
  10. Zhikai Chen, Haitao Mao, Hongzhi Wen, Haoyu Han, Wei Jin, Haiyang Zhang, Hui Liu, and Jiliang Tang. 2024e. Label-free Node Classification on Graphs with Large Language Models (LLMS). Preprint, arXiv:2310.04668. — [arXiv:2310.04668](https://arxiv.org/abs/2310.04668) find
  11. Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S Mirrokni. 2004. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational geometry, pages 253–262. find
  12. Junnan Dong, Qinggang Zhang, Chuang Zhou, Hao Chen, Daochen Zha, and Xiao Huang. 2024. Cost-efficient knowledge-based question answering with large language models. arXiv preprint arXiv:2405.17337. — [arXiv:2405.17337](https://arxiv.org/abs/2405.17337) find
  13. Keyu Duan, Qian Liu, Tat-Seng Chua, Shuicheng Yan, Wei Tsang Ooi, Qizhe Xie, and Junxian He. 2023. Simteg: A frustratingly simple approach improves textual graph learning. arXiv preprint arXiv:2308.02565. — [arXiv:2308.02565](https://arxiv.org/abs/2308.02565) find
  14. Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat-Seng Chua. 2025a. Alphaedit: Null-space constrained knowledge editing for language models. ICLR. find
  15. Junfeng Fang, Yukai Wang, Ruipeng Wang, Zijun Yao, Kun Wang, An Zhang, Xiang Wang, and Tat-Seng Chua. 2025b. Safemlrm: Demystifying safety in multi-modal large reasoning models. arXiv preprint arXiv:2504.08813. — [arXiv:2504.08813](https://arxiv.org/abs/2504.08813) find
  16. Yujie Feng, Xujia Wang, Zexin Lu, Shenghong Fu, Guangyuan Shi, Yongxin Xu, Yasha Wang, Philip S Yu, Xu Chu, and Xiao-Ming Wu. 2025. Recurrent knowledge identification and fusion for language model continual learning. arXiv preprint arXiv:2502.17510.
  17. Luciano Floridi and Massimo Chiriatti. 2020. GPT-3: Its Nature, Scope, Limits, and Consequences. Minds and Machines, 30(4):681–694. find
  18. Zirui Guo, Lianghao Xia, Yanhua Yu, Yuling Wang, Zixuan Yang, Wei Wei, Liang Pang, Tat-Seng Chua, and Chao Huang. 2024. GraphEdit: Large Language Models for Graph Structure Learning. Preprint, arXiv:2402.15183. — [arXiv:2402.15183](https://arxiv.org/abs/2402.15183) find
  19. Conference on Machine Learning, pages 9377–9409. PMLR. find
  20. Piotr Indyk and Rajeev Motwani. 1998. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Proceedings of the Thirtieth Annual ACM Symposium on the Theory of Computing (STOC), pages 604–613, Dallas, TX. find
  21. Houcheng Jiang, Junfeng Fang, Ningyu Zhang, Guojun Ma, Mingyang Wan, Xiang Wang, Xiangnan He, and Tat-seng Chua. 2025. Anyedit: Edit any knowledge encoded in language models. ICML. find
  22. Bowen Jin, Yu Zhang, Yu Meng, and Jiawei Han. 2023. Edgeformers: Graph-empowered transformers for representation learning on textual-edge networks. In The Eleventh International Conference on Learning Representations. find
  23. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations.
  24. Rui Li, Jiwei Li, Jiawei Han, and Guoyin Wang. 2024a. Similarity-based Neighbor Selection for Graph LLMs. Preprint, arXiv:2402.03720. — [arXiv:2402.03720](https://arxiv.org/abs/2402.03720) find
  25. Yuhan Li, Zhixun Li, Peisong Wang, Jia Li, Xiangguo Sun, Hong Cheng, and Jeffrey Xu Yu. 2024b. A Survey of Graph Meets Large Language Model: Progress and Future Directions. Preprint, arXiv:2311.12399. — [arXiv:2311.12399](https://arxiv.org/abs/2311.12399) find
  26. William L. Hamilton, Rex Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, page 1025–1035.
  27. Yuhan Li, Peisong Wang, Zhixun Li, Jeffrey Xu Yu, and Jia Li. 2024c. Zerog: Investigating cross-dataset zero-shot transferability in graphs. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1725–1735. find
  28. Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web, page 507–517. find
  29. Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. 2023a. One for All: Towards Training One Graph Model for All Classification Tasks. Preprint, arXiv:2310.00149. — [arXiv:2310.00149](https://arxiv.org/abs/2310.00149) find
  30. Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, and Bryan Hooi. 2024. Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning. Preprint, arXiv:2305.19523. find
  31. Zipeng Liu, Likang Wu, Ming He, Zhong Guan, Hongke Zhao, and Nan Feng. 2024. Dr.E Bridges Graphs with Large Language Models through Words. Preprint, arXiv:2406.15504. — [arXiv:2406.15504](https://arxiv.org/abs/2406.15504) find
  32. Zijin Hong, Hao Wu, Su Dong, Junnan Dong, Yilin Xiao, Yujing Zhang, Zhu Wang, Feiran Huang, Linyi Li, Hongxia Yang, et al. 2025. Benchmarking large language models via random variables. arXiv preprint arXiv:2501.11790. — [arXiv:2501.11790](https://arxiv.org/abs/2501.11790) find
  33. Zirui Liu, Chen Shengyuan, Kaixiong Zhou, Daochen Zha, Xiao Huang, and Xia Hu. 2023b. Rsc: accelerate graph neural networks training via randomized sparse computations. In International Conference on Machine Learning, pages 21951–21968. PMLR. find
  34. Shengxiang Hu, Guobing Zou, Song Yang, Yanglan Gan, Bofeng Zhang, and Yixin Chen. 2024. Large Language Model Meets Graph Neural Network in Knowledge Distillation. In KDD2024. arXiv. find
  35. Zihan Luo, Xiran Song, Hong Huang, Jianxun Lian, Chenhao Zhang, Jinqi Jiang, and Xing Xie. 2024. GraphInstruct: Empowering Large Language Models with Graph Understanding and Reasoning Capability. Preprint, arXiv:2403.04483. — [arXiv:2403.04483](https://arxiv.org/abs/2403.04483) find
  36. Zhongyu Huang, Yingheng Wang, Chaozhuo Li, and Huiguang He. 2022. Going deeper into permutation-sensitive graph neural networks. In International Giannis Nikolentzos, Antoine Tixier, and Michalis Vazirgiannis. 2020. Message passing attention networks for document understanding. In Proceedings of the… find
  37. Bo Pan, Zheng Zhang, Yifei Zhang, Yuntong Hu, and Liang Zhao. 2024. Distilling Large Language Models for Text-Attributed Graph Learning. Preprint, arXiv:2402.12022. — [arXiv:2402.12022](https://arxiv.org/abs/2402.12022) find
  38. Hyunjin Seo, Taewon Kim, June Yong Yang, and Eunho Yang. 2024. Unleashing the Potential of Text-attributed Graphs: Automatic Relation Decomposition via Large Language Models. Preprint, arXiv:2405.18581. — [arXiv:2405.18581](https://arxiv.org/abs/2405.18581) find
  39. Chen Shengyuan, Yunfeng Cai, Huang Fang, Xiao Huang, and Mingming Sun. 2024. Differentiable neuro-symbolic reasoning on large-scale knowledge graphs. Advances in Neural Information Processing Systems, 36. find
  40. Qinggang Zhang, Shengyuan Chen, Yuanchen Bei, Zheng Yuan, Huachi Zhou, Zijin Hong, Junnan Dong, Hao Chen, Yi Chang, and Xiao Huang. 2025. A survey of graph retrieval-augmented generation for customized large language models. arXiv preprint arXiv:2501.13958.
  41. Qinggang Zhang, Junnan Dong, Hao Chen, Wentao Li, Feiran Huang, and Xiao Huang. 2024a. Structure guided large language model for sql generation. arXiv preprint arXiv:2402.13284. — [arXiv:2402.13284](https://arxiv.org/abs/2402.13284) find
  42. Qinggang Zhang, Junnan Dong, Hao Chen, Daochen Zha, Zailiang Yu, and Xiao Huang. 2024b. Knowgpt: Knowledge graph based prompting for large language models. Advances in Neural Information Processing Systems, 37:6052–6080. find
  43. Yanchao Tan, Zihao Zhou, Hang Lv, Weiming Liu, and Carl Yang. WalkLM: A Uniform Language Model Fine-tuning Framework for Attributed Graph Embedding. find
  44. Qinggang Zhang, Keyu Duan, Junnan Dong, Pai Zheng, and Xiao Huang. 2024c. Logical reasoning with relation network for inductive knowledge graph completion. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4268–4277. find
  45. Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. 2024. GraphGPT: Graph Instruction Tuning for Large Language Models. Preprint, arXiv:2310.13023. find
  46. Yin Zhang, Rong Jin, and Zhi-Hua Zhou. 2010. Understanding bag-of-words model: a statistical framework. International journal of machine learning and cybernetics, 1:43–52. find
  47. Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008. Arnetminer: Extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 990–998. find
  48. Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, and Jian Tang. 2023. Learning on large-scale text-attributed graphs via variational inference. In The Eleventh International Conference on Learning Representations. find
  49. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  50. Chuang Zhou, Junnan Dong, Xiao Huang, Zirui Liu, Kaixiong Zhou, and Zhaozhuo Xu. 2024. Quest: Efficient extreme multi-label text classification with large language models on commodity hardware. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3929–3940. find
  51. Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. 2019. Simplifying graph convolutional networks. In International conference on machine learning, pages 6861–6871. find
  52. Chuang Zhou, Jiahe Du, Huachi Zhou, Hao Chen, Feiran Huang, and Xiao Huang. 2025a. Text-attributed graph learning with coupled augmentations. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10865–10876.
  53. Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. 2021. Graphform-ers: Gnn-nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems, 34:28798–28810. find
  54. Huachi Zhou, Hao Chen, Junnan Dong, Daochen Zha, Chuang Zhou, and Xiao Huang. 2023. Adaptive popularity debiasing aggregator for graph collaborative filtering. In Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval, pages 7–17. find
  55. Yang Yao, Xin Wang, Zeyang Zhang, Yijian Qin, Ziwei Zhang, Xu Chu, Yuekui Yang, Wenwu Zhu, and Hong Mei. 2024. Exploring the Potential of Large Language Models in Graph Generation. Preprint, arXiv:2403.14358. — [arXiv:2403.14358](https://arxiv.org/abs/2403.14358) find
  56. Huachi Zhou, Jiahe Du, Chuang Zhou, Chang Yang, Yilin Xiao, Yuxuan Xie, and Xiao Huang. 2025b. Each graph is a new language: Graph learning with llms. arXiv e-prints, pages arXiv–2501. find
  57. Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, Yongfeng Zhang, et al. 2023. Natural language is all a graph needs. arXiv preprint arXiv:2308.07134, 4(5):7. find
  58. Yun Zhu, Yaoke Wang, Haizhou Shi, and Siliang Tang. 2024. Efficient Tuning and Inference for Large Language Models on Textual Graphs. Preprint, arXiv:2401.15569. — [arXiv:2401.15569](https://arxiv.org/abs/2401.15569) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2025-taming-language-models-text-attributed/