Elamur Research⌕
☰
home / papers / 2026 / A Graph Talks, But Who's Listening? Rethinking Evaluations for Graph-Language Models

A Graph Talks, But Who's Listening? Rethinking Evaluations for Graph-Language Models

Paper · 2026 · Soham Petkar, Hari Aakash K, Anirudh Vempati et al.

Summary

Graph-Language Models (GLMs) pair a GNN encoder with an LLM. This paper asks whether the benchmarks used to evaluate them actually test joint reasoning over graph structure and text. The authors re-run standard evaluations on node classification and link prediction, in both supervised and zero-shot settings. They compare GLMs against two unimodal baselines: linear probes on graph encoders, and soft-prompted LLMs that never see the graph. Either baseline frequently matches or beats the GLMs, so these datasets can be solved with a single modality.

To close this gap, they build CLeGR, a synthetic question-answering benchmark based on subway networks. Synthetic graphs avoid contamination from LLM pretraining, and the questions are constructed so that neither text nor structure alone is enough. On CLeGR, GLMs score poorly and land at roughly the same level as soft-prompted LLMs. This parity holds on larger graphs and under zero-shot transfer. The authors conclude that current GLMs do not really exploit their graph encoders, and that the field needs evaluations that genuinely require multimodal reasoning.

Key points

  • On Cora, CiteSeer, PubMed-style citation graphs, Computers, Photo, History and Arxiv, unimodal baselines perform comparably to GLMs such as TEA-GLM, G-Token and GraphPrompter.
  • CLeGR (Compositional Language-Graph Reasoning) contains 500 graphs and 32,000 questions.
  • The questions cover node, edge and subgraph levels. Four kinds of reasoning are assessed: filtering, aggregation, reasoning over paths, and analysis of topology.
  • Backbones include Llama3-8B, Phi3-3.5B and Phi4-14B. The metric is exact-match accuracy, and zero-shot transfer is tested on computer networks.
  • Stated limitations: synthetic data from a single domain, no models above 14B parameters, and exact match may miss correct answers phrased differently.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of VirginiaCharlottesville · United States36 works · 9,730 citations🎓International Institute of Information Technology, HyderabadHyderabad · India38 works · 1,821 citations

Also on the byline: Plaksha University.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Soham Petkar, Hari Aakash K, Anirudh Vempati, Akshit Sinha, Ponnurangam Kumaraguru and Chirag Agarwal. 2026. A Graph Talks, But Who's Listening? Rethinking Evaluations for Graph-Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1624

@inproceedings{petkar2026graph,
  title = {{A Graph Talks, But Who's Listening? Rethinking Evaluations for Graph-Language Models}},
  author = {Soham Petkar, Hari Aakash K, Anirudh Vempati, Akshit Sinha, Ponnurangam Kumaraguru and Chirag Agarwal},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1624},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1624}
}

References

32 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bowen Jin, Wentao Zhang, Yu Zhang, Yu Meng, Xinyang Zhang, Qi Zhu, and Jiawei Han. 2023. Patton: Language model pretraining on text-rich networks. Preprint, arXiv:2305.12268. find
  2. Palaash Agrawal, Shavak Vasania, and Cheston Tan. 2024. Can llms perform structured graph reasoning? Preprint, arXiv:2402.01805. find
  3. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. Preprint, arXiv:1609.02907.
  4. Arvindh Arun, Sumit Kumar, Mojtaba Nayyeri, Bo Xiong, Ponnurangam Kumaraguru, Antonio Vergari, and Steffen Staab. 2025. Semma: A semantic Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. Preprint, arXiv:2104.08691.
  5. Yuhan Li, Zhixun Li, Peisong Wang, Jia Li, Xiangguo Sun, Hong Cheng, and Jeffrey Xu Yu. 2024a. A survey of graph meets large language model: Progress and future directions. Preprint, arXiv:2311.12399. — [arXiv:2311.12399](https://arxiv.org/abs/2311.12399) find
  6. Jianheng Tang, Qifan Zhang, Yuhan Li, Nuo Chen, and Jia Li. 2025. Grapharena: Evaluating and exploring large language models on graph computation. Preprint, arXiv:2407.00379. — [arXiv:2407.00379](https://arxiv.org/abs/2407.00379) find
  7. Yuhan Li, Peisong Wang, Xiao Zhu, Aochuan Chen, Haiyun Jiang, Deng Cai, Victor Wai Kin Chan, and Jia Li. 2024b. Glbench: A comprehensive benchmark for graph with large language models. Preprint, arXiv:2407.07457. — [arXiv:2407.07457](https://arxiv.org/abs/2407.07457) find
  8. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. Preprint, arXiv:1710.10903.
  9. Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. 2024a. One for all: Towards training one graph model for all classification tasks. Preprint, arXiv:2310.00149. — [arXiv:2310.00149](https://arxiv.org/abs/2310.00149) find
  10. Duo Wang, Yuan Zuo, Fengzhi Li, and Junjie Wu. 2024. Llms as zero-shot graph learners: Alignment of gnn representations with llm token embeddings. Preprint, arXiv:2408.14512. — [arXiv:2408.14512](https://arxiv.org/abs/2408.14512) find
  11. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Preprint, arXiv:2304.08485.
  12. Qiming Wu, Zichen Chen, Will Corcoran, Misha Sra, and Ambuj K. Singh. 2025. Grapheval36k: Benchmarking coding and reasoning capabilities of large language models on graph datasets. Preprint, arXiv:2406.16176. find
  13. Zheyuan Liu, Xiaoxin He, Yijun Tian, and Nitesh V. Chawla. 2024b. Can we soft prompt llms for graph learning tasks? In Companion Proceedings of the ACM Web Conference 2024, WWW ’24, page 481–484. ACM. find
  14. Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. 2024a. Benchmark data contamination of large language models: A survey. Preprint, arXiv:2406.04244.
  15. Yuankai Luo, Lei Shi, and Xiao-Ming Wu. 2024. Classic gnns are strong baselines: Reassessing gnns for node classification. Preprint, arXiv:2406.08993. — [arXiv:2406.08993](https://arxiv.org/abs/2406.08993) find
  16. Yuhao Xu, Xinqi Liu, Keyu Duan, Yi Fang, Yu-Neng Chuang, Daochen Zha, and Qiaoyu Tan. 2024b. Graphfm: A comprehensive benchmark for graph foundation model. Preprint, arXiv:2406.08310. find
  17. D. Mack and A. Jefferson. 2018. Clevr graph: A dataset for graph question answering. https://github.com/Octavian-ai/clevr-graph. ↗
  18. Andrew Kachites McCallum and 1 others. 2000. Automating the construction of internet portals with machine learning. In AAAI Spring Symposium on Mining Answers from Textual Data. find
  19. Hao Yan, Chaozhuo Li, Ruosong Long, Chao Yan, Jianan Zhao, Wenwen Zhuang, Jun Yin, Peiyan Zhang, Weihao Han, Hao Sun, Weiwei Deng, Qi Zhang, Lichao Sun, Xing Xie, and Senzhang Wang. 2023. A comprehensive study on text-attributed graphs: Benchmarking and rethinking. In Thirty-seventh Conference on Ne… find
  20. Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mehran Kazemi, Rami Al-Rfou, and Jonathan Halcrow. 2024. Let your graph do the talking: Encoding structured data for llms. Preprint, arXiv:2402.05862. — [arXiv:2402.05862](https://arxiv.org/abs/2402.05862) find
  21. Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. 2023a. Graphformers: Gnn-nested transformers for representation learning on textual graph. Preprint, arXiv:2105.02605. — [arXiv:2105.02605](https://arxiv.org/abs/2105.02605) find
  22. Swarnadeep Saha, Prateek Yadav, Lisa Bauer, and Mohit Bansal. 2021. Explagraphs: An explanation graph generation task for structured commonsense reasoning. Preprint, arXiv:2104.07644. find
  23. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023b. The dawn of lmms: Preliminary explorations with gpt-4v(ision). Preprint, arXiv:2309.17421.
  24. Abulhair Saparov, Srushti Pawar, Shreyas Pimpalgaonkar, Nitish Joshi, Richard Yuanzhe Pang, Vishakh Padmakumar, Seyed Mehran Kazemi, Najoung Kim, and He He. 2025. Transformers struggle to learn to search. Preprint, arXiv:2412.04703. find
  25. Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. 2016. Revisiting semi-supervised learning with graph embeddings. Preprint, arXiv:1603.08861. find
  26. Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008. Collective classification in network data. AI magazine, 29(3):93–106. find
  27. Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. 2019. Pitfalls of graph neural network evaluation. Preprint, arXiv:1811.05868. — [arXiv:1811.05868](https://arxiv.org/abs/1811.05868) find
  28. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. Preprint, arXiv:1809.09600.
  29. Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang… find
  30. Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201–206, Berlin, Ger…
  31. Zike Yuan, Ming Liu, Hui Wang, and Bing Qin. 2025. Gracore: Benchmarking graph comprehension and complex reasoning in large language models. Preprint, arXiv:2407.02936.
  32. Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, and Jian Tang. 2023. Learning on large-scale text-attributed graphs via variational inference. Preprint, arXiv:2210.14709. and combined with the query, then processed by a frozen LLM with soft prompting for fine-tuned, contextua… — [arXiv:2210.14709](https://arxiv.org/abs/2210.14709) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/petkar-2026-graph-talks-but-who-s/