Elamur Research⌕
☰
home / papers / 2025 / Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering

Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering

Paper · 2025 · Yuan Sui, Yufei He, Zifeng Ding et al.

Summary

Most benchmarks for LLMs combined with knowledge graphs use closed-ended questions with short entity answers. In that setting, accuracy cannot tell a retrieval failure apart from a fabricated answer, so hallucination goes unmeasured. The authors built OKGQA, a benchmark of open-ended questions that need paragraph-length answers grounded in DBpedia. The questions span descriptive, explanatory, predictive, comparative and critical types. They were generated by an LLM from templates and refined in a loop that aligned automatic quality scores with human ratings.

Each question comes with a 2-hop DBpedia subgraph, pruned with Personalized PageRank. A companion variant, OKGQA-P, deliberately corrupts these graphs to test how robust models are. Factuality is checked with FActScore and SAFE, and overall answer quality with G-Eval.

The main findings:

  • Adding KG evidence generally lowers factual errors.
  • Purely internal reasoning, such as Chain-of-Thought or Self-Consistency, does not reliably do so.
  • Subgraph-based retrieval is the strongest and most robust option.

This matters for anyone grounding generation in imperfect real-world graphs, because the gains hold even when the KG is moderately noisy.

Key points

  • OKGQA contains 850 queries (637 simple, 213 complex) covering 816 unique DBpedia entities. Code and data are available at github.com/Y-Sui/OKGQA, and a multilingual extension is described in an appendix.
  • PPR pruning cuts the average subgraph from about 7,171 to 48 nodes and from roughly 349K to 2.5K tokens.
  • OKGQA-P applies four perturbations: edge deletion, edge rewiring, relation replacement and relation swapping. Deviation from the original graph is measured with ATS (semantic) and with SC2D and SD2 (structural).
  • Triplet- and path-based retrieval falls below the CoT baseline once about 50% of edges are perturbed. Under relation replacement, the KG keeps an advantage only below roughly 40% perturbation. Subgraph retrieval stays ahead at all perturbation levels.
  • Stated limitations: the benchmark relies only on DBpedia, which may not transfer to specialized domains, and it assumes a static KG.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🎓National University of SingaporeSingapore · Singapore180 works · 34,304 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksQuestion answering · 441 worksKnowledge graphs & ontologies · 192 works

Cite

Cite as Yuan Sui, Yufei He, Zifeng Ding and Bryan Hooi. 2025. Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.622

@inproceedings{sui2025can,
  title = {{Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering}},
  author = {Yuan Sui, Yufei He, Zifeng Ding and Bryan Hooi},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.622},
  url = {https://doi.org/10.18653/v1/2025.acl-long.622}
}

References

58 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Garima Agrawal, Tharindu Kumarage, Zeyad Alghami, and Huan Liu. 2023. Can knowledge graphs reduce hallucinations in llms?: A survey. ArXiv preprint, abs/2311.07914. find
  2. Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023a. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pages 78–106, Toronto, Canada. Association for…
  3. Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023b. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pages 78–106, Toronto, Canada. Association for…
  4. Bahman Bahmani, Abdur Chowdhury, and Ashish Goel. 2010. Fast incremental and personalized pagerank. ArXiv preprint, abs/1006.2880. — [arXiv:1006.2880](https://arxiv.org/abs/1006.2880) find
  5. Egon Balas. 2010. The prize collecting traveling salesman problem. ArXiv preprint, abs/10.1002. find
  6. Yulin Chen, Haoran Li, Yuan Sui, Yufei He, Yue Liu, Yangqiu Song, and Bryan Hooi. 2025. Can indirect prompt injection attacks be detected and removed? ArXiv preprint, abs/2502.16580.
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. ArXiv preprint, abs/2407.21783.
  8. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. ArXiv preprint, abs/2404.16130. find
  9. Giorgio Fagiolo. 2007. Clustering in complex directed networks. Physical Review E, 76(2):026107. find
  10. Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. 2024. Does fine-tuning llms on new knowledge encourage hallucinations? ArXiv preprint, abs/2405.05904. find
  11. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 20 2. A survey on llm-as-a-judge. ArXiv preprint, abs/ 2411. find
  12. Hangfeng He, Hongming Zhang, and Dan Roth. 2023a. Rethinking with retrieval: Faithful large language model inference. ArXiv preprint, abs/2301.00303. find
  13. Kai He, Rui Mao, Qika Lin, Yucheng Ruan, Xiang Lan, Mengling Feng, and Erik Cambria. 2023b. A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics. ArXiv preprint, abs/2310.05694.
  14. Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. ArXiv preprint, abs/2402.07630.
  15. Yufei He, Yuan Sui, Xiaoxin He, and Bryan Hooi. 20 2. Unigraph: Learning a unified cross-domain foundation model for text-attributed graphs. ArXiv preprint, abs/ 2402. find
  16. Yufei He, Yuan Sui, Xiaoxin He, Yue Liu, Yifei Sun, and Bryan Hooi. 2025. Unigraph2: Learning a unified embedding space to bind multimodal graphs. In Proceedings of the ACM on Web Conference 2025, WWW ’25, page 1759–1770, New York, NY, USA. Association for Computing Machinery. find
  17. Linmei Hu, Zeyi Liu, Ziwang Zhao, Lei Hou, Liqiang Nie, and Juanzi Li. 2023. A survey of knowledge enhanced pre-trained language models. IEEE Transactions on Knowledge and Data Engineering. find
  18. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ArXiv preprint, abs/2311.05232.
  19. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  20. Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. 2023b. Structgpt: A general framework for large language model to reason over structured data. ArXiv preprint, abs/2305.09645. find
  21. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. ArXiv preprint, abs/2009.13081.
  22. Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Leong, Jia Qing Tan, Enamul Hoque, and Shafiq Joty. 2022. OpenCQA: Open-ended question answering with charts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11817–11837, Abu Dhabi, United Arab Emirates.… find
  23. Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. 2023. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. ArXiv preprint, abs/2305.14045.
  24. J Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33 1:159–74. find
  25. Jukka-Pekka Onnela, Jari Saramäki, János Kertész, and Kimmo Kaski. 2005. Intensity and coherence of motifs in weighted complex networks. Physical Review E, 71(6):065103. find
  26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  27. Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2024. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering.
  28. Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale Fung, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Factuality enhanced language models for open-ended text generation. ArXiv preprint, abs/2206.04624. find
  29. Haritz Puerto, Gözde Gül Şahin, and Iryna Gurevych. 2021. Metaqa: Combining expert agents for multi-skill question answering. ArXiv preprint, abs/2112.01922. find
  30. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 20 2. Llms-as-judges: A comprehensive survey on llm-based evaluation methods. ArXiv preprint, abs/ 2412.
  31. Han Qiu, Jiaxing Huang, Peng Gao, Qin Qi, Xiaoqin Zhang, Ling Shao, and Shijian Lu. 2024. Long-halqa: Long-context hallucination evaluation for multimodal large language models. ArXiv preprint, abs/2410.09962. — [arXiv:2410.09962](https://arxiv.org/abs/2410.09962) find
  32. Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016. Commonsense knowledge base completion. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1445–1455, Berlin, Germany. Association for Computational Linguistics.
  33. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. ArXiv preprint, abs/2303.16634.
  34. Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2023a. Reasoning on graphs: Faithful and interpretable large language model reasoning. ArXiv preprint, abs/2310.01061. find
  35. Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2023b. Reasoning on graphs: Faithful and interpretable large language model reasoning. ArXiv preprint, abs/2310.01061. find
  36. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association fo… find
  37. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural L…
  38. Mrigank Raman, Aaron Chan, Siddhant Agarwal, PeiFeng Wang, Hansen Wang, Sungchul Kim, Ryan Rossi, Handong Zhao, Nedim Lipka, and Xiang Ren. 2020. Learning to deceive knowledge graph augmented models via targeted perturbation. ArXiv preprint, abs/2010.12872. — [arXiv:2010.12872](https://arxiv.org/abs/2010.12872) find
  39. Jari Saramäki, Mikko Kivelä, Jukka-Pekka Onnela, Kimmo Kaski, and Janos Kertesz. 2007. Generalizations of the clustering coefficient to weighted complex networks. Physical Review E, 75(2):027105. find
  40. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36.
  41. Yuan Sui. 2021. Question answering system based on tourism knowledge graph. In Journal of physics: Conference series, volume 1883, page 012064. IOP Publishing. find
  42. Yuan Sui, Shanshan Feng, Huaxiang Zhang, Jian Cao, Liang Hu, and Nengjun Zhu. 2022. Causality-aware enhanced model for multi-hop question answering over knowledge graphs. Knowledge-Based Systems, 250:108943. find
  43. Yuan Sui, Yufei He, Nian Liu, Xiaoxin He, Kun Wang, and Bryan Hooi. 2024. Fidelis: Faithful reasoning in large language model for knowledge graph question answering. ArXiv preprint, abs/2405.13873. — [arXiv:2405.13873](https://arxiv.org/abs/2405.13873) find
  44. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–651, New Orleans, L…
  45. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  46. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. ArXiv preprint, abs/2211.09085.
  47. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, et al. 2024. Gemma 2: Improving open language models at a practical size. ArXiv preprint, abs/2408.00118. of thoughts in large language models. ArXiv… find
  48. Lilian Weng. 2024. Extrinsic hallucinations in llms. lilianweng.github.io. find
  49. Yike Wu, Nan Hu, Guilin Qi, Sheng Bi, Jie Ren, Anhuan Xie, and Wei Song. 2023. Retrieve-rewrite-answer: A kg-to-text enhanced llms framework for knowledge graph question answering. ArXiv preprint, abs/2309.11206. find
  50. Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201–206, Berlin, Ger…
  51. Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. 2023. Why does chatgpt fall short in answering questions faithfully? ArXiv preprint, abs/2304.10513. — [arXiv:2304.10513](https://arxiv.org/abs/2304.10513) find
  52. H. Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022a. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. Annual Meeting of the Association for Computational Linguistics.
  53. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022b. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. ArXiv preprint, abs/2212.10509.
  54. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? a preliminary study. ArXiv preprint, abs/2303.04048.
  55. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. ArXiv preprint, abs/2203.11171.
  56. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  57. Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. 2024. Long-form factuality in large language models. ArXiv preprint, abs/2403.18802.
  58. Yilin Wen, Zifeng Wang, and Jimeng Sun. 2023. Mindmap: Knowledge graph prompting sparks graph find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sui-2025-can-knowledge-graphs-make-large/