Elamur Research⌕
☰
home / papers / 2026 / What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledge

What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledge

Paper · 2026 · Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang et al.

Summary

KG-RAG systems are meant to combine LLM reasoning with structured knowledge graph evidence. Standard benchmarks such as WebQSP, CWQ and GrailQA usually include a triple that states the answer directly, so a correct response may reflect lookup rather than inference. Evaluation is also inconsistent: "accuracy" can mean exact match or loose substring matching, and some codebases compute Hits@1 without any ranking.

The authors propose a general recipe for building benchmarks that test reasoning when knowledge is missing, and they use it to create BRINK. They mine Horn rules with AMIE3, delete the triple that would answer each question directly, and keep the rule body facts so the answer can still be inferred. Questions are generated from these rule groundings, and answer frequencies are balanced through downsampling.

Experiments on data derived from Family, FB15k-237 and Wikidata5m show that most KG-RAG methods fail once direct evidence is removed. Trained methods hold up better. Results also depend heavily on whether entities have textual labels. Together this suggests models rely on memorized knowledge rather than reasoning over graph structure, which motivates better retrieval of alternative paths.

Key points

  • AMIE3 mines rules of length 4 or less, keeping only those that reach a minimum confidence of 0.3 and a minimum head coverage of 0.1. Questions are built only from removed triples that the remaining graph can still support.
  • Earlier ways of simulating incompleteness, such as random deletion or cutting shortest paths, can leave questions unanswerable. BRINK guarantees that supporting evidence remains for every question.
  • Training-based systems (RoG, GNN-RAG) cope with incompleteness better than non-trained ones. However, they show signs of overfitting to particular relation patterns.
  • Freebase IDs (e.g., /m/02mjmr) give F1 results almost identical to random private IDs, while textual labels raise performance considerably. This indicates that LLMs treat IDs as opaque tokens.
  • The case studies identify two failure types. One is retrieval failure, where spurious paths lead to hallucinated answers. The other is reasoning failure, where the correct fact is retrieved but distracting context produces a wrong answer such as "Marriage".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations🎓University of StuttgartStuttgart · Germany91 works · 46,376 citations🎓University of ManchesterManchester · United Kingdom87 works · 44,938 citations🎓University of OsloOslo · Norway51 works · 4,155 citations🎓University of SouthamptonSouthampton · United Kingdom10 works · 2,495 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksKnowledge graphs & ontologies · 192 works

Cite

Cite as Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang, Hongkuan Zhou, Yuan He, Jiaoyan Chen, Steffen Staab and Evgeny Kharlamov. 2026. What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledge. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.114

@inproceedings{zhou2026what,
  title = {{What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledge}},
  author = {Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang, Hongkuan Zhou, Yuan He, Jiaoyan Chen, Steffen Staab and Evgeny Kharlamov},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.114},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.114}
}

References

48 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinheon Baek, Alham Fikri Aji, Jens Lehmann, and Sung Ju Hwang. 2023. Direct fact retrieval from knowledge graphs without entity linking. arXiv preprint arXiv:2305.12416. find
  2. Russa Biswas, Lucie-Aimée Kaffee, Michael Cochez, Stefania Dumbrava, Theis E Jendal, Matteo Lissandrini, Vanessa Lopez, Eneldo Loza Mencía, Heiko Paulheim, Harald Sack, and 1 others. 2023. Knowledge graph embeddings: open challenges and opportunities. Transactions on Graph Data and Knowledge, 1(1):4… find
  3. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cas… find
  4. Lihu Chen, Simon Razniewski, and Gerhard Weikum. 2023. Knowledge base completion for long-tail entities. In Proceedings of the First Workshop on Matching From Unstructured and Structured Data (MATCHING 2023), pages 99–108. find
  5. Liyi Chen, Panrong Tong, Zhongming Jin, Ying Sun, Jieping Ye, and Hui Xiong. 2024. Plan-on-graph: Self-correcting adaptive planning of large language model on knowledge graphs. In Proceedings of the 38th Conference on Neural Information Processing Systems. find
  6. Fengxiang Cheng, Haoxuan Li, Fenrong Liu, Robert van Rooij, Kun Zhang, and Zhouchen Lin. 2025. Empowering llms with logical reasoning: A comprehensive survey. arXiv preprint arXiv:2502.15652. find
  7. Xin Luna Dong, Evgeniy Gabrilovich, Geremy Heitz, Wiko Horn, Kevin Murphy, Shaohua Sun, and Wei Zhang. 2014. From data fusion to knowledge fusion. Proceedings of the VLDB Endowment, 7(10). find
  8. Jinyuan Fang, Zaiqiao Meng, and Craig Macdonald. 2024. Reano: Optimising retrieval-augmented reader models through knowledge graph generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2094–2112. find
  9. Luis Galárraga, Christina Teflioudi, Katja Hose, and Fabian M Suchanek. 2015. Fast rule mining in ontological knowledge bases with amie+. The VLDB Journal, 24(6):707–730. find
  10. Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond iid: three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021, pages 3477–3488. ACM. find
  11. Haoyu Han, Yu Wang, Harry Shomer, Kai Guo, Jiayuan Ding, Yongjia Lei, Mahantesh Halappanavar, Ryan A Rossi, Subhabrata Mukherjee, Xianfeng Tang, and 1 others. 2024. Retrieval-augmented generation with graphs (graphrag). arXiv preprint arXiv:2501.00309. find
  12. Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. Advances in Neural Information Processing Systems, 37:132876–132907.
  13. Yuan He, Bailan He, Zifeng Ding, Alisia Lupidi, Yuqicheng Zhu, Shuo Chen, Caiqi Zhang, Jiaoyan Chen, Yunpu Ma, Volker Tresp, and 1 others. 2025. find
  14. Supposedly equivalent facts that aren’t? entity frequency in pre-training induces asymmetry in llms. arXiv preprint arXiv:2503.22362. — [arXiv:2503.22362](https://arxiv.org/abs/2503.22362) find
  15. Alfred Horn. 1951. On sentences which are true of direct unions of algebras1. The Journal of Symbolic Logic, 16(1):14–21. find
  16. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In EACL, pages 874– 880. Association for Computational Linguistics.
  17. Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Structgpt: A general framework for large language model to reason over structured data. arXiv preprint arXiv:2305.09645. find
  18. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In ICLR. OpenReview.net. find
  19. Jonathan Lajus, Luis Galárraga, and Fabian Suchanek. 2020. Fast and exact rule mining with amie 3. In The Semantic Web: 17th International Conference, ESWC 2020, Heraklion, Crete, Greece, May 31–June 4, 2020, Proceedings 17, pages 36–52. Springer. find
  20. Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. A survey on complex knowledge base question answering: Methods, challenges and solutions. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 4483–4491. International J… find
  21. Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2024. Reasoning on graphs: Faithful and interpretable large language model reasoning. In ICLR. OpenReview.net. find
  22. Costas Mavromatis and George Karypis. 2024. Gnn-rag: Graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.20139. find
  23. Aisha Mohamed, Shameem Parambath, Zoi Kaoudi, and Ashraf Aboulnaga. 2020. Popularity agnostic evaluation of knowledge graph embeddings. In Conference on Uncertainty in Artificial Intelligence, pages 1059– 1068. PMLR. find
  24. OpenAI. 2024. Chatgpt(3.5)[large language model]. https://chat.openai.com. ↗
  25. Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. arXiv preprint arXiv:2404.15522.
  26. Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. 2024. Graph retrieval-augmented generation: A survey. arXiv preprint arXiv:2408.08921.
  27. Nico Potyka, Yuqicheng Zhu, Yunjie He, Evgeny Kharlamov, and Steffen Staab. 2024. Robust knowledge extraction from large language models using social choice theory. In Proceedings of the 23rd International Conference on Autonomous Agents and Multi-agent Systems, pages 1593–1601. find
  28. Yunqi Qiu, Yuanzhuo Wang, Xiaolong Jin, and Kun Zhang. 2020. Stepwise reasoning for multi-relation question answering over knowledge graph with weak supervision. In Proceedings of the 13th international conference on web search and data mining, pages 474–482. find
  29. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Trans. Assoc. Comput. Linguistics, 11:1316–1331.
  30. Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. 2019. Drum: End-to-end differentiable rule mining on knowledge graphs. Advances in neural information processing systems, 32. find
  31. Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M Ni, Heung-Yeung Shum, and Jian Guo. 2023. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. arXiv preprint arXiv:2307.07697. find
  32. Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, Heung-Yeung Shum, and Jian Guo. 2024. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. In ICLR. OpenReview.net. find
  33. Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. 2020. Proofwriter: Generating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012.13048.
  34. Alon Talmor and Jonathan Berant. 2018a. The web as a knowledge-base for answering complex questions. In NAACL-HLT, pages 641–651. Association for Computational Linguistics.
  35. Alon Talmor and Jonathan Berant. 2018b. The web as a knowledge-base for answering complex questions. arXiv preprint arXiv:1803.06643.
  36. Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. 2021. Diagnosing the first-order logical reasoning ability through logicnli. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3738–3747.
  37. Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd workshop on continuous vector space models and their compositionality, pages 57–66.
  38. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85.
  39. Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176–194. find
  40. Yao Xu, Shizhu He, Jiabei Chen, Zihao Wang, Yangqiu Song, Hanghang Tong, Guang Liu, Kang Liu, and Jun Zhao. 2024. Generate-on-graph: Treat llm as both agent and kg in incomplete knowledge graph question answering. arXiv preprint arXiv:2404.14741. — [arXiv:2404.14741](https://arxiv.org/abs/2404.14741) find
  41. Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Kg-bert: Bert for knowledge graph completion. arXiv preprint arXiv:1909.03193. find
  42. Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2021. Rng-kbqa: Generation augmented iterative ranking for knowledge base question answering. arXiv preprint arXiv:2109.08678. find
  43. Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In ACL (2). The Association for Computer Linguistics.
  44. Dongzhuoran Zhou, Yuqicheng Zhu, Yuan He, Jiaoyan Chen, Evgeny Kharlamov, and Steffen Staab. 2025a. Evaluating knowledge graph based retrieval augmented generation methods under knowledge incompleteness. arXiv preprint arXiv:2504.05163. — [arXiv:2504.05163](https://arxiv.org/abs/2504.05163) find
  45. Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang, Hongkuan Zhou, Jiaoyan Chen, Steffen Staab, Yuan He, and Evgeny Kharlamov. 2025b. Gr-agent: Adaptive graph reasoning agent under incomplete knowledge. arXiv preprint arXiv:2512.14766. — [arXiv:2512.14766](https://arxiv.org/abs/2512.14766) find
  46. Yuqicheng Zhu, Daniel Hernández, Yuan He, Zifeng Ding, Bo Xiong, Evgeny Kharlamov, and Steffen Staab. 2025a. Predicate-conditional conformalized answer sets for knowledge graph embeddings. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4145–4167, Vienna, Austria. Assoc… find
  47. Yuqicheng Zhu, Nico Potyka, Daniel Hernández, Yuan He, Zifeng Ding, Bo Xiong, Dongzhuoran Zhou, Evgeny Kharlamov, and Steffen Staab. 2025b. Ar-grag: Explainable retrieval augmented generation using quantitative bipolar argumentation. In Conference on Neurosymbolic Learning and Reasoning, pages 697–7… find
  48. Yuqicheng Zhu, Nico Potyka, Mojtaba Nayyeri, Bo Xiong, Yunjie He, Evgeny Kharlamov, and Steffen Staab. 2024. Predictive multiplicity of knowledge graph embeddings in link prediction. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 334–354, Miami, Florida, USA. Associa… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-what-breaks-knowledge-graph-based/