Elamur Research⌕
☰
home / papers / 2026 / Fact Finder - Enhancing Domain Expertise of Large Language Models by Incorporating Knowledge Graphs

Fact Finder - Enhancing Domain Expertise of Large Language Models by Incorporating Knowledge Graphs

Paper · 2026 · Daniel Steinigen, Roman Teucher, Timm Heine Ruland et al.

Summary

LLMs often lack specialised domain knowledge and may hallucinate or give incomplete answers. This is a serious problem in life sciences tasks such as drug target identification. The authors present FactFinder, a question-answering system that grounds LLM output in a medical knowledge graph. It uses PrimeKG, stored in Neo4j and queried with Cypher, and the pipeline is built from LangChain components. The pipeline works in five stages:

  1. An LLM translates a user question into a Cypher query, using the graph schema and descriptions of the relations.
  2. Rule-based fixes clean the query: formatting, lowercasing, synonym mapping, replacing deprecated syntax, and mapping child nodes to parent nodes.
  3. The query is run on the graph.
  4. The returned nodes are passed back to the LLM.
  5. The LLM turns them into a natural-language answer, and the interface shows the subgraph behind it as evidence.

The authors evaluated the system on a hand-built benchmark of 69 question–Cypher pairs. GPT-4o retrieved the correct nodes with about 78% precision. LLM-as-a-Judge comparisons favoured the hybrid system over a plain LLM for both correctness and completeness. The work shows that current LLMs can write usable Cypher queries for complex biomedical questions. The code, dataset and prompts are released.

Key points

  • Knowledge graph: PrimeKG combines 20 sources and covers 17,080 diseases. It has 4,050,249 relationships and 29 edge types, linking diseases to drugs, genes, exposures and phenotypes.
  • Retrieval results: On the 69-pair text-to-Cypher dataset, GPT-4o without entity enhancement scored IoU 75.2%, precision 77.5% and recall 77.8%. Llama-3.1-70B and Mixtral-8x7B performed much worse.
  • Entity enhancement: Adding entity enhancement (Linnaeus-based synonym replacement) lowered scores for every model except GPT-4-turbo. The causes were merged gene–protein nodes and terminology mismatches.
  • Answer quality: Judged against an LLM-only baseline, the hybrid system gave more correct answers in 94.12% of cases and more complete answers in 96.08%. Verbalization of graph results was rated 89.13% correct and 80.43% complete.
  • Handling wrong queries: When fed deliberately wrong queries, GPT-4o declined to answer in 94.2% of cases and GPT-4-turbo in 91.3%. The code is available at github.com/chrschy/fact-finder.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Fraunhofer Institute for Intelligent Analysis and Information SystemsSankt Augustin · Germany9 works · 3,875 citations🏢BayerLeverkusen · Germany3 works · 401 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksKnowledge graphs & ontologies · 192 works

Cite

Cite as Daniel Steinigen, Roman Teucher, Timm Heine Ruland, Max Rudat, Nicolas Flores-Herr, Peter Fischer, Nikola Milosevic, Christopher Schymura and Angelo Ziletti. 2026. Fact Finder - Enhancing Domain Expertise of Large Language Models by Incorporating Knowledge Graphs. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-demo.8

@inproceedings{steinigen2026fact,
  title = {{Fact Finder - Enhancing Domain Expertise of Large Language Models by Incorporating Knowledge Graphs}},
  author = {Daniel Steinigen, Roman Teucher, Timm Heine Ruland, Max Rudat, Nicolas Flores-Herr, Peter Fischer, Nikola Milosevic, Christopher Schymura and Angelo Ziletti},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-demo.8},
  url = {https://doi.org/10.18653/v1/2026.eacl-demo.8}
}

References

32 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Carlos Badenes-Olmedo and Oscar Corcho. 2023. Lessons learned to enable question answering on knowledge graphs extracted from scientific publications: A case study on the coronavirus literature. Journal of Biomedical Informatics, 142:104382. find
  2. Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pages 78–106, Toronto, Canada. Association for C…
  3. Bojana Bašaragin, Adela Ljajić, Darija Medvecki, Lorenzo Cassano, Miloš Košprdić, and Nikola Milošević. 2024. How do you know that? teaching generative language models to reference answers to biomedical questions. find
  4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  5. Nilesh Chakraborty, Denis Lukovnikov, Gaurav Maheshwari, Priyansh Trivedi, Jens Lehmann, and Asja Fischer. 2021. Introduction to neural network-based question answering over knowledge graphs. WIREs Data Mining and Knowledge Discovery, 11(3):e1389. find
  6. Payal Chandak, Kexin Huang, and Marinka Zitnik. 2023. Building a knowledge graph to enable precision medicine. Scientific Data, 10(1):67.
  7. Shuaichen Chang and Eric Fosler-Lussier. 2023. How to prompt llms for text-to-sql: A study in zero-shot, single-domain, and cross-domain settings. Preprint, arXiv:2305.11853. — [arXiv:2305.11853](https://arxiv.org/abs/2305.11853) find
  8. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130. find
  9. Guandong Feng, Guoliang Zhu, Shengze Shi, Yue Sun, Zhongyi Fan, Sulin Gao, and Jun Hu. 2023. Robust nl-to-cypher translation for kbqa: Harnessing large language model with chain of prompts. In Knowledge Graph and Semantic Computing: Knowledge Graph Empowers Artificial General Intelligence, pages 317… find
  10. Yichun Feng, Lu Zhou, Yikai Zheng, Ruikun He, Chao Ma, and Yixue Li. 2024. Knowledge graph-based thought: a knowledge graph enhanced llms framework for pan-cancer question answering. bioRxiv. find
  11. Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023. Text-to-sql empowered by large language models: A benchmark evaluation. Preprint, arXiv:2308.15363.
  12. Martin Gerner, G. Nenadic, and Casey M. Bergman. 2010. Linnaeus: A species name identification system for biomedical literature. BMC Bioinformatics, 11:85 – 85. find
  13. Aibo Guo, Xinyi Li, Guanchen Xiao, Zhen Tan, and Xiang Zhao. 2022. Spcql: A semantic parsing dataset for converting natural language into cypher. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, page 3973–3977, New York, NY, USA. Association fo… find
  14. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).
  15. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  16. Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Xin Zhao, and Ji-Rong Wen. 2023b. StructGPT: A general framework for large language model to reason over structured data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9237–9251, Singapore. Association… find
  17. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2023. Gpt-4 passes the bar exam. Available at SSRN 4389233. find
  18. Adela Ljajić, Miloš Košprdić, Bojana Bašaragin, Darija Medvecki, Lorenzo Cassano, and Nikola Milošević. 2024. Scientific qa system with verifiable answers. The 6th International Open Search Symposium. find
  19. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2023. Expertqa: Expert-curated questions and attributed answers. arXiv preprint arXiv:2309.07852.
  20. Nikola Milošević and Wolfgang Thielemann. 2023. Comparison of biomedical relationship extraction methods and models for knowledge graph creation. Journal of Web Semantics, 75:100756. find
  21. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375.
  22. OpenAI. 2023. Gpt-4 technical report. arXiv:2303.08774.
  23. Preprint, Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2024. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering.
  24. A Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22. find
  25. Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction. In Thirty-seventh Conference on Neural Information Processing Systems.
  26. Priyanka Sen, Sandeep Mavadia, and Amir Saffari. 2023. Knowledge graph-augmented language models for complex question answering. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pages 1–8, Toronto, Canada. Association for Computational Linguistics… find
  27. Saurabh Srivastava, Mayur Patidar, Sudip Chowdhury, Puneet Agarwal, Indrajit Bhattacharya, and Gautam Shroff. 2021. Complex question answering on knowledge graphs using machine translation and multi-task learning. In Proceedings of the 16th Conference of the European Chapter of the Association for C… find
  28. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  29. Ethan Waisberg, Joshua Ong, Mouayad Masalkhi, Sharif Amit Kamran, Nasif Zaman, Prithul Sarker, Andrew G Lee, and Alireza Tavakkoli. 2023. Gpt-4: a new era of artificial intelligence in medicine. Irish Journal of Medical Science (1971-), 192(6):3197– 3200. find
  30. Ziyu Zhao, Wei Liu, Tim French, and Michael Stewart. 2024. Cyspider: A neural semantic parsing corpus with baseline models for property graphs. In AI 2023: Advances in Artificial Intelligence, pages 120–132, Singapore. Springer Nature Singapore. find
  31. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.
  32. Angelo Ziletti and Leonardo D’Ambrosi. 2024. Retrieval augmented text-to-sql generation for epidemiological question answering using electronic health records. Preprint, arXiv:2403.09226. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/steinigen-2026-fact-finder-enhancing-domain-expertise/