Elamur Research⌕
☰
home / papers / 2026 / Joint Knowledge Base Completion and Question Answering by Combining Large Language Models and Small Language Models

Joint Knowledge Base Completion and Question Answering by Combining Large Language Models and Small Language Models

Paper · 2026 · Yinan Liu, Dongying Lin, Sigang Luo et al.

Summary

The tasks of filling in missing facts in a knowledge base (KBC) and answering questions over one (KBQA) are mutually dependent. A more complete KB makes questions easier to answer, and facts found while answering can fill gaps in the KB. Earlier joint approaches (BiNet, KGT5) relied only on small language models (SLMs). Pipelines built purely on LLMs are expensive and prone to hallucination. The authors propose JCQL, which couples an LLM agent with an SLM in an iterative loop. A T5-like encoder-decoder is trained for KBC on inputs that combine the head entity and relation, the entity's description, and neighbouring triples ranked by relation co-occurrence. For KBQA, a ToG-style LLM agent reasons through thought, action and observation steps. The trained KBC model is one of its tools. The agent's reasoning paths are then fed back as extra data to incrementally fine-tune the SLM using experience replay. On WebQSP and CWQ with deliberately incomplete KBs, JCQL beats all baselines on both tasks while needing fewer LLM calls. This shows that small and large models can offset each other's weaknesses.

Key points

  • The agent's action space contains search(entity), generate(sub-question), complete(entity, relation) and finish. In generate, the LLM produces triples from its own knowledge, using BM25 retrieval over earlier observations. In complete, the fine-tuned SLM predicts the missing triples.
  • In the ablation, removing complete hurts KBQA more than removing generate. This suggests the SLM supplies more reliable entities than the LLM's parametric knowledge.
  • Efficiency was measured with 30% KB on CWQ. JCQL averaged 7.3 LLM calls and 25.3 s per question, compared with 18.5 calls and 33.7 s for ToG. On WebQSP, JCQL averaged 5.1 calls and 16.6 s, compared with 17.5 calls and 26.6 s for ToG. Other baselines were GoG, PoG, LMP and PDRR.
  • For KBC, performance grows as the number of SLM decoder sequences R rises from 100 to 1000, then levels off.
  • The stated limitations are three: the SLM context uses only one-hop neighbours, the method works with a single KB, and incremental learning relies on a simple replay strategy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northeastern UniversityShenyang · China42 works · 1,940 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksQuestion answering · 441 worksKnowledge graphs & ontologies · 192 works

Cite

Cite as Yinan Liu, Dongying Lin, Sigang Luo, Xiaochun Yang and Bin Wang. 2026. Joint Knowledge Base Completion and Question Answering by Combining Large Language Models and Small Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1704

@inproceedings{liu2026joint,
  title = {{Joint Knowledge Base Completion and Question Answering by Combining Large Language Models and Small Language Models}},
  author = {Yinan Liu, Dongying Lin, Sigang Luo, Xiaochun Yang and Bin Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1704},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1704}
}

References

47 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Tom Ayoola, Shubhi Tyagi, Joseph Fisher, Christos Christodoulopoulos, and Andrea Pierleoni. 2022. Re-FinED: An efficient zero-shot-capable approach to end-to-end entity linking. In NAACL, pages 209–220. find
  3. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In SIGMOD, pages 1247–1250. find
  4. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multi-relational Data. In NeurIPS, pages 2787–2795.
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  6. Chen Chen, Yufei Wang, Aixin Sun, Bing Li, and Kwok-Yan Lam. 2023. Dipping PLMs sauce: Bridging structure and text for effective knowledge graph completion via conditional soft prompting. In ACL, pages 11489–11503.
  7. Liyi Chen, Panrong Tong, Zhongming Jin, Ying Sun, Jieping Ye, and Hui Xiong. 2024. Plan-on-graph: Self-correcting adaptive planning of large language model on knowledge graphs. In NeurIPS, pages 37665–37691. find
  8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, and 16 others. 2022.… find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186.
  10. Ju Fan, Zihui Gu, Songyue Zhang, Yuxin Zhang, Zui Chen, Lei Cao, Guoliang Li, Samuel Madden, Xiaoyong Du, and Nan Tang. 2024. Combining small language models and large language models for zero-shot NL2SQL. Proceedings of the VLDB Endowment, 17(11):2750–2763. find
  11. Zhifeng Jia, Hanmo Liu, Haoyang Li, and Lei Chen. 2025. SIT: Selective Incremental Training for Dynamic Knowledge Graph Embedding . In ICDE, pages 1607–1621. find
  12. Youmin Ko, Hyemin Yang, Taeuk Kim, and Hyunjoon Kim. 2025. Subgraph-aware training of language models for knowledge graph completion using structure-aware contrastive learning. In WWW, page 72–85. find
  13. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, and 1 others. 2015. Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167–195.
  14. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL, pages 7871–7880. event temporal reasonin…
  15. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. Preprint, arXiv:1312.5602. find
  16. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21:140:1–140:67.
  17. Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. find
  18. Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022. Sequence-to-sequence knowledge graph completion and question answering. In ACL, pages 2814– 2828. find
  19. Rui Li, Chaozhuo Li, Yanming Shen, Zeyu Zhang, and Xu Chen. 2024a. Generalizing knowledge graph embedding with universal orthogonal parameterization. In ICML, pages 28040–28059. find
  20. Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel Ni, Heung-Yeung Shum, and Jian Guo. 2024. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. In ICLR, pages 1–31. find
  21. Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. 2024b. Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. In ICLR, pages 1–23. find
  22. Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. RotatE: Knowledge graph embedding by relational rotation in complex space. In ICLR, pages 1649–1667.
  23. Lihui Liu, Boxin Du, Jiejun Xu, Yinglong Xia, and Hanghang Tong. 2022a. Joint knowledge graph completion and question answering. In SIGKDD, pages 1098–1108. find
  24. Yang Liu, Xiaobin Tian, Zequn Sun, and Wei Hu. 2024. Finetuning generative large language models with discrimination instructions for knowledge graph completion. In ISWC, pages 199–217. find
  25. Yinan Liu, Hu Chen, and Wei Shen. 2022b. Personal attribute prediction from conversations. In WWW, pages 223–227. find
  26. Yinan Liu, Hu Chen, Wei Shen, and Jiaoyan Chen. 2023. Low-resource personal attribute prediction from conversations. In AAAI, pages 4507–4515. find
  27. Yinan Liu, Wei Shen, and Xiaojie Yuan. 2016. Deola: A system for linking author entities in web document with DBLP. In CIKM, pages 2449–2452. ACM. find
  28. Yinan Liu, Ziyang Zhang, Bin Wang, and Xiaochun Yang. 2026. SEFEL: A simple yet effective framework for fast event linking. In AAAI, pages 15377– 15385. find
  29. Sigang Luo, Yinan Liu, Dongying Lin, Yingying Zhai, Bin Wang, Xiaochun Yang, and Junpeng Liu. 2025. ETRQA: A comprehensive benchmark for evaluating Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal…
  30. Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In ICML, pages 2071–2080.
  31. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: A free collaborative knowledgebase. Communications of the ACM, 57(10):78–85.
  32. Junhong Wan, Tao Yu, Kunyu Jiang, Yao Fu, Weihao Jiang, and Jiang Zhu. 2025. Digest the knowledge: Large language models empowered message passing for knowledge graph question answering. In ACL, pages 15426–15442. find
  33. Cunda Wang, Ziying Ma, Po Hu, Weihua Wang, and Feilong Bao. 2026. Debate to align: Reliable entity alignment through two-stage multi-agent debate. Preprint, arXiv:2604.13551.
  34. Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022. SimKGC: Simple contrastive knowledge graph completion with pre-trained language models. In ACL, pages 4281–4294. find
  35. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In ICLR, pages 1–24.
  36. Yihua Zhu, Qianying Liu, Akiko Aizawa, and Hidetoshi Shimodaira. 2025. Beyond chains: Bridging large language models and knowledge bases in complex question answering. Preprint, arXiv:2505.14099. — [arXiv:2505.14099](https://arxiv.org/abs/2505.14099) find
  37. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, pages 24824–24837.
  38. Changyi Xiao and Yixin Cao. 2024. Knowledge graph completion by intermediate variables regularization. In NeurIPS, pages 110218–110245. find
  39. Yao Xu, Shizhu He, Jiabei Chen, Zihao Wang, Yangqiu Song, Hanghang Tong, Guang Liu, Jun Zhao, and Kang Liu. 2024. Generate-on-graph: Treat LLM as both agent and KG for incomplete knowledge graph question answering. In EMNLP, pages 18410– 18430. find
  40. Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In ICLR, pages 1–12.
  41. Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. KG-BERT: BERT for knowledge graph completion. Preprint, arXiv:1909.03193. find
  42. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In ICLR, pages 1–33.
  43. Donghan Yu, Sheng Zhang, Patrick Ng, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu, William Wang, Zhiguo Wang, and Bing Xiang. 2023. Decaf: Joint decoding of answers and logical forms for question answering over knowledge bases. Preprint, arXiv:2210.00063. find
  44. Duanyang Yuan, Sihang Zhou, Xiaoshu Chen, Dong Wang, Ke Liang, Xinwang Liu, and Jian Huang. 2025. Knowledge graph completion with relation-aware anchor enhancement. In AAAI, pages 15239–15247. find
  45. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. Preprint, arXiv:2506.05176.
  46. Ruilin Zhao, Feng Zhao, Long Wang, Xianzhi Wang, and Guandong Xu. 2024. Kg-cot: Chain-of-thought prompting of large language models over knowledge graphs for knowledge-aware question answering. In IJCAI, pages 6642–6650. find
  47. Zihan Zhou, Yinan Liu, Yuyang Xie, Bin Wang, Xiaochun Yang, and Zezheng Feng. 2026. Diaglink: A dual-user diagnostic assistance system by synergizing experts with llms and knowledge graphs. Preprint, arXiv:2601.20311. — [arXiv:2601.20311](https://arxiv.org/abs/2601.20311) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-joint-knowledge-base-completion-question/