Elamur Research⌕
☰
home / papers / 2023 / QA-KGNet: 一种语言模型驱动的知识图谱问答模型

QA-KGNet: 一种语言模型驱动的知识图谱问答模型

Summary

The paper addresses question answering that draws on both a pre-trained language model and a knowledge graph. Two issues motivate it. First, retrieving the few-hop neighbourhood of topic entities pulls in many nodes that have nothing to do with the question. Second, existing LM+KG systems process the text and the graph separately, which weakens structured reasoning. QA-KGNet extends the earlier QA-GNN design in three steps:

  • Joint graph: the question and a candidate answer are concatenated into a QA context. This context becomes an extra node linked to the question entities and answer-choice entities of a retrieved KG subgraph.
  • Relevance scoring: RoBERTa scores how relevant each KG node is to that context, and the score is attached to the node as an additional feature.
  • Reasoning and output: a multi-head graph attention network passes messages over several rounds. Graph attention pooling then downsamples the graph. An MLP combines the LM representation, the context node and the pooled graph to score each answer.

On CommonsenseQA, OpenBookQA and MedQA-USMLE, the authors report that QA-KGNet beats the baseline models. They also claim it is more interpretable. The work matters because it fuses text and graph knowledge in one representation rather than keeping them apart.

Key points

  • Published in 软件学报 (Journal of Software), 2023, 34(10): 4584–4600, by 乔少杰 (Qiao Shao-Jie) et al. (Chengdu University of Information Technology and partner institutions).
  • Subgraphs contain the nodes on paths of up to g = 3 hops between topic entities. On ConceptNet this yields roughly 400 nodes per question–answer pair, many of them noise.
  • Relevance scoring feeds the concatenated context and entity text into RoBERTa as the encoder, with RobertaLMHead as the scoring head.
  • QA-GNN uses a single set of attention weights. The multi-head attention here is meant to capture graph information from several perspectives, which the authors argue improves generalization.
  • The provided text omits the accuracy figures, so specific scores cannot be cited. The references suggest the medical setup uses UMLS/DrugBank-based knowledge, but this is not confirmed in the excerpt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cited by

1 paper of our library cites it (of the 140673 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.13328/j.cnki.jos.006882

@article{2023qa,
  title = {{QA-KGNet: 一种语言模型驱动的知识图谱问答模型}},
  author = {乔少杰 and 杨国平 and 于泳 and 韩楠 and 覃晓 and 屈露露 and 冉黎琼 and 李贺},
  journal = {软件学报 (Journal of Software)},
  year = {2023},
  doi = {10.13328/j.cnki.jos.006882},
  url = {https://doi.org/10.13328/j.cnki.jos.006882}
}

References

51 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xiong F, Wang HZ. Mining simple path traversal patterns in knowledge graph. Journal of Web Engineering, 2022, 21(2): 258  271. [doi: 10.13052/jwe1540-9589.2128] — [doi:10.13052/jwe1540-9589.2128](https://doi.org/10.13052/jwe1540-9589.2128) find
  2. Jamehshourani M, Fatemi A, Nematbakhsh MA. Implicit relation-based question answering to answer simple questions over DBpedia. Turkish Journal of Electrical Engineering and Computer Sciences, 2020, 28(3):1474  1490. [doi: 10.3906/elk-1908-102] — [doi:10.3906/elk-1908-102](https://doi.org/10.3906/elk-1908-102) find
  3. Jiang K, Wu DK, Jiang H. FreebaseQA: A new factoid QA data set matching trivia-style question-answer pairs with Freebase. In: Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. New York: ACM, 2019. 318  323. [doi: 10.… find
  4. Lu XY, Wang LF, Jiang ZJ, He SC, Liu SZ. MMKRL: A robust embedding approach for multi-modal knowledge graph representation learning. Applied Intelligence, 2022, 52(7): 7480  7497. [doi: 10.1007/s10489-021-02693-9] — [doi:10.1007/s10489-021-02693-9](https://doi.org/10.1007/s10489-021-02693-9) find
  5. Menin A, Torchelsen RP, Nedel LP. The effects of VR in training simulators: Exploring perception and knowledge gain. Computers & Graphics, 2022, 102(5): 402  412. [doi: 10.1016/j.cag.2021.09.015] — [doi:10.1016/j.cag.2021.09.015](https://doi.org/10.1016/j.cag.2021.09.015) find
  6. Li JZ, Hou L. Reviews on knowledge graph research. Journal of Shanxi University (Nat. Sci. Ed.), 2017, 40(3): 454  459 (in Chinese with English abstract). [doi: 10.13451/j.cnki.shanxi.univ(nat.sci.).2017.03.008] — [doi:10.13451/j.cnki.shanxi.univ(nat.sci.).2017.03.008](https://doi.org/10.13451/j.cnki.shanxi.univ%28nat.sci.%29.2017.03.008) find
  7. Nayyeri M, Cil GM, Vahdati S, Osborne F, Rahman M, Angioni S, Salatino AA, Recupero DR, Vassilyeva R, Motta E, Lehmann J. Trans4E: Link prediction on scholarly knowledge graphs. Neurocomputing, 2021, 461(3): 530  542. [doi: 10.1016/j.neucom.2021.02.100] — [doi:10.1016/j.neucom.2021.02.100](https://doi.org/10.1016/j.neucom.2021.02.100) find
  8. Dessì D, Osborne F, Recupero DR, Buscaldi D, Motta E. Generating knowledge graphs by employing natural language processing and machine learning techniques within the scholarly domain. Future Generation Computer Systems, 2021, 116(7): 253  264. [doi: 10.1016/j.future.2020.10.026] — [doi:10.1016/j.future.2020.10.026](https://doi.org/10.1016/j.future.2020.10.026) find
  9. Lin QK, Zhu YF, Lu H, Shi K, Niu ZD. Improving university faculty evaluations via multi-view knowledge graph. Future Generation Computer Systems, 2021, 117(5): 181  192. [doi: 10.1016/j.future.2020.11.021] — [doi:10.1016/j.future.2020.11.021](https://doi.org/10.1016/j.future.2020.11.021) find
  10. Yue Q, Li X, Li D. Chinese relation extraction on forestry knowledge graph construction. Computer Systems: Science & Engineering, 2021, 37(3): 423  442. [doi: 10.32604/csse.2021.014448] — [doi:10.32604/csse.2021.014448](https://doi.org/10.32604/csse.2021.014448) find
  11. Yu CM, Wang F, Liu YH, An L. Research on knowledge graph alignment model based on deep learning. Expert Systems with Applications, 2021, 186(4): 115  129. [doi: 10.1016/j.eswa.2021.115768] — [doi:10.1016/j.eswa.2021.115768](https://doi.org/10.1016/j.eswa.2021.115768) find
  12. Petroni F, Rocktaschel T, Riedel S, Lewis PSH, Bakhtin A, Wu YX, Miller AH. Language models as knowledge bases? In: Proc. of the 2019 Conf. on Empirical Methods in Natural Language Processing. Washington: ACL, 2019. 2463  2473. [doi: 10.18653/v1/D19-1250] find our card
  13. Bosselut A, Rashkin H, Sap M, Malaviya C, Celikyilmaz A, Choi Y. COMET: Commonsense transformers for automatic knowledge graph construction. In: Proc. of the 57th Conf. of the Association for Computational Linguistics. Washington: ACL, 2019. 4762  4779. [doi: 10.18653/v1/p19-1470] find our card
  14. Speer R, Chin J, Havasi C. ConceptNet 5.5: An open multilingual graph of general knowledge. In: Proc. of the 31st AAAI Conf. on Artificial Intelligence. San Francisco: AAAI, 2017. 4444  4451. find
  15. Huang SS, Zhu KQ, Liao QZ, Shen LB, Zhao YG. Enhanced story representation by ConceptNet for predicting story endings. In: Proc. of the 29th ACM Int’l Conf. on Information and Knowledge Management. New York: ACM, 2020. 3277  3280. [doi: 10.1145/3340531.3417466] — [doi:10.1145/3340531.3417466](https://doi.org/10.1145/3340531.3417466) find
  16. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou YQ, Li W, Liu PJ. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020, 21(2): 1  14. find
  17. Liao WX, Zeng B, Yin XW, Wei PF. An improved aspect-category sentiment analysis model for text sentiment analysis based on RoBERTa. Applied Intelligence, 2021, 51(6): 3522  3533. find
  18. Lin YB, Chen XY, Chen JM, Ren X. KagNet: Knowledge-aware graph networks for commonsense reasoning. In: Proc. of the 2019 Conf. on Empirical Methods in Natural Language Processing. New York: ACM, 2019. 2829  2839. [doi: 10.18653/v1/D19-1282] find
  19. Bao JW, Duan N, Yan Z, Zhou M, Zhao TJ. Constraint-based question answering with knowledge graph. In: Proc. of the 26th Int’l Conf. on Computational Linguistics. Washington: ACL, 2016. 2503  2514. find
  20. Sun HT, Dhingra B, Zaheer M, Mazaitis K, Salakhutdinov R, Cohen WW. Open domain question answering using early fusion of knowledge bases and text. In: Proc. of the 2018 Conf. on Empirical Methods in Natural Language Processing. New York: ACM, 2018. 4231  4242. [doi: 10.18653/v1/d18-1455] find
  21. Wang XY, Kapanipathi P, Musa R, Yu M, Talamadupula K, Abdelaziz I, Chang M, Fokoue A, Makni B, Mattei N, Witbrock M. Improving natural language inference using external knowledge in the science questions domain. In: Proc. of the 33rd AAAI Conf. on Artificial Intelligence. San Francisco: AAAI, 2019.… — [doi:10.1609/aaai.v33i01.33017208](https://doi.org/10.1609/aaai.v33i01.33017208) find
  22. Feng YL, Chen XY, Lin YB, Wang PF, Yan J, Ren X. Scalable multi-hop relational reasoning for knowledge-aware question answering. In: Proc. of the 2020 Conf. on Empirical Methods in Natural Language Processing. New York: ACM, 2020. 1295  1306. [doi: 10.18653/v1/2020.emnlp-main.99] find
  23. Yasunaga M, Ren HY, Bosselut A, Liang P, Leskovec J. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In: Proc. of the 2021 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Washington: ACL, 2021. 535… find
  24. Pan XM, Sun K, Yu D, Chen JS, Ji H, Cardie C, Yu D. Improving question answering with external knowledge. In: Proc. of the 2nd Workshop on Machine Reading for Question Answering. Washington: ACL, 2019. 27  37. [doi: 10.18653/v1/D19-5804] find
  25. Mihaylov T, Frank A. Knowledgeable reader: Enhancing cloze-style reading comprehension with external commonsense knowledge. In: Proc. of the 56th Annual Meeting of the Association for Computational Linguistics. Washington: ACL, 2018. 821  832. [doi: 10.18653/v1/P18-1076] find
  26. Bosselut A, Bras RL, Choi YJ. Dynamic neuro-symbolic knowledge graph construction for zero-shot commonsense question answering. In: Proc. of 35th AAAI Conf. on Artificial Intelligence. San Francisco: AAAI, 2021. 4923  4931. find
  27. Paul D, Frank A. Ranking and selecting multi-hop knowledge paths to better predict human needs. In: Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Washington: ACL, 2019. 3671  3681. [doi: 10.18653/v1/n19-1368] find
  28. Yang A, Wang Q, Liu J, Liu K, Lyu YJ, Wu H, She QQ, Li SJ. Enhancing pre-trained language representations with rich knowledge for machine reading comprehension. In: Proc. of the 57th Conf. of the Association for Computational Linguistics. Washington: ACL, 2019. 2346  2357. [doi: 10.18653/v1/p19-122… find
  29. Yih WT, Richardson M, Meek C, Chang MW, Suh J. The value of semantic parse labeling for knowledge base question answering. In: Proc. of the 54th Annual Meeting of the Association for Computational Linguistics. Washington: ACL, 2019. 546  555. [doi: 10.18653/v1/p16-2033] find
  30. Bevilacqua M, Navigli R. Quasi bidirectional encoder representations from transformers for word sense disambiguation. In: Proc. of the Int’l Conf. on Recent Advances in Natural Language Processing. Varna: INCOMA Ltd, 2019. 122  131. [doi: 10.26615/978-954-452-056-4\\_015] find
  31. Sun HT, Weiss TB, Cohen WW. PullNet: Open domain question answering with iterative retrieval on knowledge bases and text. In: Proc. of the 2019 Conf. on Empirical Methods in Natural Language Processing. Washington: ACL, 2019. 2380  2390. [doi: 10.18653/v1/D19-1242] find
  32. Xiong WH, Yu M, Chang SY, Guo XX, Wang YW. Improving question answering over incomplete kbs with knowledge-aware reader. In: Proc. of the 57th Conf. of the Association for Computational Linguistics. Washington: ACL, 2019. 4258  4264. [doi: 10.18653/v1/p19-1417] find
  33. Zhang YH, Qi P, Manning CD. Graph convolution over pruned dependency trees improves relation extraction. In: Proc. of the 2018 Conf. on Empirical Methods in Natural Language Processing. Washington: ACL, 2018. 2205  2215. [doi: 10.18653/v1/d18-1244] find our card
  34. Yasunaga M, Liang P. Graph-based, self-supervised program repair from diagnostic feedback. In: Proc. of the 37th Int’l Conf. on Machine Learning. Berlin: PMLR, 2020. 10799  10808. find
  35. Lee J, Lee I, Kang J. Self-attention graph pooling. In: Proc. of the 36th Int’l Conf. on Machine Learning. PMLR, 2019. 3734  3743. find
  36. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I. Attention is all you need. In: Proc. of the Advances in Neural Information Processing Systems 30: Annual Conf. on Neural Information Processing Systems 2017. New York: ACM, 2017. 5998  6008. find
  37. Velickovic P, Cucurull G, Casanova A, Romero A, Lio P, Bengio Y. Graph attention networks. In: Proc. of the 6th Int’l Conf. on Learning Representations. New York: ACM, 2018. 1  12. find
  38. Talmor A, Herzig J, Lourie N, Berant J. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In: Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Washington: ACL, 2019. 4149  4158. [doi: 10… find
  39. Mihaylov T, Clark P, Khot T, Sabharwal A. Can a suit of armor conduct electricity? A new dataset for open book question answering. In: Proc. of the 2018 Conf. on Empirical Methods in Natural Language Processing. Washington: ACL, 2018. 2381  2391. [doi: 10.18653/v1/d18-1260] find
  40. Jin D, Pan B, Oufattole N, Weng WH, Fang HY, Szolovits P. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Applied Sciences, 2021, 11(4): 1  17. [doi: 10.3390/app11146421] find our card
  41. Bodenreider O. The unified medical language system (UMLS): Integrating biomedical terminology. Nucleic Acids Research, 2004, 32(4): 267  270. [doi: 10.1093/nar/gkh061] — [doi:10.1093/nar/gkh061](https://doi.org/10.1093/nar/gkh061) find
  42. Wishart DS, Feunang YD, Guo AC, Lo EJ, Marcu A, Grant JR, Sajed T, Johnson D, Li C, Sayeeda Z, Assempour N, Iynkkaran I, Liu YF, Maciejewski A, Gale N, Wilson A, Chin L, Cummings R, Le D, Pon A, Knox C, Wilson M. DrugBank 5.0: A major update to the DrugBank database for 2018. Nucleic Acids Research,… — [doi:10.1093/nar/gkx1037](https://doi.org/10.1093/nar/gkx1037) find
  43. Liu FY, Shareghi E, Meng ZQ, Basaldella M, Collier N. Self-alignment pretraining for biomedical entity representations. In: Proc. of the 2021 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Washington: ACL, 2021. 4228  4238. [doi: 1… find
  44. Liu YH, Ott M, Goyal M, Du JF, Joshi M, Chen DQ, Levy O, Lewis M, Zettlemoyer L, Stoyanov V. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692, 2019. find
  45. Clark P, Etzioni O, Khashabi D, Khot T, Mishra BD, Richardson K, Sabharwal A, Schoenick C, Tafjord O, Tandon N, Bhakthavatsalam S, Groeneveld D, Guerquin M, Schmitz M. From ‘f’ to ‘a’ on the NY regent science exams: An overview of the aristo project. arXiv:1909.01958, 2019. find
  46. Santoro A, Raposo D, Barrett DGT, Malinowski M, Pascanu R, Battaglia PW, Lillicrap T. A simple neural network module for relational reasoning. In: Proc. of the Advances in Neural Information Processing Systems 30: Annual Conf. on Neural Information Processing Systems. Washington: ACL, 2017. 4967  4… find our card
  47. Schlichtkrull MS, Kipf TN, Bloem P, Berg RVD, Titov I, Welling M. Modeling relational data with graph convolutional networks. In: Proc. of the 15th Int’l Conf. on Semantic Web. Berlin: Springer, 2018. 593  607. [doi: 10.1007/978-3-319-93417-4\\_38] find
  48. Wang XY, Kapanipathi P, Musa R, Yu M, Talamadupula K, Abdelaziz I, Chang M, Fokoue A, Makni B, Mattei N, Witbrock M. Improving natural language inference using external knowledge in the science questions domain. In: Proc. of the 33rd AAAI Conf. on Artificial Intelligence. San Francisco: AAAI, 2019.… — [doi:10.1609/aaai.v33i01.33017208](https://doi.org/10.1609/aaai.v33i01.33017208) find
  49. Kingma DP, Ba J. Adam: A method for stochastic optimization. In: Proc. of the 3rd Int’l Conf. on Learning Representations. Washington: ACL, 2015. 1  15. find
  50. Devlin J, Chang MW, Lee KT, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Washington: ACL, 2019. 4171  4186. [doi… find
  51. Lee J, Yoon W, Kim SD, Kim D, Kim SY, So CH, Kang J. BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 2020, 36(4): 1234  1240. [doi: 10.1093/bioinformatics/btz682] 附中文参考文献 : [6] 李涓子 , 侯磊 . 知识图谱研究综述 . 山西大学学报 ( 自然科学版 ), 2017, 40(3): 454  459… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/qiao-2023-qa-kgnet-language/