Elamur Research⌕
☰
home / papers / 2019 / Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs

Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs

Paper · 2019 · Deepak Nathani, Jatin Chauhan, Charu Sharma et al.

Summary

Knowledge graphs are often incomplete, so relation prediction (knowledge base completion) aims to decide whether a missing triple is valid. Earlier translational and CNN-based embedding models, such as TransE, ConvE and ConvKB, score each triple in isolation and ignore information in an entity's surroundings. The authors build an encoder–decoder model to address this. The encoder extends graph attention networks (GATs) so that attention weights cover both neighboring entities and the relations connecting them. To keep distant nodes from fading out as layers stack, it adds auxiliary edges between n-hop neighbors using relation composition. ConvKB then serves as the decoder that scores triples. The model was tested on FB15k-237, WN18RR, NELL-995, Kinship and UMLS against reproduced baselines. It gave large gains, including a 104% relative improvement in Hits@1 on FB15k-237. The work matters because it shows that neighborhood- and relation-aware attention produces stronger KG embeddings than triple-by-triple scoring. It was the first graph-attention embedding aimed specifically at relation prediction.

Key points

  • Architecture: a generalized graph attention encoder over entity and relation embeddings, with multi-head attention, paired with a ConvKB decoder.
  • Multi-hop context: auxiliary edges for n-hop paths (shown for 2 hops) let information flow from distant entities. This also captures semantically similar relation clusters.
  • Results: a 104% Hits@1 gain over the prior best method on FB15k-237. The model is best on five metrics for FB15k-237 and two for WN18RR.
  • PageRank analysis: denser graphs show larger MRR gains relative to DistMult (correlation r = 0.808). WN18RR, which is sparse and hierarchical, is an exception.
  • Ablation on NELL-995: removing relation information hurts much more than removing n-hop paths (−PG). This highlights how central relation embeddings are to the model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Indian Institute of Technology Hyderabad.

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

🧭 Topics 1Research subjects and languages the work is about.Knowledge graphs & ontologies · 103 works

Cite

Cite as Deepak Nathani, Jatin Chauhan, Charu Sharma and Manohar Kaul. 2019. Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1466

@inproceedings{nathani2019learning,
  title = {{Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs}},
  author = {Deepak Nathani, Jatin Chauhan, Charu Sharma and Manohar Kaul},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1466},
  url = {https://doi.org/10.18653/v1/p19-1466}
}

References

27 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
  2. Jonathan Berant, Noga Alon, Ido Dagan, and Jacob Goldberger. 2015. Efficient global learning of entailment graphs. Computational Linguistics, 41(2):221–263. find
  3. Stanley Kok and Pedro Domingos. 2007. Statistical predicate invention. In Proceedings of the 24th International Conference on Machine Learning, (ICML), 2007. find
  4. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013, pages 1533– 1544.
  5. Lili Kotlerman, Ido Dagan, Bernardo Magnini, and Luisa Bentivogli. 2015. Textual entailment graphs. Natural Language Engineering, 21(5):699–724. \#\# Acknowledgments Jonathan Berant, Ido Dagan, and Jacob Goldberger. 2010. Global learning of focused entailment graphs. In Proceedings of the 48th Annual… find
  6. Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), 2014.
  7. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems (NIPS… find
  8. Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), 2018. find
  9. Dennis Diefenbach, Kamal Singh, and Pierre Maret. 2018. Wdaqua-core1: a question answering service for rdf knowledge bases. In Companion of the The Web Conference 2018 on The Web Conference (WWW), 2018, pages 1087–1091. International World Wide Web Conferences Steering Committee. find
  10. He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2017. find
  11. Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2018. Multi-hop knowledge graph reasoning with reward shaping. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. find
  12. Yankai Lin, Zhiyuan Liu, Huanbo Luan, Maosong Sun, Siwei Rao, and Song Liu. 2015. Modeling relation paths for representation learning of knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015.
  13. Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. 2018. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te…
  14. Maximilian Nickel, Lorenzo Rosasco, Tomaso A Poggio, et al. 2016. Holographic embeddings of knowledge graphs. In (AAAI), 2016, volume 2, pages 3–2. find
  15. Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on International Conference on Machine Learning, (ICML), 2011. find
  16. Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In European Semantic Web Conference (ESWC), 2018, pages 593–607. find
  17. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013a. Reasoning with neural tensor networks for knowledge base completion. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26 (NIPS), 2013, pa…
  18. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013b. Reasoning with neural tensor networks for knowledge base completion. In Advances in neural information processing systems (NIPS), 2013, pages 926–934.
  19. Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015, pages 1499– 1509.
  20. Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In International Conference on Machine Learning (ICML), 2016, pages 2071–2080. find
  21. Jorge Carlos Valverde-Rebaza and Alneu de Andrade Lopes. 2012. Link prediction in complex networks based on cluster information. In Proceedings of the 21st Brazilian Conference on Advances in Artificial Intelligence, (SBIA), 2012, pages 92–101. find
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces…
  23. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. International Conference on Learning Representations (ICLR), 2018. find
  24. Robert West, Evgeniy Gabrilovich, Kevin Murphy, Shaohua Sun, Rahul Gupta, and Dekang Lin. 2014. Knowledge base completion via search-based question answering. In Proceedings of the 23rd International Conference on World Wide Web, (WWW), 2014, pages 515–526. find
  25. Wenhan Xiong, Thien Hoang, and William Yang Wang. 2017. Deeppath: A reinforcement learning method for knowledge graph reasoning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017. find
  26. Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. International Conference on Learning Representations (ICLR), 2015.
  27. Yuanzhe Zhang, Kang Liu, Shizhu He, Guoliang Ji, Zhanyi Liu, Hua Wu, and Jun Zhao. 2016. Question answering over knowledge base with neural attention combining global knowledge information. arXiv preprint arXiv:1606.00979, 2016. — [arXiv:1606.00979](https://arxiv.org/abs/1606.00979) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nathani-2019-learning-attention-based-embeddings-relation/