Elamur Research⌕
☰
home / papers / 2013 / Reasoning With Neural Tensor Networks for Knowledge Base Completion

Reasoning With Neural Tensor Networks for Knowledge Base Completion

Summary

Knowledge bases such as WordNet or Freebase are incomplete, and reasoning over their discrete entities is difficult. The authors address the task of knowledge base completion: given already known facts, the model must assess whether a new triple (entity, relation, entity) is true. External texts are not re-parsed for this. To this end, they propose the Neural Tensor Network (NTN). In this model, each relation has its own bilinear tensor layer that multiplicatively combines the vectors of the two entities. Several earlier approaches, including distance, single layer and bilinear, turn out to be special cases of it. The second idea is to represent an entity as the average of the word vectors of its name. Then, for instance, different species of tigers can share statistical strength. The third idea is to initialize these vectors with word representations learned in advance, without supervision, on large corpora. On held-out triples, NTN outperforms all the models it was compared with. Word vectors and pre-training improve the results of almost every one of them. The work became one of the key papers in the early line of neural models for knowledge base completion.

Key points

  • The classification accuracy on unseen triples is 86.2% on WordNet and 90.0% on FreeBase.
  • Even with randomly initialized word vectors, NTN reaches 84.7% on WordNet and 88.9% on FreeBase. Hence, the model reasons well even without external text resources.
  • Three ways of representing entities were compared: EV (a separate vector for each entity), WV (randomly initialized word vectors) and WV-init (word vectors pre-trained without supervision). On WordNet, word vectors are consistently better than EV; on FreeBase, mostly so.
  • Baseline models for comparison: Distance (Bordes et al.), Single Layer, Hadamard and Bilinear. NTN uses k tensor slices and the tanh nonlinearity. Each slice can correspond to a separate type of entity pair.
  • An example from FreeBase: the model inferred the gender and nationality of Francesco Guicciardini. To do so, it used his place of birth (Florence) and the similarity of his name to other entries, in particular Francesco Patrizi.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics Reasoning Knowledge graphs & ontologies

Cite

Cite as Richard Socher, Danqi Chen, Christopher D. Manning and Andrew Y. Ng. 2013. Reasoning With Neural Tensor Networks for Knowledge Base Completion. NeurIPS 2013.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{socher2013reasoning,
  title = {{Reasoning With Neural Tensor Networks for Knowledge Base Completion}},
  author = {Richard Socher, Danqi Chen, Christopher D. Manning and Andrew Y. Ng},
  booktitle = {NeurIPS 2013},
  year = {2013},
  url = {https://papers.nips.cc/paper_files/paper/2013/hash/b337e84de8752b27eda3a12363109e80-Abstract.html}
}

References

24 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G.A. Miller. WordNet: A Lexical Database for English. Communications of the ACM, 1995. our card
  2. F. M. Suchanek, G. Kasneci, and G. Weikum. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web, 2007. find
  3. J. Graupmann, R. Schenkel, and G. Weikum. The SphereSearch engine for unified ranked retrieval of heterogeneous XML and web documents. In Proceedings of the 31st international conference on Very large data bases, VLDB, 2005. find
  4. V. Ng and C. Cardie. Improving machine learning approaches to coreference resolution. In ACL, 2002. our card
  5. R. Snow, D. Jurafsky, and A. Y. Ng. Learning syntactic patterns for automatic hypernym discovery. In NIPS, 2005. find
  6. A. Fader, S. Soderland, and O. Etzioni. Identifying relations for open information extraction. In EMNLP, 2011. find
  7. G. Angeli and C. D. Manning. Philosophers are mortal: Inferring the truth of unseen facts. In CoNLL, 2013. find
  8. A. Bordes, J. Weston, R. Collobert, and Y. Bengio. Learning structured embeddings of knowledge bases. In AAAI, 2011. find
  9. R. Jenatton, N. Le Roux, A. Bordes, and G. Obozinski. A latent factor model for highly multi-relational data. In NIPS, 2012. find
  10. A. Bordes, X. Glorot, J. Weston, and Y. Bengio. Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing. AISTATS, 2012. find
  11. I. Sutskever, R. Salakhutdinov, and J. B. Tenenbaum. Modelling relational data using Bayesian clustered tensor factorization. In NIPS, 2009. find
  12. M. Ranzato and A. Krizhevsky G. E. Hinton. Factored 3-Way Restricted Boltzmann Machines For Modeling Natural Images. AISTATS, 2010. find
  13. D. Yu, L. Deng, and F. Seide. Large vocabulary speech recognition using deep tensor neural networks. In INTERSPEECH, 2012. find
  14. R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013. find
  15. A. Yates, M. Banko, M. Broadhead, M. J. Cafarella, O. Etzioni, and S. Soderland. Textrunner: Open information extraction on the web. In HLT-NAACL (Demonstrations), 2007. find
  16. M. Nickel, V. Tresp, and H. Kriegel. A three-way model for collective learning on multi-relational data. In ICML, 2011. find
  17. A. Bordes, N. Usunier, A. Garca-Durn, J. Weston, and O. Yakhnenko. Irreflexive and hierarchical relations as translations. CoRR, abs/1304.7158, 2013. — [arXiv:1304.7158](https://arxiv.org/abs/1304.7158) find
  18. J. Turian, L. Ratinov, and Y. Bengio. Word representations: a simple and general method for semi-supervised learning. In Proceedings of ACL, pages 384–394, 2010. our card
  19. R. Socher, B. Huval, C. D. Manning, and A. Y. Ng. Semantic Compositionality Through Recursive Matrix-Vector Spaces. In EMNLP, 2012. our card
  20. R. Collobert and J. Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML, 2008. find
  21. R. Socher, E. H. Huang, J. Pennington, A. Y. Ng, and C. D. Manning. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In NIPS. MIT Press, 2011. find
  22. E. H. Huang, R. Socher, C. D. Manning, and A. Y. Ng. Improving Word Representations via Global Context and Multiple Word Prototypes. In ACL, 2012. our card
  23. K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, SIGMOD, 2008. find
  24. N. Tandon, G. de Melo, and G. Weikum. Deriving a web-scale commonsense fact database. In AAAI Conference on Artificial Intelligence (AAAI 2011), 2011. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/socher-2013-reasoning-neural-tensor-networks-knowledge/