Elamur Research⌕
☰
home / papers / 2013 / Translating Embeddings for Modeling Multi-relational Data

Translating Embeddings for Modeling Multi-relational Data

Paper · 2013 · Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durán et al.

Summary

The paper addresses learning vector representations (embeddings) for multi-relational data. These are knowledge graphs in which each fact is written as a triple (head, label, tail). The authors sought a model that is easy to train, has few parameters, and works on very large databases. To this end, they proposed TransE. In it, each entity and each relation has a single low-dimensional vector, and a relation is treated as a translation in the space. If a triple is true, the head vector plus the relation vector should end up close to the tail vector. The model is trained with stochastic gradient descent using a margin-based ranking loss. This loss contrasts true triples with "corrupted" ones, in which the head or the tail is replaced by a random entity. The norm of entity vectors is constrained to one. Experiments on Wordnet and Freebase showed that TransE predicts links noticeably better than more complex earlier models, in particular SE. Moreover, it could be trained on a Freebase fragment with one million entities. The value of the work lies in the fact that a simple geometric idea yielded a strong and scalable model for knowledge base completion.

Key points

  • Datasets. WN has 40,943 entities, 18 relations, and 141,442 training triples. FB15k has 14,951 entities, 1,345 relations, and 483,142 triples. FB1M has 1M entities, 23,382 relations, and about 17.5M training triples.
  • Number of parameters. It grows as O(n_e·k + n_r·k). On FB15k this amounts to roughly 0.81M parameters, whereas RESCAL has 87.8M and SE has 7.47M. The distance between vectors is computed with the L1 or L2 norm, and quality is evaluated with the mean rank and hits@10 metrics.
  • Relation types. The FB15k relations are divided into four classes according to the average number of linked entities (threshold 1.5): 1-to-1 — 26.2%, 1-to-Many — 22.7%, Many-to-1 — 28.3%, Many-to-Many — 22.8%. The largest gain from the translation compared with the Unstructured model is seen when the entity to be predicted is on the "single" side of the relation.
  • New relations from few examples. For this experiment, 40 relations were set aside in a separate set, FB15k-40rel. TransE learns them the fastest: just 10 examples yield hits@10 at 18%, and the result rises steadily as the number of examples increases.
  • Future work. The authors propose applying TransE to combine knowledge bases with text. They also mention that they have already integrated the model into a system for relation extraction from text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Google (Canada)Waterloo · Canada5 works · 6,841 citations

Also on the byline: Université de Technologie de Compiègne · Heuristics and Diagnostics for Complex Systems.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Knowledge graphs & ontologies · 192 works

💡 Concepts 2Key ideas and methods explained on our concept pages.Natural language processing · 131 worksKnowledge graph · 49 works

Cite

Cite as Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durán, Jason Weston and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multi-relational Data. NeurIPS 2013.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bordes2013translating,
  title = {{Translating Embeddings for Modeling Multi-relational Data}},
  author = {Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durán, Jason Weston and Oksana Yakhnenko},
  booktitle = {NeurIPS 2013},
  year = {2013},
  url = {https://papers.nips.cc/paper_files/paper/2013/hash/1cecc7a77928ca8133fa24680a88d2f9-Abstract.html}
}

References

17 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, 2008. find
  2. A. Bordes, X. Glorot, J. Weston, and Y. Bengio. A semantic matching energy function for learning with multi-relational data. Machine Learning, 2013. find
  3. A. Bordes, J. Weston, R. Collobert, and Y. Bengio. Learning structured embeddings of knowledge bases. In Proceedings of the 25th Annual Conference on Artificial Intelligence (AAAI), 2011. find
  4. X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS)., 2010. find
  5. R. A. Harshman and M. E. Lundy. Parafac: parallel factor analysis. Computational Statistics & Data Analysis, 18(1):39–72, Aug. 1994. find
  6. R. Jenatton, N. Le Roux, A. Bordes, G. Obozinski, et al. A latent factor model for highly multi-relational data. In Advances in Neural Information Processing Systems (NIPS 25), 2012. find
  7. C. Kemp, J. B. Tenenbaum, T. L. Griffiths, T. Yamada, and N. Ueda. Learning systems of concepts with an infinite relational model. In Proceedings of the 21st Annual Conference on Artificial Intelligence (AAAI), 2006. find
  8. T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (NIPS 26), 2013.
  9. G. Miller. WordNet: a Lexical Database for English. Communications of the ACM, 38(11):39– 41, 1995.
  10. K. Miller, T. Griffiths, and M. Jordan. Nonparametric latent feature models for link prediction. In Advances in Neural Information Processing Systems (NIPS 22), 2009. find
  11. M. Nickel, V. Tresp, and H.-P. Kriegel. A three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on Machine Learning (ICML), 2011. find
  12. M. Nickel, V. Tresp, and H.-P. Kriegel. Factorizing YAGO: scalable machine learning for linked data. In Proceedings of the 21st international conference on World Wide Web (WWW), 2012. find
  13. A. P. Singh and G. J. Gordon. Relational learning via collective matrix factorization. In Proceedings of the 14th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2008. find
  14. R. Socher, D. Chen, C. D. Manning, and A. Y. Ng. Learning new facts from knowledge bases with neural tensor networks and semantic word vectors. In Advances in Neural Information Processing Systems (NIPS 26), 2013. find
  15. I. Sutskever, R. Salakhutdinov, and J. Tenenbaum. Modelling relational data using bayesian clustered tensor factorization. In Advances in Neural Information Processing Systems (NIPS 22), 2009. find
  16. J. Weston, A. Bordes, O. Yakhnenko, and N. Usunier. Connecting language and knowledge bases with embedding models for relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013. find
  17. J. Zhu. Max-margin nonparametric latent feature models for link prediction. In Proceedings of the 29th International Conference on Machine Learning (ICML), 2012. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bordes-2013-translating-embeddings-modeling-multi-relational/