Elamur Research⌕
☰
home / papers / 2014 / Knowledge Graph and Text Jointly Embedding

Knowledge Graph and Text Jointly Embedding

Paper · 2014 · Zhen Wang, Jianwen Zhang, Jianlin Feng et al.

Summary

Embedding methods for knowledge graph completion, such as TransE, can only score a candidate fact when both entities already exist in the graph. Many missing facts involve names or phrases that occur in text but have no graph entry. This paper places Freebase entities and corpus words/phrases in a single vector space. Its probabilistic model has three parts:

  • Knowledge model (pTransE): a softmax version of the TransE translation score, trained on graph triplets.
  • Text model: treats co-occurring words as linked by an unknown relation. It turns out to be nearly equivalent to Skip-Gram.
  • Alignment model: ties the two spaces together using Wikipedia anchors and entity names.

The authors test the approach at scale on Freebase, Wikipedia and NY Times text. Joint training predicts facts more accurately than training graph and text embeddings separately. It can also score facts whose arguments are absent from the graph. When its scores are used as a prior for relation extractors, it improves fact extraction. On analogy benchmarks its word vectors match or slightly beat word2vec. In effect, unlabeled text acts as semi-supervision for knowledge embedding.

Key points

  • Scoring function: a fact (h, r, t) gets the score b − ½‖h + r − t‖², with a bias b = 7 added for numerical stability. Conditional probabilities of head, tail and relation are each normalized with a softmax.
  • Text model: the hidden relation between co-occurring words is absorbed into a second vector per word. This gives 2×|V| vectors, an objective close to Skip-Gram's.
  • Fact extraction: embedding scores are combined with the Mintz and Sm2r extractors on the Freebase+NYTimes benchmark. On entity–entity triplets the joint model is on par with TransE/pTransE. On triplets involving words it clearly outperforms them.
  • Analogy data: the word2vec sets are used (19,544 word and 3,218 phrase analogies), plus analogies built from held-out Freebase triplets.
  • Phrase analogies: anchor-aligned joint embedding reaches 27.6% accuracy and 65.0% Hits@10, against 18.0% and 56.1% for Skip-gram. Name alignment does worse on words and phrases but best on the constructed analogies (11.6% vs 10.5% accuracy with anchor+name).

Interesting ideas

  • I1. Hidden relations in text: reading every co-occurrence in a text window as an unlabeled relation lets one translation-based formulation cover both graph facts and raw text. The same trick could fold unannotated corpora into any relation-learning setup.
  • I2. Cheap alignment signals: hyperlink anchors and entity names, or pseudo-anchors from an entity linker, tie a structured resource to free text at almost no cost. This could suit low-resource languages that have a Wikipedia and a Wikidata-style graph.
  • I3. Anchors as disambiguation: mapping a context word to its linked entity, rather than to an ambiguous surface form, keeps mixed senses from spreading to neighbouring words. This is a general argument for training on entity-linked text.
  • I4. Analogies from a graph: pairing two held-out triplets that share a relation, then replacing entities with their names, gives a relation-specific test set for any language that has a knowledge base.
  • I5. Embeddings as a prior for extractors: adding embedding plausibility to the output of a pattern-based extractor is a simple ensemble that exploits the complementary strengths of the two.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sun Yat-sen UniversityGuangzhou · China23 works · 8,980 citations🏢Microsoft Research (United Kingdom)Cambridge · United Kingdom6 works · 2,344 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1Research subjects and languages the work is about.Knowledge graphs & ontologies · 103 works

Cite

Cite as Zhen Wang, Jianwen Zhang, Jianlin Feng and Zheng Chen. 2014. Knowledge Graph and Text Jointly Embedding. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/d14-1167

@inproceedings{wang2014knowledge,
  title = {{Knowledge Graph and Text Jointly Embedding}},
  author = {Zhen Wang, Jianwen Zhang, Jianlin Feng and Zheng Chen},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1167},
  url = {https://doi.org/10.3115/v1/d14-1167}
}

References

18 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155.
  2. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, pages 1247–1250. ACM. find
  3. Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2011. Learning structured embeddings of knowledge bases. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, pages 301–306. find
  4. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems, pages 2787–2795. find
  5. Kai-Wei Chang, Wen-tau Yih, and Christopher Meek. 2013. Multi-relational latent semantic analysis. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1602–1612, Seattle, Washington, USA, October. Association for Computational Linguistics. find
  6. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537.
  7. Miao Fan, Deli Zhao, Qiang Zhou, Zhiyuan Liu, Thomas Fang Zheng, and Edward Y. Chang. 2014. Distant supervision for relation extraction with matrix completion. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 839–849, Baltimore… find
  8. Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1… find
  9. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  10. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111–3119.
  11. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41.
  12. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2… find
  13. Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In Machine Learning and Knowledge Discovery in Databases, pages 148–163. Springer. find
  14. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In Advances in Neural Information Processing Systems, pages 926–934.
  15. Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D Manning. 2012. Multi-instance multi-label learning for relation extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 455– 46… find
  16. Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, pages 1112–1119. find
  17. Jason Weston, Antoine Bordes, Oksana Yakhnenko, and Nicolas Usunier. 2013. Connecting language and knowledge bases with embedding models for relation extraction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1366–1371, Seattle, Washington, USA, Octo… find
  18. Xingxing Zhang, Jianwen Zhang, Junyu Zeng, Jun Yan, Zheng Chen, and Zhifang Sui. 2013. Towards accurate distant supervision for relational facts extraction. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 810–815, Sofia, Bulg… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2014-knowledge-graph-text-jointly-embedding/