Elamur Research⌕
☰
home / papers / 2016 / Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation

Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation

Summary

Named Entity Disambiguation (NED) links ambiguous mentions in text, such as "Washington", to the correct entry in a knowledge base, here Wikipedia. The authors argue that both local textual cues and document-level coherence can come from one vector space shared by words and entities. They build this space by extending the skip-gram model with two extra training objectives. The KB graph model pulls together entities whose incoming Wikipedia links overlap, which is an embedding counterpart of WLM. The anchor context model treats Wikipedia hyperlinks as unambiguous entity occurrences and trains each linked entity to predict its surrounding words. All three objectives are optimized jointly on Wikipedia using negative sampling. Cosine similarities from the resulting space then serve as features for context similarity and coherence. These features are combined with standard signals, such as prior probability, in a supervised NED system. The system reached state-of-the-art results on the CoNLL and TAC 2010 benchmarks. This shows that a single, fairly simple shared embedding can effectively capture both kinds of evidence that NED needs.

Key points

  • The joint objective sums three skip-gram-style parts: word–word (L_w), entity–entity over the Wikipedia link graph (L_e), and entity–word from anchor contexts (L_a). Training uses SGD with negative sampling, in a lock-free, multiprocess Python/Cython/NumPy implementation.
  • Reported accuracy is 93.1% on the CoNLL dataset and 85.2% on TAC 2010, both above earlier state-of-the-art methods.
  • In the feature study, the embedding-based textual context features gave a large improvement. Coherence and the two-step approach helped on CoNLL (about 20 mentions per document) but not on TAC 2010 (about one mention per document).
  • In the CoNLL error analysis, about 48.6% of errors involved metonymy mentions, such as "South Africa" referring to the national rugby team. Roughly 9.6% concerned entities that fewer than 10 anchors pointed to, while 4.5% concerned entities that no anchor linked to at all.
  • The entity vectors were also evaluated separately on an entity relatedness dataset and proved to be good representations. Planned future work includes adding knowledge-graph relations, for example from Freebase.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition

Cited by

95 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 94 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k16-1025 · arXiv 1601.01343

@inproceedings{yamada2016joint,
  title = {{Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation}},
  author = {Yamada, Ikuya and Shindo, Hiroyuki and Takeda, Hideaki and Takefuji, Yoshiyasu},
  booktitle = {Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning},
  year = {2016},
  doi = {10.18653/v1/k16-1025},
  eprint = {1601.01343},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/k16-1025}
}

References

34 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roi Blanco, Giuseppe Ottaviano, and Edgar Meij. 2015. Fast and Space-Efficient Entity Linking for Queries. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining (WSDM), pages 179–188. find
  2. Zhiting Hu, Poyao Huang, Yuntian Deng, Yingkai Gao, and Eric Xing. 2015. Entity Hierarchy Embedding. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP) (Volume 1: Long Paper… find
  3. Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2011. Learning Structured Embeddings of Knowledge Bases. In Proceedings of the 25th Conference on Artificial Intelligence (AAAI), pages 301–306. find
  4. Razvan Bunescu and Marius Pasca. 2006. Using Encyclopedic Knowledge for Named Entity Disambiguation. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 9–16. find our card
  5. Hongzhao Huang, Larry Heck, and Heng Ji. 2015. Leveraging Deep Neural Networks and Knowledge Graphs for Entity Disambiguation. CoRR, abs/1504.0. find
  6. Diego Ceccarelli, Claudio Lucchese, Salvatore Orlando, Raffaele Perego, and Salvatore Trani. 2013. Learning Relatedness Measures for Entity Linking. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM), pages 139–148. find
  7. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):422– 446. find
  8. Heng Ji, Ralph Grishman, Hoa Trang Dang, Kira Griffitt, and Joe Ellis. 2010. Overview of the TAC 2010 Knowledge Base Population Track. In Proceeding of Text Analytics Conference (TAC). find
  9. Andrew Chisholm and Ben Hachey. 2015. Entity Disambiguation with Web Links. Transactions of the Association for Computational Linguistics, 3:145– 156. find
  10. Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning Entity and Relation Embeddings for Knowledge Graph Completion. In Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI). find
  11. Silviu Cucerzan. 2007. Large-Scale Named Entity Disambiguation Based on Wikipedia Data. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 708–716. find
  12. Xiao Ling, Sameer Singh, and Daniel S. Weld. 2015. Design Challenges for Entity Linking. Transactions of the Association for Computational Linguistics, 3:315–328. find
  13. Jerome H. Friedman. 2001. Greedy Function Approximation: A Gradient Boosting Machine. The Annals of Statistics, 29(5):1189–1232. find
  14. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press. find
  15. Evgeniy Gabrilovich, Michael Ringgaard, and Amarnag Subramanya. 2013. FACC1: Freebase annotation of ClueWeb corpora, Version 1 (Release date 2013-06-26, Format version 1, Correction level 0). find
  16. P McNamee and HT Dang. 2009. Overview of the TAC 2009 Knowledge Base Population Track. In Proceeding of Text Analysis Conference (TAC). find
  17. Zhaochen Guo and Denilson Barbosa. 2014. Entity Linking with a Unified Semantic Representation. In Proceedings of the 23rd International Conference on World Wide Web (WWW), pages 1305–1310. find
  18. Edgar Meij, Wouter Weerkamp, and Maarten de Rijke. 2012. Adding Semantics to Microblog Posts. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining (WSDM), pages 563–572. find
  19. Zhengyan He, Shujie Liu, Mu Li, Ming Zhou, Longkai Zhang, and Houfeng Wang. 2013. Learning Entity Representation for Entity Disambiguation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL) (Volume 2: Short Papers), pages 30–34. find
  20. Rada Mihalcea and Andras Csomai. 2007. Wikify!: Linking Documents to Encyclopedic Knowledge. In Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management (CIKM), pages 233–242. find
  21. Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust Disambiguation of Named Entities in Text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)… find
  22. Tomas Mikolov, Greg Corrado, Kai Chen, and Jeffrey Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. In Proceedings of the International Conference on Learning Representations (ICLR), pages 1–12. find
  23. Johannes Hoffart, Stephan Seufert, Dat Ba Nguyen, Martin Theobald, and Gerhard Weikum. 2012. KORE: Keyphrase Overlap Relatedness for Entity Disambiguation. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management (CIKM), pages 545–554. find
  24. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013b. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems (NIPS), pages 3111–3119. find
  25. David Milne and Ian H. Witten. 2008a. An Effective, Low-Cost Measure of Semantic Relatedness Obtained from Wikipedia Links. In Proceedings of the First AAAI Workshop on Wikipedia and Artificial Intelligence (WIKIAI). find
  26. David Milne and Ian H. Witten. 2008b. Learning to Link with Wikipedia. In Proceeding of the 17th ACM Conference on Information and Knowledge Management (CIKM), pages 509–518. find
  27. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  28. Maria Pershina, Yifan He, and Ralph Grishman. 2015. Personalized Page Rank for Named Entity Disambiguation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 238–243. find
  29. Lev Ratinov, Dan Roth, Doug Downey, and Mike Anderson. 2011. Local and Global Algorithms for Disambiguation to Wikipedia. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, volume 1, pages 1375–1384. find
  30. Sameer Singh, Amarnag Subramanya, Fernando Pereira, and Andrew McCallum. 2012. Wikilinks: A Large-scale Cross-Document Coreference Corpus Labeled via Links to Wikipedia. Technical Report UM-CS-2012-015. find
  31. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning With Neural Tensor Networks for Knowledge Base Completion. In Advances in Neural Information Processing Systems (NIPS), pages 926–934. find
  32. Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015. Modeling Mention, Context and Entity with Neural Networks for Entity Disambiguation. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI), pages 1333–1339. find
  33. Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge Graph and Text Jointly Embedding. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1591–1601. find
  34. Yadollah Yaghoobzadeh and Hinrich Schütze. 2015. Corpus-level fine-grained entity typing using contextual information. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 715–725. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yamada-2016-joint-learning-embedding-words-entities/