Elamur Research⌕
☰
home / papers / 2017 / Entity Linking via Joint Encoding of Types, Descriptions, and Context

Entity Linking via Joint Encoding of Types, Descriptions, and Context

Paper · 2017 · Nitish Gupta, Sameer Singh and Dan Roth

Summary

The paper tackles entity linking: mapping a textual mention to the correct knowledge-base entity, using one system that works across domains without dataset-specific training or hand-engineered features. The authors build a modular neural model that learns one dense vector per entity, trained so it agrees with several separate encoders. These encoders cover the mention's local context (two LSTMs reading toward the mention from the left and from the right), the document context (a bag of mention surface strings passed through a feed-forward layer), the entity's Wikipedia description, and its fine-grained Freebase types. Supervision comes only indirectly, from Wikipedia links and Freebase. The resulting linker is competitive with, and sometimes better than, state-of-the-art systems across benchmarks, including systems trained on in-domain data. Because the entity vector is tied to encoders of descriptions and types, new entities with no linked mentions can be embedded without retraining the rest of the model. This matters for KBs that grow constantly. The same embeddings also support fine-grained type prediction.

Key points

  • Each entity has a single embedding that is jointly optimised against four signals: mention context, description, entity types and mention types. The training data are Wikipedia mentions, Wikipedia descriptions and Freebase fine-grained types.
  • Model variants are compared by information source: CD (context + description), CT (context + types) and the full CDTE.
  • New-entity test setup:
    • 1000 rare Wikipedia entities unseen in training, with 3791 ambiguous mentions and about 6 candidates per mention.
    • No priors are available, so linking relies on context alone.
    • Using entity information cuts errors by 47–60% compared with random initial embeddings.
    • Combining description and types reaches roughly 80% accuracy.
  • Fine-grained typing on FIGER (GOLD): the model scores 57.7 accuracy, 72.8 macro F1 and 72.1 micro F1. Its accuracy beats SSIR-LSTM (55.6) but trails SSIR-Full, which adds attention and hand-crafted features.
  • Code and datasets are released at nitishgupta.github.io/neural-el.

Interesting ideas

  • I1. Compositional embedding training. Rather than feeding all features into one network, each information source gets its own encoder, and the entity vector is pushed to be close to every encoder's output. This keeps the system modular and tolerant of missing sources. It would suit any KB where coverage is uneven, such as Ukrainian entities with sparse Wikipedia links.
  • I2. Cold-start by freezing the model. A new entity is added by optimising only its own vector against its description and type encodings, with everything else fixed. This is a cheap way to extend a linker or a product/place catalogue without retraining.
  • I3. Types as auxiliary distant supervision. Fine-grained types are not used as a hard filtering step. Instead, they serve as an extra training loss that shapes the mention-context encoder. Multi-task signals like this can improve representations even when they are never used at inference time.
  • I4. Separate left and right LSTMs ending at the mention. Each side is read toward the target, so the same sentence yields different representations for different mentions in it. This is a simple design for any mention-centred classification task.
  • I5. Ablation by information source plus qualitative examples. The CD, CT and CDTE variants are compared on cases where only one kind of knowledge is decisive. This evaluation pattern shows clearly what each source contributes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States170 works · 60,739 citations🎓University of California, IrvineIrvine · United States29 works · 36,049 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1 Named entity recognition · 101 works

Cite

Cite as Nitish Gupta, Sameer Singh and Dan Roth. 2017. Entity Linking via Joint Encoding of Types, Descriptions, and Context. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1284

@inproceedings{gupta2017entity,
  title = {{Entity Linking via Joint Encoding of Types, Descriptions, and Context}},
  author = {Nitish Gupta, Sameer Singh and Dan Roth},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1284},
  url = {https://doi.org/10.18653/v1/d17-1284}
}

References

38 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Luisa Bentivogli, Pamela Forner, Claudio Giuliano, Alessandro Marchetti, Emanuele Pianta, and Kateryna Tymoshenko. 2010. Extending english ace 2005 corpus annotation with ground-truth links to wikipedia. In Proceedings of the International Conference on Machine Learning (ICML). find
  2. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In The Conference on Advances in Neural Information Processing Systems (NIPS). find
  3. Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2011. Learning structured embeddings of knowledge bases. In Proceedings of the National Conference on Artificial Intelligence (AAAI). find
  4. Razvan Bunescu and Marius Pasca. 2006. Using encyclopedic knowledge for named entity disambiguation. In Proceedings of the European Chapter of the ACL (EACL). our card
  5. Xiao Cheng and Dan Roth. 2013. Relational inference for wikification. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP). our card
  6. Silviu Cucerzan. 2007. Large-scale named entity disambiguation based on Wikipedia data. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 708–716. find
  7. Rajarshi Das, Arvind Neelakantan, David Belanger, and Andrew McCallum. 2017. Chains of reasoning over entities, relations, and text using recurrent neural networks. In Proceedings of the European Chapter of the ACL (EACL). find
  8. Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. Transactions of the Association for Computational Linguistics (TACL). our card
  9. Matthew Francis-Landau, Greg Durrett, and Dan Klein. 2016. Capturing semantic similarity for entity linking with convolutional neural networks. In Proceedings of the Annual Meeting of the North American Association of Computational Linguistics (NAACL). find
  10. Amir Globerson, Nevena Lazic, Soumen Chakrabarti, Amarnag Subramanya, Michael Ringgaard, and Fernando Pereira. 2016. Collective entity resolution with multi-focal attention. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  11. Hannaneh Hajishirzi, Leila Zilles, Daniel S Weld, and Luke S Zettlemoyer. 2013. Joint coreference resolution and named-entity linking with multi-pass sieves. In Proceedings of the Conference on Empirical Methods for Natural Language Processing (EMNLP). our card
  12. Zhengyan He, Shujie Liu, Mu Li, Ming Zhou, Longkai Zhang, and Houfeng Wang. 2013. Learning entity representation for entity disambiguation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  13. Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the Conference on Empirical Methods for Natural Language Processing (EMNLP… find
  14. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  15. Sayali Kulkarni, Amit Singh, Ganesh Ramakrishnan, and Soumen Chakrabarti. 2009. Collective annotation of wikipedia entities in web text. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. find
  16. Nevena Lazic, Amarnag Subramanya, Michael Ringgaard, and Fernando Pereira. 2015. Plato: A selective context model for entity resolution. Transactions of the Association for Computational Linguistics (TACL). find
  17. Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the National Conference on Artificial Intelligence (AAAI). find
  18. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In The Conference on Advances in Neural Information Processing Systems (NIPS). our card
  19. Xiao Ling, Sameer Singh, and Daniel S Weld. 2015. Design challenges for entity linking. Transactions of the Association for Computational Linguistics (TACL). find
  20. Valentin I Spitkovsky and Angel X Chang. 2012. A cross-lingual dictionary for english wikipedia concepts. In LREC. find
  21. Xiao Ling and Daniel S Weld. 2012. Fine-grained entity recognition. In Proceedings of the National Conference on Artificial Intelligence (AAAI). find
  22. Paul McNamee, Mark Dredze, Adam Gerber, Nikesh Garera, Tim Finin, James Mayfield, Christine D Piatko, Delip Rao, David Yarowsky, and Markus Dreyer. 2009. Hltcoe approaches to knowledge base population at tac 2009. In TAC. find
  23. Rada Mihalcea and Andras Csomai. 2007. Wikify!: Linking documents to encyclopedic knowledge. In Proceedings of ACM Conference on Information and Knowledge Management (CIKM). find
  24. David Milne and Ian H. Witten. 2008. Learning to link with Wikipedia. In Proceedings of ACM Conference on Information and Knowledge Management (CIKM), pages 509–518. find
  25. Thien Huu Nguyen, Nicolas Fauceglia, Mariano Rodriguez-Muro, Oktie Hassanzadeh, Alfio Massimiliano Gliozzo, and Mohammad Sadoghi. 2016. Joint learning of local and global features for entity linking via neural networks. In Proceedings the International Conference on Computational Linguistics (COLING… find
  26. NIST. 2005. The ACE evaluation plan. find
  27. US NIST. 2004. The ace evaluation plan. US National Institute for Standards and Technology (NIST). find
  28. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods for Natural Language Processing (EMNLP). our card
  29. L. Ratinov, D. Roth, D. Downey, and M. Anderson. 2011. Local and global algorithms for disambiguation to wikipedia. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  30. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research. find
  31. Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015. Modeling mention, context and entity with neural networks for entity disambiguation. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). find
  32. Kristina Toutanova, Xi Victoria Lin, Wen-tau Yih, Hoifung Poon, and Chris Quirk. 2016. Compositional learning of embeddings for relation paths in knowledge bases and text. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  33. Patrick Verga, David Belanger, Emma Strubell, Benjamin Roth, and Andrew McCallum. 2016a. Multilingual relation extraction using compositional universal schema. In Proceedings of the Annual Meeting of the North American Association of Computational Linguistics (NAACL). find
  34. Patrick Verga, Arvind Neelakantan, and Andrew Mc-Callum. 2016b. Generalizing to unseen entities and entity pairs with row-less universal schema. In Proceedings of the European Chapter of the ACL (EACL). find
  35. Michael Wick, Sameer Singh, Harshal Pandya, and Andrew McCallum. 2013. A joint model for discovering and linking entities. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction. find
  36. Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575. our card
  37. Sonse Shimaoka, Pontus Stenetorp, Kentaro Inui, and Sebastian Riedel. 2017. Neural architectures for fine-grained entity type classification. In Proceedings of the European Chapter of the ACL (EACL). find
  38. Sameer Singh, Amarnag Subramanya, Fernando Pereira, and Andrew McCallum. 2012. Wikilinks: A large-scale cross-document coreference corpus labeled via links to Wikipedia. Technical report, University of Massachusetts, Amherst. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gupta-2017-entity-linking-joint-encoding-types/