Elamur Research
☰
home / papers / 2020 / Scalable Zero-shot Entity Linking with Dense Entity Retrieval

Scalable Zero-shot Entity Linking with Dense Entity Retrieval

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.519.pdf

Summary

Entity linking means mapping each mention in a text to an entry in a knowledge base that may hold millions of entities. Earlier systems narrowed this huge space with external resources such as alias tables, link popularity or Wikipedia categories. This paper shows that BERT models can do the job in a zero-shot setting, where test entities never appear in training and each entity is known only through its title and a short description.

The method, released as BLINK, has two stages:

  • A bi-encoder embeds the mention context and the entity text separately. It retrieves candidates by dot-product nearest-neighbour search over precomputed entity vectors.
  • A cross-encoder reads the mention and each candidate jointly and re-ranks a small shortlist.

Training uses in-batch negatives plus hard negatives. The system sets new state-of-the-art results on the Wikia zero-shot corpus, WikilinksNED Unseen-Mentions and TACKBP-2010, without hand-built heuristics. It also stays fast when linking against all of Wikipedia. This makes accurate, large-scale linking feasible for new domains that lack curated resources.

Key points

  • Accuracy gains: about 6 points absolute on the Wikia zero-shot benchmark, over 7 points on WikilinksNED Unseen-Mentions, and more than 30% relative error reduction on TACKBP-2010.
  • Speed at scale: linking against 5.9M Wikipedia entities takes roughly 2 ms per mention with approximate nearest-neighbour search via FAISS. HNSW indexes reach 2.6 ms with under 1.2% accuracy loss, or 1.4 ms with under 2.1% loss.
  • Input setup:
    • Mention context is capped at about 32 word-pieces, with special tokens marking the mention span.
    • Wikipedia entities are described by their first ten sentences.
    • Retrieving k = 10 candidates for re-ranking gave the best overall accuracy.
  • Knowledge distillation: using the cross-encoder as teacher improves the bi-encoder. Accuracy rises from 74.4 to 75.7 on Unseen-Mentions (teacher: 76.8) and from 92.9 to 93.5 on TACKBP-2010 (teacher: 94.5).
  • Release: code and models are public at github.com/facebookresearch/BLINK, including a ready-to-use Wikipedia linker.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Topics: Named entity recognition

Cited by

242 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 234 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.519 · arXiv 1911.03814

@inproceedings{wu2020scalable,
  title = {{Scalable Zero-shot Entity Linking with Dense Entity Retrieval}},
  author = {Wu, Ledell and Petroni, Fabio and Josifoski, Martin and Riedel, Sebastian and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.519},
  eprint = {1911.03814},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.519}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cristian Buciluundefined, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’06, page 535–541, New York, NY, USA. Association for Computing Machinery. find
  2. Yixin Cao, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2018. Neural collective entity linking. In Proceedings of the 27th International Conference on Computational Linguistics, pages 675–686. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  4. Yotam Eshel, Noam Cohen, Kira Radinsky, Shaul Markovitch, Ikuya Yamada, and Omer Levy. 2017. Named entity disambiguation for noisy text. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 58–68, Vancouver, Canada. Association for Computational Lingui… find
  5. Paolo Ferragina and Ugo Scaiella. 2011. Fast and accurate annotation of short texts with wikipedia pages. IEEE software, 29(1):70–75. find
  6. Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2619–2629, Copenhagen, Denmark. Association for Computational Linguistics. find
  7. Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019. Learning dense representations for entity retrieval. arXiv preprint arXiv:1909.10506. find
  8. Amir Globerson, Nevena Lazic, Soumen Chakrabarti, Amarnag Subramanya, Michael Ringaard, and Fernando Pereira. 2016. Collective entity resolution with multi-focal attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 621–6… find
  9. Nitish Gupta, Sameer Singh, and Dan Roth. 2017. Entity linking via joint encoding of types, descriptions, and context. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2681–2690. find
  10. Zhengyan He, Shujie Liu, Mu Li, Ming Zhou, Longkai Zhang, and Houfeng Wang. 2013. Learning entity representation for entity disambiguation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 30–34, Sofia, Bulgaria. Association f… find
  11. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. stat, 1050:9. find our card
  12. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. find
  13. Heng Ji, Ralph Grishman, Hoa Trang Dang, Kira Griffitt, and Joe Ellis. 2010. Overview of the tac 2010 knowledge base population track. In Third Text Analysis Conference (TAC 2010), volume 3, pages 3–3. find
  14. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data. find our card
  15. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense passage retrieval for open-domain question answering. find our card
  16. Sammy Khalife and Michalis Vazirgiannis. 2018. Scalable graph-based individual named entity identification. CoRR, abs/1811.10547. — [arXiv:1811.10547](https://arxiv.org/abs/1811.10547) find
  17. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015. find
  18. Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas Hofmann. 2018. End-to-end neural entity linking. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 519–529, Brussels, Belgium. Association for Computational Linguistics. find
  19. Adam Lerer, Ledell Wu, Jiajun Shen, Timothée Lacroix, Luca Wehrstedt, Abhijit Bose, and Alexander Peysakhovich. 2019. Pytorch-biggraph: A large-scale graph embedding system. In Proceedings of the 2nd SysML Conference. find
  20. Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity descriptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3449–3460, Florence, Italy. Associatio… find
  21. David Mueller and Greg Durrett. 2018. Effective use of context in noisy entity linking. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1024–1029. find