Elamur Research⌕
☰
home / papers / 2019 / Learning Dense Representations for Entity Retrieval

Learning Dense Representations for Entity Retrieval

Summary

Entity linking systems typically begin with an alias table. The table maps surface names to a limited set of candidate Wikipedia entities, and a separate model then re-ranks those candidates. Such tables ignore context, miss unusual names such as colloquial party names, and prune candidates by popularity. As a result, the correct entity may never be considered.

The authors replace this stage with a learned dual encoder (two-tower) model. One tower encodes a mention together with its surrounding text. The other encodes an entity from its title, first paragraph and categories. Both map into a shared vector space, where candidates are found by approximate nearest neighbor search. Training uses only Wikipedia anchor-text links, combined with an unsupervised hard negative mining scheme that gives large gains over random negatives.

The system beats alias-table and BM25 retrieval and comes close to the best comparable TACKBP-2010 results. It also transfers well to a new Wikinews test set. This shows fully learned retrieval can practically replace heuristic two-stage pipelines.

Key points

  • Training data: the 2018-10-22 English Wikipedia dump, with 5.7M entities and 112.7M hyperlinked mentions. Of this, 99.9% is used for training and the rest for model selection.
  • Architecture: compound sub-encoders on each side, scored by cosine similarity.
    • Mention side: span, left/right context (five tokens each) and sentence.
    • Entity side: title, first paragraph and categories.
    • Text is represented by averaged unigram/bigram embeddings (300-d) followed by feed-forward layers.
  • Evaluation: disambiguation only, with NIL mentions ignored and no in-domain fine-tuning.
    • TACKBP-2010: 1020 mention/entity pairs from 1013 documents.
    • New Wikinews 2018 set: 2263 pairs from 1801 documents, released on GitHub (google-research).
  • Speed: retrieval over all 5.7M entities takes about 3 ms per mention, compared with prior systems reporting seconds per mention.
  • Qualitative analysis: the model uses context and world knowledge. Changing the context around ambiguous names such as Paris, Texas or Phoenix shifts the retrieved neighbors. It also finds entities the alias table misses, for example Communist Refoundation Party.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

103 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 94 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k19-1049

@inproceedings{gillick2019learning,
  title = {{Learning Dense Representations for Entity Retrieval}},
  author = {Gillick, Daniel and Kulkarni, Sayali and Lansing, Larry and Presta, Alessandro and Baldridge, Jason and Ie, Eugene and Garcia-Olano, Diego},
  booktitle = {Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)},
  year = {2019},
  doi = {10.18653/v1/k19-1049},
  url = {https://doi.org/10.18653/v1/k19-1049}
}

References

29 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexandr Andoni and Piotr Indyk. 2008. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. Communications of the ACM 51(1):117. find
  2. Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. In Transactions of the ACL. find our card
  3. Yotam Eshel, Noam Cohen, Kira Radinsky, Shaul Markovitch, Ikuya Yamada, and Omer Levy. 2017. Named entity disambiguation for noisy text. In CoNLL. Vancouver, Canada, pages 58–68. find
  4. Feng Nie, Yunbo Cao, Jinpeng Wang, Chin-Yew Lin, and Rong Pan. 2018. Mention and entity description co-attention for entity disambiguation. In AAAI. Vancouver, Canada, pages 5908–5915. find
  5. Matthew Francis-Landau, Greg Durrett, and Dan Klein. 2016. Capturing semantic similarity for entity linking with convolutional neural networks. In NAACL-HLT. San Diego, California, pages 1256–1261. find
  6. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In In EMNLP. find
  7. Kuzman Ganchev and Mark Dredze. 2008. Small statistical models by random feature mixing. In Proceedings of the ACL-08: HLT Workshop on Mobile Language Processing. pages 19–20. find
  8. Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. CoRR abs/1704.04920. find
  9. Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. 2018. End-to-end retrieval in continuous space . find
  10. Amir Globerson, Nevena Lazic, Soumen Chakrabarti, Amarnag Subramanya, Michael Ringaard, and Fernando Pereira. 2016. Collective entity resolution with multi-focal attention. In ACL. find
  11. Ruiqi Guo, Sanjiv Kumar, Krzysztof Choromanski, and David Simcha. 2016. Quantization based fast inner product search. In Artificial Intelligence and Statistics. pages 482–490. find
  12. Nitish Gupta, Sameer Singh, and Dan Roth. 2017. Entity linking via joint encoding of types, descriptions, and context. In EMNLP. pages 2681–2690. find
  13. Ben Hachey, Will Radford, Joel Nothman, Matthew Honnibal, and James R Curran. 2013. Evaluating entity linking with wikipedia. Artificial intelligence 194:130–150. find
  14. Zhengyan He, Shujie Liu, Mu Li, Ming Zhou, Longkai Zhang, and Houfeng Wang. 2013. Learning entity representation for entity disambiguation. In ACL. Sofia, Bulgaria, pages 30–34. find
  15. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. arXiv preprint arXiv:1705.00652 . — [arXiv:1705.00652](https://arxiv.org/abs/1705.00652) find
  16. Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi, and Noah A. Smith. 2017. Dynamic entity representations in neural language models. CoRR abs/1708.00781. find
  17. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410 . find
  18. Xiao Ling, Sameer Singh, and Daniel S Weld. 2015. Design challenges for entity linking. Transactions of the ACL 3:315–328. find
  19. Lev Ratinov, Dan Roth, Doug Downey, and Mike Anderson. 2011. Local and global algorithms for disambiguation to wikipedia. In ACL-HLT. Stroudsburg, PA, USA, pages 1375–1384. find
  20. Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laurent Charlin, and Joelle Pineau. 2018. A survey of available corpora for building data-driven dialogue systems: The journal version. D&D 9(1):1–49. find
  21. Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. 2016. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 761–769. find
  22. Avirup Sil, Gourab Kundu, Radu Florian, and Wael Hamza. 2018. Neural cross-lingual entity linking. In AAAI. Vancouver, Canada, pages 5465–5472. find
  23. Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015. Modeling mention, context and entity with neural networks for entity disambiguation. In Twenty-Fourth International Joint Conference on Artificial Intelligence. find
  24. Ricardo Usbeck, Michael Röder, Axel-Cyrille Ngonga Ngomo, Ciro Baron, Andreas Both, Martin Brümmer, Diego Ceccarelli, Marco Cornolti, Didier Cherix, Bernd Eickmann, et al. 2015. In Proceedings of the 24th international conference on World Wide Web. pages 1133–1143. find
  25. Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint learning of the embedding of words and entities for named entity disambiguation. arXiv preprint arXiv:1601.01343 . find our card
  26. Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2017a. Learning distributed representations of texts and entities from knowledge base. TACL 5:397–411. find
  27. Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2017b. Learning distributed representations of texts and entities from knowledge base. CoRR abs/1705.02494. find
  28. Wen-tau Yih, Kristina Toutanova, John C Platt, and Christopher Meek. 2011. Learning discriminative projections for text similarity measures. In CoNLL. pages 247–256. find our card
  29. Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip S. Yu. 2018. Generative discovery of relational medical entity pairs. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gillick-2019-learning-dense-representations-entity-retrieval/