Elamur Research⌕
☰
home / papers / 2020 / Entities as Experts: Sparse Memory Access with Entity Supervision

Entities as Experts: Sparse Memory Access with Entity Supervision

Summary

The paper asks how a language model can store factual knowledge about real-world entities inside its own parameters, rather than relying on external knowledge bases. The authors propose Entities as Experts (EaE), a Transformer with an entity memory layer placed between its lower and upper blocks. Unlike ERNIE or KnowBERT, which plug in fixed embeddings trained elsewhere, EaE learns its entity embeddings jointly with the rest of the network.

EaE is pre-trained on English Wikipedia with three combined objectives: BIO mention detection, entity linking supervised by hyperlinks, and masked prediction of whole entity mentions. For each detected mention, the model retrieves a weighted mix of the closest entity vectors and adds it back into the token stream. Because only the memories of mentioned entities are used, access to the memory is sparse.

On TriviaQA, EaE beats an encoder-generator Transformer roughly ten times its size, and on the LAMA probes it holds more facts than a comparably sized BERT. The approach matters because it shows that compact, entity-centric parameter sharing can match or exceed much larger models while touching only a small share of their weights at inference.

Key points

  • Architecture: 4 Transformer layers, then the entity memory layer, then 8 more layers, followed by two task heads (TokenPred and EntityPred). Mention detection is built in, so no external detector is needed at inference.
  • Pre-training data: English Wikipedia with a 1m-entity vocabulary, giving 32m contexts, 17m hyperlinked entity links and 140m mentions (from hyperlinks plus the Google Cloud Natural Language API). 20% of mentions are fully masked.
  • Sparse retrieval: The model attends over all entities during training and the top 100 at inference. Cutting this to 10 entities barely hurts accuracy, and on TriviaQA EaE outperforms T5-3B while using about 3% of its parameters.
  • Learned vs. fixed embeddings: In tests with 200k entities, learning the entity embeddings beats keeping TransE or Deep-Ed embeddings frozen. Initializing from Deep-Ed gives further gains on most knowledge probes.
  • TACRED relation extraction (F1): EaE (366m parameters) is compared with KnowBERT (523m).
    • Revised split: 80.6 vs. 79.3
    • Weighted split: 61.3 vs. 58.4
    • Original split: 70.2 vs. 71.5 (slightly behind)
  • Limitation: EaE cannot represent entities that are absent from its fixed vocabulary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

75 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 68 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.400

@inproceedings{fevry2020entities,
  title = {{Entities as Experts: Sparse Memory Access with Entity Supervision}},
  author = {Févry, Thibault and Soares, Livio Baldini and FitzGerald, Nicholas and Choi, Eunsol and Kwiatkowski, Tom},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.400},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.400}
}

References

45 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2619–2629, Copenhagen, Denmark. Association for Computational Linguistics. find
  2. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrieval-augmented language model pre-training. arXiv preprint 2002.08909. find
  3. Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, and Yoshua Bengio. 2016. A neural knowledge language model. arXiv preprint 1608.00318. find
  4. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. find
  5. Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. TACRED revisited: A thorough evaluation of the TACRED relation extraction task. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1558– 1569, Online. Association for Computational Linguist… find our card
  6. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. arXiv preprint 1702.08734. find our card
  7. Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2895–2905, Florence, Italy. Association for Comp… find
  8. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544. find
  9. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NeurIPS. find
  10. Kyunghyun Cho and Yoshua Bengio. 2014. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint 1406.7362. — [arXiv:1406.7362](https://arxiv.org/abs/1406.7362) find
  11. Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. 2018. Ultra-fine entity typing. In ACL. find
  12. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NeurIPS. find
  13. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint 1705.03551. find our card
  14. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint 1412.6980. find
  15. Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2019. Large memory layers with product keys. In NeurIPS. find
  16. Anne Lauscher, Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš. 2019. Informing unsupervised pretraining with external linguistic knowledge. arXiv preprint 1909.02339. — [arXiv:1909.02339](https://arxiv.org/abs/1909.02339) find
  17. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. arXiv preprint 1906.00300. find our card
  18. Yoav Levine, Barak Lenz, Or Dagan, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2019. Sensebert: Driving some sense into bert. arXiv preprint 1908.05646. find our card
  19. Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2020. Question and answer test-train overlap in open-domain question answering datasets. find
  20. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1810.04805. find
  21. Jeffrey Ling, Nicholas FitzGerald, Zifei Shan, Livio Baldini Soares, Thibault Févry, David Weiss, and Tom Kwiatkowski. 2020. Learning cross-context entity representations from text. arXiv preprint 2001.03765. — [arXiv:2001.03765](https://arxiv.org/abs/2001.03765) find
  22. David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever. 2013. Learning factored representations in a deep mixture of experts. arXiv preprint 1312.4314. — [arXiv:1312.4314](https://arxiv.org/abs/1312.4314) find
  23. Robert Logan, Nelson F Liu, Matthew E Peters, Matt Gardner, and Sameer Singh. 2019. Baracks wife hillary: Using knowledge graphs for fact-aware language modeling. In ACL. find
  24. Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a. A discrete hard em approach for weakly supervised question answering. arXiv preprint 1909.04849. find our card
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Neurips. find
  26. Sewon Min, Danqi Chen, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019b. Knowledge guided text retrieval and reading for open domain question answering. arXiv preprint 1911.03868. find
  27. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Cuihong Cao, Daxin Jiang, Ming Zhou, et al. 2020. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint 2002.01808. find our card
  28. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint 1802.05365. find
  29. Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. arXiv preprint 1410.3916. find
  30. Matthew E Peters, Mark Neumann, Robert L Logan, IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. 2019. Knowledge enhanced contextual word representations. arXiv. find
  31. Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2019a. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. arXiv preprint 1912.09637. find
  32. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint 1909.01066. find our card
  33. Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2019b. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. arXiv preprint 1912.09637. find
  34. Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2019. Bert is not a knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised qa. arXiv preprint 1911.03681. find
  35. Zichao Yang, Phil Blunsom, Chris Dyer, and Wang Ling. 2016. Reference-aware language models. arXiv preprint 1611.01628. find
  36. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint 1910.10683. find
  37. Parikshit Ram and Alexander G Gray. 2012. Maximum inner-product search using cone trees. In International conference on Knowledge discovery and data mining. find
  38. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D Manning. 2017. Position-aware attention and supervised data improve slot filling. In EMNLP. find
  39. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. Ernie: Enhanced language representation with informative entities. arXiv preprint 1905.07129. find
  40. Michael Ringgaard, Rahul Gupta, and Fernando CN Pereira. 2017. Sling: A framework for frame semantic parsing. arXiv preprint 1710.07032. — [arXiv:1710.07032](https://arxiv.org/abs/1710.07032) find
  41. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? ArXiv, abs/2002.08910. find
  42. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint 1701.06538. find
  43. Fumin Shen, Wei Liu, Shaoting Zhang, Yang Yang, and Heng Tao Shen. 2015. Learning binary codes for maximum inner product search. In ICCV. find
  44. Anshumali Shrivastava and Ping Li. 2014. Asymmetric lsh (alsh) for sublinear time maximum inner product search (mips). In NeurIPS. find
  45. Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In NeurIPS. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fevry-2020-entities-as-experts-sparse-memory/