Elamur Research⌕
☰
home / papers / 2019 / E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

Paper · 2019 · Nina Poerner, Ulli Waltinger and Hinrich Schutze

Summary

Pretrained language models like BERT hold some factual knowledge, but earlier ways of adding entity knowledge (ERNIE, KnowBert) insert new encoder layers and need costly extra pretraining. E-BERT avoids that cost. The authors train Wikipedia2Vec, which places words and Wikipedia entities in one shared space. They then fit a linear map from Wikipedia2Vec word vectors to BERT's wordpiece embeddings, using the words both vocabularies have in common. The same map projects entity vectors into BERT's input space, so BERT receives them as ordinary wordpiece vectors with its encoder unchanged. Entities can either sit next to their surface wordpieces (E-BERT-concat) or replace them (E-BERT-replace). E-BERT achieves a new state of the art on the unsupervised LAMA benchmark and beats BERT and other baselines on relation classification and entity linking. The authors also show that plain BERT often guesses from how a name sounds rather than from real facts. To measure this, they built LAMA-UHN, a harder subset of LAMA.

Key points

  • Wikipedia2Vec was trained on a 2019-09-02 Wikipedia dump, giving 2.7M entities. This took about 6 hours on 32 CPUs with no GPU, whereas KnowBert required 1.25M training steps across as many as four Titan RTX GPUs.
  • LAMA (Google-RE, T-REx subsets) is scored with Hits@k, which the authors use instead of the name "P@k". E-BERT outperforms BERT, ERNIE and KnowBert on it.
  • LAMA-UHN (UnHelpfulNames) drops questions whose answers can be guessed from the entity name. For example, BERT says the French actor Daniel Ceccaldi speaks Italian.
  • For entity linking on AIDA, E-BERT-MLM uses iterative refinement with up to 3 passes. It reaches 85.0 micro F1 on AIDA-B, compared with 82.4 for Kolitsas et al. (2018) and 73.7 for KnowBert.
  • The alignment step matters for entity linking. Before finetuning, aligned vectors give about 31–34 dev F1, while non-aligned Wikipedia2Vec baselines get about 2. The non-aligned baselines also converge more slowly during finetuning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ludwig-Maximilians-Universität MünchenMunich · Germany113 works · 56,817 citations

Also on the byline: Siemens (Germany).

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Nina Poerner, Ulli Waltinger and Hinrich Schutze. 2019. E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT. arXiv:1911.03681.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1911.03681 · arXiv 1911.03681

@misc{poerner2019e,
  title = {{E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT}},
  author = {Nina Poerner, Ulli Waltinger and Hinrich Schutze},
  year = {2019},
  doi = {10.48550/arxiv.1911.03681},
  eprint = {1911.03681},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1911.03681}
}

References

31 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel Broscheit. 2019. Investigating entity knowledge in bert with simple neural end-to-end entity linking. In CoNLL, pages 677–685, Hong Kong, China.
  2. Haotian Chen, Sahil Wadhwa, Xi David Li, and Andrej Zukov-Gregoric. 2019. YELM: End-to-end contextualized entity linking. arXiv preprint arXiv:1911.03834. — [arXiv:1911.03834](https://arxiv.org/abs/1911.03834) find
  3. Rajarshi Das, Manzil Zaheer, Siva Reddy, and Andrew McCallum. 2017. Question answering on knowledge bases and text using universal schema and memory networks. In ACL, pages 358–365, Vancouver, Canada. find
  4. Joe Davison, Joshua Feldman, and Alexander M Rush. 2019. Commonsense knowledge mining from pre-trained models. In EMNLP-IJCNLP, pages 1173– 1178, Hong Kong, China.
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171–4186, Minneapolis, USA.
  6. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A Smith. 2019. Show your work: Improved reporting of experimental results. In EMNLP-IJCNLP, pages 2185–2194, Hong Kong, China. find
  7. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frédérique Laforest, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In LREC, pages 3448–3452, Miyazaki, Japan.
  8. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-Predict: Parallel decoding of conditional masked language models. In EMNLP-IJCNLP, pages 6114–6123, Hong Kong, China.
  9. Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In EMNLP, pages 782–792, Edinburgh, UK. find
  10. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2019. How can we know what language models know? arXiv preprint arXiv:1911.12543.
  11. Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas Hofmann. 2018. End-to-end neural entity linking. In CoNLL, pages 519–529, Brussels, Belgium. find
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  13. Ilya Loshchilov and Frank Hutter. 2018. Fixing weight decay regularization in adam. find
  14. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press. find
  15. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  16. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168. find
  17. Matthew E Peters, Mark Neumann, IV Logan, L Robert, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. 2019. Knowledge enhanced contextual word representations. In EMNLP-IJCNLP, Hong Kong, China. find
  18. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? In EMNLP-IJCNLP, Hong Kong, China.
  19. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  20. Timo Schick and Hinrich Schütze. 2019. BERTRAM: Improved word embeddings have big impact on contextualized model performance. arXiv preprint arXiv:1910.07181. find
  21. Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In ICLR, Toulon, France.
  22. Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. In EMNLP, pages 4231–4242, Brussels, Belgium. find
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998–6008, Long Beach, USA.
  24. Hai Wang, Dian Yu, Kai Sun, Janshu Chen, and Dong Yu. 2019a. Improving pre-trained multilingual models with vocabulary expansion. In CoNLL, pages 316–327, Hong Kong, China. find
  25. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Cuihong Cao, Daxin Jiang, Ming Zhou, et al. 2020. K-Adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint arXiv:2002.01808.
  26. Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Liwei Peng, and Luo Si. 2019b. StructBERT: Incorporating language structures into pre-training for deep language understanding. arXiv preprint arXiv:1908.04577.
  27. Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2019c. KEPLER: A unified model for knowledge embedding and pre-trained language representation. arXiv preprint arXiv:1911.06136. find
  28. Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint learning of the embedding of words and entities for named entity disambiguation. In CoNLL, Berlin, Germany.
  29. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In NeurIPS, pages 5754– 5764.
  30. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In ACL, pages 1441–1451, Florence, Italy. find
  31. E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (Appendix) Unsupervised QA (LAMA) Data We downloaded the LAMA dataset from https://dl.fbaipublicfiles.com/LAMA/data.zip . We use the LAMA-T-REx and LAMA-Google-RE relations, which are aimed at factual knowledge. Table 9 shows results on indi… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/poerner-2019-e-bert-efficient-yet-effective/