Elamur Research⌕
☰
home / papers / 2019 / Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking

Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking

Paper · 2019 · Samuel Broscheit

Summary

End-to-end entity linking usually runs as a pipeline: mention detection (MD), candidate generation (CG) and entity disambiguation (ED). This paper asks whether BERT can learn all three at once, how much entity knowledge pretrained BERT already holds, and whether adding entity knowledge helps other tasks.

The author treats entity linking as classification of each token over roughly 700K entities, the most frequent ones in English Wikipedia, with "O" meaning Nil. Training data comes from intra-Wikipedia hyperlinks, with adjustments for links that are missing from the annotation. BERT-base-uncased is first trained this way (BERT+Entity) and then fine-tuned on CoNLL03/AIDA.

The model beats a pipeline baseline that handles the steps separately and ranks second only to the joint MD+ED system of Kolitsas et al. (2018). It also learns noticeably more entity knowledge than a frozen-encoder variant.

In downstream tasks (GLUE, SQUAD V2, SWAG, WMT14 EN-DE), the extra entity knowledge mostly gives no benefit. The exception is RTE, a task with little training data. The work shows that a single neural model can do linking without pipelines or heuristics. It also shows that current benchmarks rarely test entity knowledge.

Key points

  • Setup: Entity linking is framed as per-token classification with a sigmoid output layer over about 700K Wikipedia entities. MD, CG and ED are solved together, with no BIO span supervision.
  • Frozen vs. trained encoder: Frozen-BERT+Entity, where only the classifier is trained, already scores 13–16% above plain BERT trained on CoNLL03/AIDA. Full BERT+Entity adds another 6–10%.
  • Error analysis: On validation data, only 3% of errors come purely from wrong spans. Most errors are wrong Nil predictions, which often involve rare entities. A second training setting (Setting II) caps the number of examples per entity, which favours rarer entities, and brings results close to Kolitsas et al. (2018).
  • Downstream results: Ensembling BERT+Entity with BERT improves RTE from 71.48 to 73.64. Other GLUE tasks, SQUAD V2 and SWAG stay roughly flat, and WMT14 EN-DE BLEU drops slightly (22.51 → 22.20).
  • Test-set drop: Performance falls from AIDA/testa to AIDA/testb more sharply for BERT+Entity than for the other models. The author suspects overfitting, possibly to sports-related documents.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MannheimMannheim · Germany18 works · 6,984 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1 Named entity recognition · 101 works

Cite

Cite as Samuel Broscheit. 2019. Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking. Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k19-1063

@inproceedings{broscheit2019investigating,
  title = {{Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking}},
  author = {Samuel Broscheit},
  booktitle = {Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)},
  year = {2019},
  doi = {10.18653/v1/k19-1063},
  url = {https://doi.org/10.18653/v1/k19-1063}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iz Beltagy, Arman Cohan, and Kyle Lo. 2019. Scibert: Pretrained contextualized embeddings for scientific text. CoRR, abs/1903.10676. our card
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  3. Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. TACL, 2:477–490. our card
  4. Yuhang Guo, Bing Qin, Yuqin Li, Ting Liu, and Sheng Li. 2013. Improving candidate generation for entity linking. In Natural Language Processing and Information Systems - 18th International Conference on Applications of Natural Language to Information Systems, NLDB 2013, Salford, UK, June 19-21, 2013… find
  5. Ben Hachey, Will Radford, Joel Nothman, Matthew Honnibal, and James R. Curran. 2013. Evaluating entity linking with wikipedia. Artif. Intell., 194:130– 150. find
  6. Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, E… find
  7. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 328–339. find
  8. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find
  9. Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas Hofmann. 2018. End-to-end neural entity linking. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 519–529, Brussels, Belgium. Association for Computational Linguistics. find
  10. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. Biobert: a pre-trained biomedical language representation model for biomedical text mining. CoRR, abs/1901.08746. our card
  11. Dat Ba Nguyen, Martin Theobald, and Gerhard Weikum. 2016. J-NERD: joint named entity recognition and disambiguation with rich linguistic features. TACL, 4:215–229. find
  12. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. our card
  13. Maria Pershina, Yifan He, and Ralph Grishman. 2015. Personalized page rank for named entity disambiguation. In NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, May 31 - June 5, 2015… find
  14. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… our card
  15. Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? adapting pre-trained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP, RepL4NLP@ACL 2019, Florence, Italy, August 2, 2019., pages 7–14. find
  16. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  17. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 784–789. our card
  18. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4593–4601. our card
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017,… our card
  20. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2018, Brussels,… our card
  21. Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint learning of the embedding of words and entities for named entity disambiguation. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016… our card
  22. Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2017. Learning distributed representations of texts and entities from knowledge base. TACL, 5:397–411. find
  23. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 93– 104. find
  24. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: enhanced language representation with informative entities. to appear. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/broscheit-2019-investigating-entity-knowledge-bert-simple/