Elamur Research⌕
☰
home / papers / 2019 / Pooled Contextualized Embeddings for Named Entity Recognition

Pooled Contextualized Embeddings for Named Entity Recognition

Summary

Contextual string embeddings, the character-level method behind Flair (Akbik et al., 2018), can embed any word in its context. They struggle, however, when a rare name appears in a sentence that offers few clues. In one CoNLL-03 line, a sports result, "Indra" could plausibly be a person or a team, and the model guesses wrong. The authors propose pooled contextualized embeddings to fix this. As a dataset is processed, each contextual vector produced for a unique string is stored in a memory. A pooling step (mean, min or max) then condenses all stored instances into one global vector. That vector is joined with the current contextual embedding, so the final representation has double the dimensionality and mixes corpus-level with local evidence. The memory is rebuilt at every training epoch and keeps growing during prediction, so representations evolve over time. With a BiLSTM-CRF tagger in Flair, the method reaches new state-of-the-art F1 on CoNLL-03 English, German and Dutch. This shows that cheaply reusing earlier mentions helps disambiguate entities.

Key points

  • CoNLL-03 F1 results: English 93.18 with min pooling (+0.32 pp over the previous best), German 88.27 with min pooling (+0.86 pp) and Dutch 90.44 with mean pooling (+0.28 pp). The comparison set includes BERT, ELMo and CVT+Multitask.
  • On WNUT-17 the method gains nothing significant over its own baseline (49.59 vs 49.49), because emerging entities rarely repeat. It still beats the best previously published result of 45.55.
  • In an ablation without classic word embeddings, pooling helps more. Dutch rises from 86.71 to 88.25 and English from 91.81 to 92.42, which suggests the pooled vectors capture global semantics similar to those of standard word embeddings.
  • The setup uses a BiLSTM-CRF with one LSTM layer of 256 hidden states and SGD with annealing. Scores are averaged over 5 random seeds, and GloVe or FastText word embeddings are stacked in.
  • The code and pre-trained models are released in Flair 0.4.1 as the PooledFlairEmbeddings class.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition

Cited by

99 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 92 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1078

@inproceedings{akbik2019pooled,
  title = {{Pooled Contextualized Embeddings for Named Entity Recognition}},
  author = {Akbik, Alan and Bergmann, Tanja and Vollgraf, Roland},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1078},
  url = {https://doi.org/10.18653/v1/n19-1078}
}

References

14 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gustavo Aguilar, Adrian Pastor Lopez Monroy, Fabio González, and Thamar Solorio. 2018. Modeling noisiness to recognize named entities using multi-task neural networks on social media. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistic… find
  2. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  3. Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1756–1765, Vancouver, Canada. Ass… find our card
  4. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. Flair: An easy-to-use framework for state-of-the-art nlp. In NAACL, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics: System Demonstrations. find our card
  5. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL. find
  6. Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In COLING 2018, 27th International Conference on Computational Linguistics, pages 1638– 1649. find our card
  7. Nils Reimers and Iryna Gurevych. 2017. Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 338–348, Copenhagen, Denmark. find our card
  8. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find
  9. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 142–147. Association for Computational Linguistics. find
  10. Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc V. Le. 2018. Semi-supervised sequence modeling with cross-view training. In EMNLP. find our card
  11. Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140–147. find our card
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  13. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  14. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/akbik-2019-pooled-contextualized-embeddings-named-entity/