Elamur Research
☰
home / papers / 2018 / Contextual String Embeddings for Sequence Labeling

Contextual String Embeddings for Sequence Labeling

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/C18-1139.pdf

Summary

Sequence labeling tasks such as NER, chunking and PoS tagging depend heavily on word embeddings. Each existing type covers only part of what is useful. Pre-trained word vectors ignore context, character features are learned only from task data, and contextual embeddings from word-level LMs are costly to train. The authors introduce contextual string embeddings, built from a bidirectional character-level LSTM language model pre-trained on unlabeled text. For each word, they concatenate two hidden states: the forward model's state right after the word's final character and the backward model's state just before its first character. The resulting vectors change with the surrounding text and treat words purely as character sequences, which helps with rare or misspelled forms. Fed into a standard BiLSTM-CRF tagger, they set new state-of-the-art results on CoNLL03 NER for English and German. They also slightly beat prior bests on CoNLL2000 chunking and Penn treebank PoS tagging. The character LM is cheap enough to retrain for new languages or domains, and all code and models are released in the Flair framework.

Key points

  • CoNLL03 NER: 93.09 F1 for English (previous best 92.22, Peters et al., 2018) and about 88.3 F1 for German (previous best 78.76, Lample et al., 2016), a gain of roughly 9.5 points.
  • On saturated tasks the gains are small: 96.72 on CoNLL2000 chunking (vs. 96.37) and 97.85 accuracy on Penn treebank PoS tagging (vs. 97.64).
  • When the BiLSTM is swapped for a simple feedforward map (Map-CRF), scores drop only about 3% on average with the proposed embeddings. With classic embeddings such as GloVe, Komninos or FastText, the drop is about 20%. This suggests the new vectors carry rich information on their own.
  • Nearest-neighbor checks on the word "Washington" in CoNLL03 show the embeddings separate person, place, legislative body and sports team readings depending on context.
  • The character LM trains in about 1 week on 1 GPU, compared with 5 weeks on 32 GPUs for the word-level LM used by Peters et al. Code and models are available at github.com/zalandoresearch/flair.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition · Part-of-speech tagging

Cited by

466 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 454 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{akbik2018contextual,
  title = {{Contextual String Embeddings for Sequence Labeling}},
  author = {Akbik, Alan and Blythe, Duncan and Vollgraf, Roland},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  url = {https://aclanthology.org/C18-1139/}
}

References

33 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005. find our card
  2. Jinho D Choi. 2016. Dynamic feature induction: The last gist to the state-of-the-art. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 271–281. find
  3. Michael Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pages 1–8. Association for Computational Linguistics. find our card
  4. Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2015. Multilingual language processing from bytes. arXiv preprint arXiv:1512.00103. find
  5. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). find our card
  6. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. find
  7. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2016. A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587. find our card
  8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  9. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991. https://github.com/zalandoresearch/flair ↗ our card
  10. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. find
  11. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078. find our card
  12. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2015. Character-aware neural language models. arXiv preprint arXiv:1508.06615. find our card
  13. Alexandros Komninos and Suresh Manandhar. 2016. Dependency based embeddings for sentence classification tasks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1490–1500. find
  14. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360. find
  15. Liyuan Liu, Jingbo Shang, Frank Xu, Xiang Ren, Huan Gui, Jian Peng, and Jiawei Han. 2017. Empower sequence labeling with task-aware neural language model. arXiv preprint arXiv:1709.04109. find
  16. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTMs-CNNs-CRF. arXiv preprint arXiv:1603.01354. find our card
  17. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. find our card
  18. Scott Menard. 2018. Applied logistic regression analysis, volume 106. SAGE publications. find
  19. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  20. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  21. Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1756–1765, Vancouver, Canada, Jul… find our card
  22. Matthew Peters, Mark Neumann, and Christopher Clark Kenton Lee Luke Zettlemoyer Mohit Iyyer, Matt Gardner. 2018. Deep contextualized word representations. 6th International Conference on Learning Representations. find
  23. Yanjun Qi, Ronan Collobert, Pavel Kuksa, Koray Kavukcuoglu, and Jason Weston. 2009. Combining labeled and unlabeled data with word-class distribution learning. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 1737–1740. ACM. find
  24. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444. find our card
  25. Nils Reimers and Iryna Gurevych. 2017. Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 338–348, Copenhagen, Denmark, 09. find our card
  26. Ronald Rosenfeld. 2000. Two decades of statistical language modeling: Where do we go from here? Proceedings of the IEEE, 88(8):1270–1278. find
  27. Dominic Seyler, Tatiana Dembelova, Luciano Del Corro, Johannes Hoffart, and Gerhard Weikum. 2017. Knowner: Incremental multilingual knowledge in named entity recognition. arXiv preprint arXiv:1709.03544. — [arXiv:1709.03544](https://arxiv.org/abs/1709.03544) find
  28. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 231–235. find our card
  29. Ilya Sutskever, James Martens, and Geoffrey E Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 1017–1024. find
  30. Ilya Sutskever, Oriol Vinyals, and Quoc Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  31. Erik F Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task: Chunking. In Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning-Volume 7, pages 127–132. Association for Computational Linguistics… find
  32. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 142–147. Association for Computational Linguistics. find
  33. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329. find our card