Elamur Research⌕
☰
home / papers / 2016 / Neural Architectures for Named Entity Recognition

Neural Architectures for Named Entity Recognition

Summary

The paper addresses named entity recognition (NER). Strong systems for this task usually depend on hand-crafted features and language-specific resources, such as gazetteers, which are expensive to build for each new language. The authors propose two neural architectures that require only a small annotated corpus and unannotated text. The first, LSTM-CRF, combines a bidirectional LSTM with a CRF layer that jointly selects the tags for the entire sentence. The second, S-LSTM, uses a transition-based algorithm in the style of shift-reduce parsers together with a Stack-LSTM: it constructs text chunks and labels them immediately. The model views each word from two sides. A character-level model captures how the word is spelled, while pretrained embeddings indicate the contexts in which it occurs. Dropout prevents the model from relying on only one of these sources. The models were evaluated on the CoNLL-2002/2003 corpora in English, Dutch, German, and Spanish. LSTM-CRF achieved state-of-the-art results at the time for three languages and near state-of-the-art results for English. The work is significant because it shows that high-quality NER can be achieved without language-specific resources. This is valuable for languages where such resources are scarce, including Ukrainian.

Key points

  • F1 for LSTM-CRF: English (CoNLL-2003) — 90.94, German — 78.76, Dutch (CoNLL-2002) — 81.74, Spanish — 85.75.
  • F1 for S-LSTM is slightly lower: 90.33, 75.66, 79.88, and 83.93, respectively. Nevertheless, for several languages it also surpassed previously published results.
  • Contribution of individual LSTM-CRF components (English): pretrained embeddings add +7.31 F1, the CRF layer +1.79, dropout +1.17, and character-based word representations about +0.74.
  • Tagging schemes: the authors used IOBES but observed no noticeable gain over IOB.
  • Open-source code: github.com/glample/tagger (LSTM-CRF) and github.com/clab/stack-lstm-ner (S-LSTM).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1030 · arXiv 1603.01360

@inproceedings{lample2016neural,
  title = {{Neural Architectures for Named Entity Recognition}},
  author = {Lample, Guillaume and Ballesteros, Miguel and Subramanian, Sandeep and Kawakami, Kazuya and Dyer, Chris},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1030},
  eprint = {1603.01360},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n16-1030}
}

References

38 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rie Kubota Ando and Tong Zhang. 2005a. A framework for learning predictive structures from multiple tasks and unlabeled data. The Journal of Machine Learning Research, 6:1817–1853. find
  2. Rie Kubota Ando and Tong Zhang. 2005b. Learning predictive structures. JMLR, 6:1817–1853. find
  3. Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based dependency parsing by modeling characters instead of words with LSTMs. In Proceedings of EMNLP. find
  4. Miguel Ballesteros, Yoav Golderg, Chris Dyer, and Noah A. Smith. 2016. Training with Exploration Improves a Greedy Stack-LSTM Parser. In arXiv:1603.03793. find
  5. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166. find
  6. Chris Biemann, Gerhard Heyer, Uwe Quasthoff, and Matthias Richter. 2007. The leipzig corpora collection-monolingual corpora of standard size. Proceedings of Corpus Linguistic. find
  7. Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar F Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and Met… find our card
  8. Jason PC Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional lstm-cnns. arXiv preprint arXiv:1511.08308. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machi… find our card
  9. Silviu Cucerzan and David Yarowsky. 1999. Language independent named entity recognition combining morphological and contextual evidence. In Proceedings of the 1999 Joint SIGDAT Conference on EMNLP and VLC, pages 90–99. find
  10. Silviu Cucerzan and David Yarowsky. 2002. Language independent ner using a unified model of internal and contextual evidence. In proceedings of the 6th conference on Natural language learning-Volume 20, pages 1–4. Association for Computational Linguistics. find
  11. Hong-Jie Dai, Po-Ting Lai, Yung-Chun Chang, and Richard Tzong-Han Tsai. 2015. Enhancing of chemical compound and drug name recognition using representative tag scheme and fine-grained tokenization. Journal of cheminformatics, 7(Suppl 1):S14. find
  12. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proc. find our card
  13. ACL. Jacob Eisenstein, Tae Yano, William W Cohen, Noah A Smith, and Eric P Xing. 2011. Structured databases of named entities from bayesian nonparametrics. In Proceedings of the First Workshop on Unsupervised Learning in NLP, pages 2–12. Association for Computational Linguistics. find
  14. Radu Florian, Abe Ittycheriah, Hongyan Jing, and Tong Zhang. 2003. Named entity recognition through classifier combination. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 168–171. Association for Computational Linguistics. find our card
  15. Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2015. Multilingual language processing from bytes. arXiv preprint arXiv:1512.00103. find
  16. David Graff. 2011. Spanish gigaword third edition (ldc2011t12). Linguistic Data Consortium, University of Pennsylvania, Philadelphia, PA. Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM networks. In Proc. find
  17. IJCNN. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Co… find
  18. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. CoRR, abs/1508.01991. find our card
  19. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2015. Character-aware neural language models. CoRR, abs/1508.06615. find our card
  20. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  21. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proc. find
  22. ICML. Dekang Lin and Xiaoyun Wu. 2009. Phrase clustering for discriminative learning. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, pages 1030–1038. Association f… find
  23. Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. 2015. Joint named entity recognition and disambiguation. In Proc. find
  24. EMNLP. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find our card
  25. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Proc. find our card
  26. NIPS. Joakim Nivre. 2004. Incrementality in deterministic dependency parsing. In Proceedings of the Workshop on Incremental Parsing: Bringing Engineering and Cognition Together. find our card
  27. Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R Curran. 2013. Learning multilingual named entity recognition from wikipedia. Artificial Intelligence, 194:151–175. find
  28. Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2009. English gigaword fourth edition (ldc2009t13). Linguistic Data Consortium, Univer-sity of Pennsylvania, Philadelphia, PA. find
  29. Alexandre Passos, Vineet Kumar, and Andrew Mc-Callum. 2014. Lexicon infused phrase embeddings for named entity resolution. arXiv preprint arXiv:1404.5367. find our card
  30. Yanjun Qi, Ronan Collobert, Pavel Kuksa, Koray Kavukcuoglu, and Jason Weston. 2009. Combining labeled and unlabeled data with word-class distribution learning. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 1737–1740. find
  31. ACM. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning, pages 147–155. Association for Computational Linguistics. Cicero Nogueira dos Santos and Victor Guimarães. 2… find our card
  32. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proc. find our card
  33. CoNLL. Erik F. Tjong Kim Sang. 2002. Introduction to the conll-2002 shared task: Language-independent named entity recognition. In Proc. find our card
  34. CoNLL. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proc. find our card
  35. ACL. Matthew D Zeiler. 2012. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701. find our card
  36. Yue Zhang and Stephen Clark. 2011. Syntactic processing using the generalized perceptron and beam search. Computational Linguistics, 37(1). find
  37. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, pages 649–657. find our card
  38. Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lample-2016-neural-architectures-named-entity-recognition/