Elamur Research
☰
home / papers / 2015 / Named Entity Recognition with Bidirectional LSTM-CNNs

Named Entity Recognition with Bidirectional LSTM-CNNs

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1511.08308

Summary

The paper addresses named entity recognition (NER) for English. Strong NER systems have usually relied on hand-built features and large gazetteers. The authors propose a neural tagger that learns most of these signals on its own. Each token is represented by a pretrained word embedding plus a character-level vector produced by a convolutional layer with max pooling. These representations feed a stacked bidirectional LSTM, and the forward and backward outputs are converted to tag log-probabilities and summed. Dropout is used for regularisation.

The authors also build a large lexicon from DBpedia for the four CoNLL categories. They introduce a new way to encode lexicon hits, including partial matches, as BIOES-style token features, and compare it with the simpler SENNA approach.

Evaluation uses CoNLL-2003 and OntoNotes 5.0. With only tokenized text and public embeddings, the model is competitive on CoNLL-2003 and sets a new best result on OntoNotes. Adding the lexicons gives state-of-the-art scores on both datasets. This shows that a mostly feature-free neural architecture can match or beat heavily engineered systems.

Key points

  • Architecture: BLSTM over words combined with a character-level CNN (25-dimensional random character embeddings).
    • The best word vectors were the 50-dimensional SENNA embeddings from Collobert et al., trained on Wikipedia and Reuters RCV-1.
    • A character-level BLSTM was not significantly better than the CNN and was more expensive to train.
  • Headline results:
    • F1 of 91.62 on CoNLL-2003 and 86.28 on OntoNotes 5.0.
    • Even without lexicons, the model beat the earlier OntoNotes best by 2.13 F1.
  • DBpedia lexicon: about 2.34 million entries, compared with roughly 171k in the SENNA lexicon.
    • A partial match counts when an n-gram matches the beginning or end of an entry and covers at least half its tokens.
  • Training data size: about 205k tokens for CoNLL-2003 and about 1.09M tokens for OntoNotes.
  • Per-genre OntoNotes results: broadcast news scores highest (about 89.9 F1), while telephone conversation is hardest (about 72.4 F1).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Named entity recognition

Cited by

367 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 355 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.08308 · arXiv 1511.08308

@misc{chiu2015named,
  title = {{Named Entity Recognition with Bidirectional LSTM-CNNs}},
  author = {Chiu, Jason P. C. and Nichols, Eric},
  year = {2015},
  doi = {10.48550/arxiv.1511.08308},
  eprint = {1511.08308},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.08308}
}

References

36 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. DBpedia: A nucleus for a web of open data. In The semantic web, pages 722–735. Springer. find
  2. Phil Blunsom, Edward Grefenstette, Nal Kalchbrenner, et al. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. find our card
  3. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111. Association for Comput… find our card
  4. Marc Claesen, Jaak Simm, Dusan Popovic, Yves Moreau, and Bart De Moor. Easy hyperparameter search using Optunity. In Proceedings of the International Workshop on Technical Computing for Machine Learning and Mathematical Engineering. find
  5. Maurice Clerc and James Kennedy. 2002. The particle swarm-explosion, stability, and convergence in a multidimensional complex space. Evolutionary Computation, IEEE Transactions on, 6(1):58–73. find
  6. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, pages 160–167. ACM. find
  7. Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. 2011a. Torch7: A Matlab-like environment for machine learning. In Proceedings of BigLearn, NIPS Workshop, number EPFL-CONF-192376. find
  8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011b. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493– 2537. find our card
  9. Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. Transactions of the Association for Computational Linguistics, 2:477–490. find
  10. Jenny Rose Finkel and Christopher D Manning. 2009. Joint parsing and named entity recognition. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 326–334. Association for Computational Lingui… find
  11. Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 2000. Learning to forget: Continual prediction with LSTM. Neural Computation, 12(10):2451–2471. find
  12. Christoph Goller and Andreas Kuchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In Neural Networks, 1996., IEEE International Conference on, volume 1, pages 347–352. IEEE. find
  13. Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6645–6649. find
  14. James Hammerton. 2003. Named entity recognition with long short-term memory. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 172–175. Association for Computational Linguistics. find
  15. Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. 2012. Lecture 6e: RMSProp: divide the gradient by a running average of its recent magnitude. In Neural Networks for Machine Learning. http: //www.cs.toronto.edu/˜tijmen/csc321/slides/lecture\_slides\_lec6.pdf. find
  16. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: the 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 57–60. Association for Computational Linguistics. find
  17. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. CoRR, abs/1508.01991. find our card
  18. Matthieu Labeau, Kevin Löser, and Alexandre Allauzen. 2015. Non-lexical neural architecture for fine-grained POS tagging. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 232–237. Association for Computational Linguistics. find
  19. Dekang Lin and Xiaoyun Wu. 2009. Phrase clustering for discriminative learning. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1030– 1038. Association for Computational Linguis… find
  20. Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Proc… find our card
  21. Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. 2015. Joint entity recognition and disambiguation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 879–888. Association for Computational Linguistics. find
  22. Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukar Burget, and Jan Cernocky. 2011. RNNLM-recurrent neural network language modeling toolkit. In Proceedings of the 2011 ASRU Workshop, pages 196–201. find
  23. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the Twenty-seventh Annual Conference on Advances in Neural Information Processing Systems, pages 3111–3119. find
  24. Yurii Nesterov. 1983. A method of solving a convex programming problem with convergence rate O(1/k 2 ). Soviet Mathematics Doklady, 27(2):372–376. find
  25. Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 78–86. Association for Computational Linguistics. find
  26. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543. find
  27. G Petasis, S Petridis, G Paliouras, V Karkaletsis, SJ Perantonis, and CD Spyropoulos. 2000. Symbolic and neural learning for named-entity recognition. In Proceedings of the Symposium on Computational Intelligence and Learning, pages 58–66. Citeseer. find
  28. Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. 2014. Dropout improves recurrent neural networks for handwriting recognition. In Proceedings of the 14th International Conference on Frontiers in Handwriting Recognition, pages 285– 290. IEEE. find
  29. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143–152. Associatio… find
  30. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning, pages 147–155. Association for Computational Linguistics. find our card
  31. David Rumelhart, Geoffrey Hinton, and Ronald Williams. 1986. Learning representations by back-propagating errors. Nature, pages 323–533. find
  32. Cıcero Santos, Victor Guimaraes, RJ Niterói, and Rio de Janeiro. 2015. Boosting named entity recognition with neural character embeddings. In Proceedings of the Fifth Named Entities Workshop, pages 25–33. find
  33. Jun Suzuki, Hideki Isozaki, and Masaaki Nagata. 2011. Learning condensed feature representations from large unsupervised data sets for supervised learning. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers, pages 636… find
  34. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. Association for Computational Linguistics. find
  35. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394. Association for Computational Linguistics. find our card
  36. Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701. find our card