Elamur Research
☰
home / papers / 2016 / End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF

End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1603.01354

Summary

Strong sequence labelers for tasks like POS tagging and NER have traditionally depended on manually designed features, gazetteers and task-specific preprocessing. These are expensive to build and hard to transfer to new tasks or domains. Ma and Hovy propose BLSTM-CNNs-CRF, a neural architecture that needs only pre-trained word embeddings.

The model works in three stages:

  • A character-level CNN, with dropout on its inputs, builds a vector for each word that captures morphology such as prefixes and suffixes.
  • This vector is concatenated with the word embedding and passed to a bi-directional LSTM, which encodes left and right context.
  • A CRF layer then decodes the full label sequence jointly instead of predicting each tag independently.

The authors test the model on Penn Treebank WSJ for POS tagging and on the English CoNLL 2003 data for NER. It beats earlier state-of-the-art systems on both. Its value is practical: because it uses no hand-engineered knowledge, the same recipe should carry over to other languages, domains and labeling tasks with little adaptation effort.

Key points

  • Results: 97.55% accuracy on WSJ POS tagging and 91.21% F1 on CoNLL 2003 NER, both without feature engineering or data preprocessing.
  • Embeddings: the main experiments use Stanford's GloVe (100 dimensions, trained on 6 billion words). The authors also compare Senna (50 dimensions), Google's Word2Vec (300 dimensions) and randomly initialized vectors.
  • Training setup:
    • Implemented in Theano and trained on one GeForce GTX TITAN X.
    • Training took about 12 hours for POS tagging and about 8 hours for NER.
    • Optimizer: SGD with momentum 0.9 and batch size 10.
    • Gradient clipping at 5.0.
  • Dropout on the BLSTM's input and output vectors clearly improves both tasks by reducing overfitting.
  • Error analysis by vocabulary subset (IV, OOTV, OOEV, OOBV) shows the CRF layer brings its biggest gains on OOBV words, which appear in neither the training set nor the embedding vocabulary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Named entity recognition · Part-of-speech tagging

Cited by

646 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 625 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1603.01354 · arXiv 1603.01354

@misc{ma2016end,
  title = {{End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF}},
  author = {Ma, Xuezhe and Hovy, Eduard},
  year = {2016},
  doi = {10.48550/arxiv.1603.01354},
  eprint = {1603.01354},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1603.01354}
}

References

61 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. The Journal of Machine Learning Research, 6:1817–1853. find
  2. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166. find
  3. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a cpu and gpu math expression compiler. In Proceedings of the Python for scientific computing conference (SciPy), volume 4,… find
  4. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of EMNLP-2014, pages 740–750, Doha, Qatar, October. find our card
  5. Hai Leong Chieu and Hwee Tou Ng. 2002. Named entity recognition: a maximum entropy approach using global information. In Proceedings of CoNLL-2003, pages 1–7. find
  6. Jason PC Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional lstm-cnns. arXiv preprint arXiv:1511.08308. find our card
  7. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–decoder approaches. Syntax, Semantics and Structure in Statistical Translation, page 103. find our card
  8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. find our card
  9. Hong-Jie Dai, Po-Ting Lai, Yung-Chun Chang, and Richard Tzong-Han Tsai. 2015. Enhancing of chemical compound and drug name recognition using representative tag scheme and fine-grained tokenization. Journal of cheminformatics, 7(S1):1–10. find
  10. Yann N Dauphin, Harm de Vries, Junyoung Chung, and Yoshua Bengio. 2015. Rmsprop and equilibrated adaptive learning rates for non-convex optimization. arXiv preprint arXiv:1502.04390. — [arXiv:1502.04390](https://arxiv.org/abs/1502.04390) find
  11. Cıcero dos Santos, Victor Guimaraes, RJ Niterói, and Rio de Janeiro. 2015. Boosting named entity recognition with neural character embeddings. In Proceedings of NEWS 2015 The Fifth Named Entities Workshop, page 25. find
  12. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of ACL-2015 (Volume 1: Long Papers), pages 334–343, Beijing, China, July. find our card
  13. Radu Florian, Abe Ittycheriah, Hongyan Jing, and Tong Zhang. 2003. Named entity recognition through classifier combination. In Proceedings of HLT-NAACL-2003, pages 168–171. find
  14. Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 2000. Learning to forget: Continual prediction with lstm. Neural computation, 12(10):2451–2471. find
  15. Felix A Gers, Nicol N Schraudolph, and Jürgen Schmidhuber. 2003. Learning precise timing with lstm recurrent networks. The Journal of Machine Learning Research, 3:115–143. find
  16. Rich Caruana Steve Lawrence Lee Giles. 2001. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference, volume 13, page 402. MIT Press. find
  17. Jesús Giménez and Lluı́s Màrquez. 2004. Svmtool: A general pos tagger generator based on support vector machines. In In Proceedings of LREC-2004. find
  18. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In International conference on artificial intelligence and statistics, pages 249–256. find
  19. Christoph Goller and Andreas Kuchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In Neural Networks, 1996., IEEE International Conference on, volume 1, pages 347–352. IEEE. find
  20. Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Proceedings of ICASSP-2013, pages 6645–6649. IEEE. find
  21. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 1026–1034. find
  22. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  23. Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard H. Hovy, and Eric P. Xing. 2016. Harnessing deep neural networks with logic rules. In Proceedings of ACL-2016, Berlin, Germany, August. find
  24. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991. find our card
  25. Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 2342–2350. find
  26. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  27. Terry Koo and Michael Collins. 2010. Efficient third-order dependency parsers. In Proceedings of ACL-2010, pages 1–11, Uppsala, Sweden, July. find
  28. Matthieu Labeau, Kevin Löser, Alexandre Allauzen, and Rue John von Neumann. 2015. Non-lexical neural architecture for fine-grained pos tagging. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 232–237. find
  29. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of ICML-2001, volume 951, pages 282–289. find
  30. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL-2016, San Diego, California, USA, June. find
  31. Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. 1989. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551. find
  32. Dekang Lin and Xiaoyun Wu. 2009. Phrase clustering for discriminative learning. In Proceedings of ACL-2009, pages 1030–1038. find
  33. Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of EMNLP-2015, pages 1520–1530, Lisbon, Portugal, September. find our card
  34. Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. 2015. Joint entity recognition and disambiguation. In Proceedings of EMNLP-2015, pages 879–888, Lisbon, Portugal, September. find
  35. Xuezhe Ma and Eduard Hovy. 2015. Efficient inner-to-outer greedy algorithm for higher-order labeled dependency parsing. In Proceedings of the EMNLP-2015, pages 1322–1328, Lisbon, Portugal, September. find
  36. Xuezhe Ma and Fei Xia. 2014. Unsupervised dependency parsing with transferring distribution via parallel guidance and entropy regularization. In Proceedings of ACL-2014, pages 1337–1348, Baltimore, Maryland, June. find
  37. Xuezhe Ma and Hai Zhao. 2012a. Fourth-order dependency parsing. In Proceedings of COLING 2012: Posters, pages 785–796, Mumbai, India, December. find
  38. Xuezhe Ma and Hai Zhao. 2012b. Probabilistic models for high-order projective dependency parsing. Technical Report, arXiv:1502.04174. — [arXiv:1502.04174](https://arxiv.org/abs/1502.04174) find
  39. Xuezhe Ma, Zhengzhong Liu, and Eduard Hovy. 2016. Unsupervised ranking model for entity coreference resolution. In Proceedings of NAACL-2016, San Diego, California, USA, June. find
  40. Christopher D Manning. 2011. Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In Computational Linguistics and Intelligent Text Processing, pages 171–189. Springer. find our card
  41. Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313– 330. find our card
  42. Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Online large-margin training of dependency parsers. In Proceedings of ACL-2005, pages 91–98, Ann Arbor, Michigan, USA, June 25-30. find our card
  43. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  44. Vincent Ng. 2010. Supervised noun phrase coreference research: The first fifteen years. In Proceedings of ACL-2010, pages 1396–1411, Uppsala, Sweden, July. Association for Computational Linguistics. find our card
  45. Joakim Nivre and Mario Scholz. 2004. Deterministic dependency parsing of English text. In Proceedings of COLING-2004, pages 64–70, Geneva, Switzerland, August 23-27. find our card
  46. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063. find
  47. Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. In Proceedings of CoNLL-2014, pages 78–86, Ann Arbor, Michigan, June. find
  48. Nanyun Peng and Mark Dredze. 2015. Named entity recognition for chinese social media with jointly trained embeddings. In Proceedings of EMNLP-2015, pages 548–554, Lisbon, Portugal, September. find
  49. Nanyun Peng and Mark Dredze. 2016. Improving named entity recognition for chinese social media with word segmentation representation learning. In Proceedings of ACL-2016, Berlin, Germany, August. find
  50. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP-2014, pages 1532–1543, Doha, Qatar, October. find
  51. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of CoNLL-2009, pages 147–155. find our card
  52. Cicero D Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proceedings of ICML-2014, pages 1818–1826. find
  53. Libin Shen, Giorgio Satta, and Aravind Joshi. 2007. Guided learning for bidirectional sequence classification. In Proceedings of ACL-2007, volume 7, pages 760–767. find
  54. Anders Søgaard. 2011. Semi-supervised condensed nearest neighbor for part-of-speech tagging. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 48– 52, Portland, Oregon, USA, June. find
  55. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929– 1958. find
  56. Xu Sun. 2014. Structure regularization for structured prediction. In Advances in Neural Information Processing Systems, pages 2402–2410. find
  57. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of CoNLL-2003 - Volume 4, pages 142–147, Stroudsburg, PA, USA. find
  58. Erik F. Tjong Kim Sang and Jorn Veenstra. 1999. Representing text chunks. In Proceedings of EACL’99, pages 173–179. Bergen, Norway. find
  59. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of NAACL-HLT-2003, Volume 1, pages 173–180. find
  60. Zhilin Yang, Ruslan Salakhutdinov, and William Cohen. 2016. Multi-task cross-lingual sequence tagging from scratch. arXiv preprint arXiv:1603.06270. — [arXiv:1603.06270](https://arxiv.org/abs/1603.06270) find
  61. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. find our card