Elamur Research
☰
home / papers / 2015 / Bidirectional LSTM-CRF Models for Sequence Tagging

Bidirectional LSTM-CRF Models for Sequence Tagging

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1508.01991

Summary

This 2015 Baidu Research paper looks at sequence tagging (part-of-speech tagging, chunking and named entity recognition) with recurrent neural networks instead of linear models such as HMM, MEMM and CRF. The authors compare five architectures under one SGD training setup: a plain LSTM, a bidirectional LSTM, a standalone CRF, an LSTM with a CRF output layer, and a bidirectional LSTM with a CRF layer (BI-LSTM-CRF). In the BI-LSTM-CRF, the forward and backward LSTM passes read context on both sides of each word. The CRF layer adds a learned tag-transition matrix and scores whole tag sequences, so labels are not chosen one position at a time. The authors say this is the first use of BI-LSTM-CRF on standard NLP tagging benchmarks. Tested on Penn TreeBank POS, CoNLL 2000 chunking and CoNLL 2003 NER, it matches or nearly matches the best published systems. It also stays strong with randomly initialized embeddings, so it depends less on pretrained word vectors than the earlier convolutional approach. The architecture went on to become a standard neural baseline for tagging.

Key points

  • Five models (LSTM, BI-LSTM, CRF, LSTM-CRF, BI-LSTM-CRF) are tested on three benchmarks: Penn TreeBank (PTB) POS, CoNLL 2000 chunking and CoNLL 2003 NER. Inputs combine word embeddings with spelling and context features.
  • With Senna embeddings, BI-LSTM-CRF scores 97.55% POS accuracy, which beat the earlier systems the paper lists. It reaches 94.46 F1 on chunking, behind only Shen and Sarkar's 95.23. On NER it reaches 90.10 F1 with a gazetteer, compared with 90.90 for Passos et al. (2014).
  • With random embeddings, BI-LSTM-CRF clearly beats Collobert et al.'s Conv-CRF on NER (84.26 vs. 81.47 F1). With Senna embeddings the gap narrows to 90.10 vs. 89.59.
  • When only word features are used, the BI-LSTM-CRF chunking F1 drops by just 0.06, while the standalone CRF loses about 8.5 points.
  • Training uses batches of sentences totaling about 100 tokens. Hidden states are reset at the start of every sentence, and the transition matrix is learned together with the LSTM weights.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Named entity recognition

Cited by

724 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 704 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1508.01991 · arXiv 1508.01991

@misc{huang2015bidirectional,
  title = {{Bidirectional LSTM-CRF Models for Sequence Tagging}},
  author = {Huang, Zhiheng and Xu, Wei and Yu, Kai},
  year = {2015},
  doi = {10.48550/arxiv.1508.01991},
  eprint = {1508.01991},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1508.01991}
}

References

35 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. K. Ando and T. Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research (JMLR). find
  2. M. Boden. 2002. A Guide to Recurrent Neural Networks and Back-propagation. In the Dallas project. find
  3. H. L. Chieu. 2003. Named entity recognition with a maximum entropy approach. Proceedings of CoNLL. find
  4. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu and P. Kuksa. 2011. Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research (JMLR). find our card
  5. J. L. Elman. 1990. Finding structure in time. Cognitive Science. find
  6. J. R. Finkel, T. Grenager, and C. Manning. 2005. Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. Proceedings of ACL. find
  7. R. Florian, A. Ittycheriah, H. Jing, and T. Zhang. 2003. Named entity recognition through classifier combination. Proceedings of NAACL-HLT. find
  8. J. Gimenez and L. Marquez. 2004. SVMTool: A general POS tagger generator based on support vector machines. Proceedings of LREC. find
  9. A. Graves and J. Schmidhuber. 2005. Framewise Phoneme Classification with Bidirectional LSTM and Other Neural Network Architectures. Neural Networks. find
  10. A. Graves, A. Mohamed, and G. Hinton. 2013. Speech Recognition with Deep Recurrent Neural Networks. arxiv. find
  11. J. Hammerton. 2003. Named Entity Recognition with Long Short-Term Memory. Proceedings of HLT-NAACL. find
  12. S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780. find
  13. T. Kudo and Y. Matsumoto. 2000. Use of support vector learning for chunk identification. Proceedings of CoNLL. find
  14. T. Kudo and Y. Matsumoto. 2001. Chunking with support vector machines. Proceedings of NAACL-HLT. find
  15. J. Lafferty, A. McCallum, and F. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. Proceedings of ICML. find
  16. A. McCallum, D. Freitag, and F. Pereira. 2000. Maximum entropy Markov models for information extraction and segmentation. Proceedings of ICML. find
  17. R. Mcdonald , K. Crammer , and F. Pereira. 2005. Flexible text segmentation with structured multilabel classification. Proceedings of HLT-EMNLP. find
  18. G. Mesnil, X. He, L. Deng, and Y. Bengio. 2013. Investigation of recurrent-neural-network architectures and learning methods for language understanding. Proceedings of IN-TERSPEECH. find
  19. T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, and S. Khudanpur. 2010. Recurrent neural network based language model. INTER-SPEECH. find
  20. T. Mikolov, A. Deoras, D. Povey, L. Burget, J. Eernocky. 2011. Strategies for Training Large Scale Neural Network Language Models. Proceedings of ASRU. find
  21. T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. Proceedings of NIPS. find
  22. A. Passos, V. Kumar, and A. McCallum. 2014. Lexicon Infused Phrase Embeddings for Named Entity Resolution. Proceedings of CoNLL. find
  23. L. R. Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE. find
  24. A. Ratnaparkhi. 1996. A maximum entropy model for part-of-speech tagging. Proceedings of EMNLP. find
  25. F. Sha and F. Pereira. 2003. Shallow parsing with conditional random fields. Proceedings of NAACL. find
  26. L. Shen, G. Sara, and A. K. Joshi. 2007. Guided learning for bidirectional sequence classification. Proceedings of ACL. find
  27. H. Shen and A. Sarkar. 2005. Voting between multiple data representations for text chunking. Canadian AI. find
  28. A. Soegaard. 2011. Semi-supervised condensed nearest neighbor for part-of-speech tagging. Proceedings of ACL-HLT. find
  29. X. Sun. 2014. Structure Regularization for Structured Prediction. Proceedings of NIPS. find
  30. X. Sun, L.P. Morency, D. OKanohara and J. Tsujii. 2008. Modeling Latent-Dynamic in Shallow Parsing: A Latent Conditional Model with Improved Inference. Proceedings of COLING. find
  31. K. Toutanova, D. Klein, C. Manning, and Y. Singer. 2003. Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network. Proceedings of HLT-NAACL. find
  32. M. Wang and C. D. Manning. 2013. Effect of Non-linear Deep Architecture in Sequence Labeling. Proceedings of IJCNLP. find
  33. P. Xu and R. Sarikaya. 2013. Convolutional neural network based triangular CRF for joint intent detection and slot filling. Proceedings of ASRU. find
  34. K. S. Yao, B. L. Peng, G. Zweig, D. Yu, X. L. Li, and F. Gao. 2014. Recurrent conditional random fields for language understanding. ICASSP. find
  35. K. S. Yao, B. Peng, Y. Zhang, D. Yu, G. Zweig, and Y. Shi. 2014. Spoken Language Understanding using Long Short-Term Memory Neural Networks. IEEE SLT. find