Elamur Research
☰
home / papers / 2017 / Semi-supervised sequence tagging with bidirectional language models

Semi-supervised sequence tagging with bidirectional language models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1705.00108

Summary

Neural sequence taggers usually start from pre-trained word vectors. However, the bidirectional RNN that turns those vectors into context-aware token representations learns only from the small labeled training set. Peters et al. propose TagLM, a semi-supervised way to supply contextual knowledge from unlabeled text without extra annotated data.

Two separate neural language models, one reading left-to-right and one right-to-left, are first trained on a large unlabeled corpus (the 1 Billion Word Benchmark). Their top-layer states for each token serve as "LM embeddings" and are fed into a standard hierarchical tagger. That tagger combines character-level and word embeddings, a two-layer bidirectional GRU/LSTM, and a CRF output layer.

The approach sets new state-of-the-art scores on CoNLL 2003 NER and CoNLL 2000 Chunking. It beats earlier systems that relied on additional labeled data or gazetteers. The work shows that representations from language models pre-trained on raw text can substantially strengthen supervised tagging, even across domains.

Key points

  • On CoNLL 2003 NER, adding LM embeddings raises F1 from 90.87% to 91.93%, a gain of over one point.
  • TagLM reaches 96.37% F1 on CoNLL 2000 Chunking, a new best result at publication time.
  • Combining forward and backward LMs consistently outperforms using a forward LM alone. The two directions are distinct models with no shared parameters.
  • A news-trained LM still helped on scientific text. On SemEval 2017 Task 10 (ScienceIE), entity-extraction dev F1 rose from 49.93 to 54.05, and the system was central to the winning Scenario 1 entry.
  • The baseline tagger uses a CNN or RNN character encoder, fine-tuned pre-trained word embeddings, two bidirectional RNN layers, and a CRF decoded with Viterbi under BIOES labeling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Part-of-speech tagging

Cited by

162 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 148 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1705.00108 · arXiv 1705.00108

@misc{peters2017semi,
  title = {{Semi-supervised sequence tagging with bidirectional language models}},
  author = {Peters, Matthew E. and Ammar, Waleed and Bhagavatula, Chandra and Power, Russell},
  year = {2017},
  doi = {10.48550/arxiv.1705.00108},
  eprint = {1705.00108},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1705.00108}
}

References

45 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Waleed Ammar, Matthew E. Peters, Chandra Bhagavatula, and Russell Power. 2017. The AI2 system at SemEval-2017 Task 10 (ScienceIE): semi-supervised end-to-end entity and relation extraction. In ACL workshop (SemEval). find
  2. Rie Kubota Ando and Tong Zhang. 2005. A high-performance semi-supervised learning method for text chunking. In ACL. find
  3. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. In JMLR. find
  4. Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In COLT. find our card
  5. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn. 2014. One billion word benchmark for measuring progress in statistical language modeling. CoRR abs/1312.3005. find our card
  6. Jason Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. In TACL. find our card
  7. Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In SSST@EMNLP. find our card
  8. Michael Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In EMNLP. find our card
  9. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. In JMLR. find our card
  10. Andrew M. Dai and Quoc V. Le. 2015. supervised sequence learning. In NIPS. find
  11. Semi-Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard M Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In ACL. find
  12. Volkmar Frinken, Alicia Fornés, Josep Lladós, and Jean-Marc Ogier. 2012. Bidirectional language model for handwriting recognition. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR). find
  13. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2016. A joint many-task model: Growing a neural network for multiple nlp tasks. CoRR abs/1611.01587. find our card
  14. Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In HLT-NAACL. find our card
  15. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9. find
  16. Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. CoRR abs/1602.02410. find
  17. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP. find
  18. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. In ICLR workshop. find our card
  19. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  20. Jamie Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In NIPS. find
  21. Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press. find
  22. John D. Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML. find
  23. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In NAACL-HLT. find
  24. Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In ICML. find our card
  25. Wei Li and Andrew McCallum. 2005. Semi-supervised sequence modeling with syntactic topic models. In AAAI. find
  26. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. In TACL. find our card
  27. Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. 2015. Joint entity recognition and disambiguation. In EMNLP. find
  28. Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In ACL. find our card
  29. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional lstm. In CoNLL. find our card
  30. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Interspeech. find
  31. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS. find
  32. Behrang Mohit and Rebecca Hwa. 2005. Syntax-based semi-supervised named entity tagging. In ACL. find
  33. Kamal Nigam, Andrew Kachites McCallum, Sebastian Thrun, and Tom Mitchell. 2000. Text classification from labeled and unlabeled documents using em. Machine learning . find
  34. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. find
  35. Álvaro Peris and Francisco Casacuberta. 2015. A bidirectional recurrent neural language model for machine translation. Procesamiento del Lenguaje Natural . find
  36. David Pierce and Claire Cardie. 2001. Limitations of co-training for natural language learning from large datasets. In EMNLP. find
  37. Lev-Arie Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In CoNLL. find our card
  38. Hasim Sak, Andrew W. Senior, and Franoise Beaufays. 2014. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH. find
  39. Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task chunking. In CoNLL/LLL. find
  40. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In CoNLL. find
  41. Anders Søgaard. 2013. Semi-supervised learning and domain adaptation in natural language processing. Synthesis Lectures on Human Language Technologies . find
  42. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In ACL. find our card
  43. Jun Suzuki, Akinori Fujino, and Hideki Isozaki. 2007. Semi-supervised structured output learning based on a hybrid generative and discriminative approach. In EMNLP-CoNLL. find
  44. Jun Suzuki and Hideki Isozaki. 2008. Semi-supervised sequential labeling and segmentation using gigaword scale unlabeled data. In ACL. find
  45. Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. In ICLR. find