Elamur Research⌕
☰
home / papers / 2017 / Semi-supervised Multitask Learning for Sequence Labeling

Semi-supervised Multitask Learning for Sequence Labeling

Paper · 2017 · Marek Rei

Summary

Neural sequence labelers are usually trained only to predict each token's annotated tag. When labels are sparse, this gives a weak learning signal: most tokens carry the majority tag (O in NER, "correct" in error detection) and teach the model little. Rei addresses this by adding an auxiliary language modeling task to a bidirectional LSTM tagger that also uses character-level word representations. The forward LSTM states learn to predict the next word, and the backward states learn to predict the previous word. This stops either direction from simply seeing the answer it must predict. The joint representation is still used for tagging, through a softmax or CRF output. The extra loss is weighted by a parameter γ and requires no additional labeled or unlabeled data. Across 10 datasets covering error detection, NER, chunking and POS-tagging, the auxiliary objective improved results on every benchmark. The biggest gains came on learner error detection. The method is cheap and general, and it shows that more value can be extracted from existing training text.

Key points

  • The baseline is the Rei et al. (2016) BiLSTM tagger with a character-level component. The method adds four weight matrices that are used only during training, so tagging cost at test time is unchanged.
  • Label sparsity motivates the approach: only about 17% of CoNLL 2003 tokens are entities, and about 14% of FCE tokens are errors.
  • For detecting errors in learner writing, the model beat the strongest earlier architecture by 3.9% in absolute terms.
  • POS-tagging was evaluated on PTB (48 tags), the GENIA corpus (2,000 PubMed abstracts, 42 labels, 210-document test set), and the Finnish and Spanish sections of UD v1.2. Gains on these datasets were small but consistent.
  • Unlike prior multitask work that relies on auxiliary datasets, the objective uses only the training corpus. The author suggests extending it to large unannotated in-domain text as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom256 works · 123,373 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Marek Rei. 2017. Semi-supervised Multitask Learning for Sequence Labeling. arXiv:1704.07156.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1704.07156 · arXiv 1704.07156

@misc{rei2017semi,
  title = {{Semi-supervised Multitask Learning for Sequence Labeling}},
  author = {Marek Rei},
  year = {2017},
  doi = {10.48550/arxiv.1704.07156},
  eprint = {1704.07156},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1704.07156}
}

References

34 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller, Dzmitry Bahdanau, Nicolas Ballas, Frédéric Bastien, Justin Bayer, Anatoly Belikov, Alexander Belopolsky, Yoshua Bengio, Arnaud Bergeron, James Bergstra, Valentin Bisson, Josh Bleecher Snyder, Nicolas Bouchard, Nicolas Boulanger… — Theano: A Python framework for fast computation of mathematical expressions
  2. Isabelle Augenstein and Anders Søgaard. 2017. Multi-Task Learning of Keyphrase Boundary Classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. http://arxiv.org/abs/1704.00514. ↗
  3. Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In arXiv preprint. http://arxiv.org/abs/1702.08303. ↗
  4. Rich Caruana. 1998. Multitask Learning. Ph.D. thesis. find
  5. Ciprian Chelba, Tomáš Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. In arXiv preprint. http://arxiv.org/abs/1312.3005. ↗
  6. Hao Cheng, Hao Fang, and Mari Ostendorf. 2015. Open-Domain Name Error Detection using a Multi-Task RNN. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. find
  7. Ronan Collobert and Jason Weston. 2008. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. Proceedings of the 25th international conference on Machine learning (ICML ’08) https://doi.org/10.1145/1390156.1390177. — [doi:10.1145/1390156.1390177](https://doi.org/10.1145/1390156.1390177) ↗
  8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research 12. https://doi.org/10.1145/2347736.2347755. ↗
  9. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. International Conference on Acoustics, Speech and Signal Processing (ICASSP) https://doi.org/10.1109/ICASSP.2013.6638947. ↗
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-term Memory. Neural Computation 9. https://doi.org/10.1.1.56.7752. find
  11. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv:1508.01991 http://arxiv.org/pdf/1508.01991v1.pdf. ↗
  12. Ozan Irsoy and Claire Cardie. 2014. Opinion Mining with Deep Recurrent Neural Networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  13. Jin-Dong Kim, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Nigel Collier. 2004. Introduction to the Bio-entity Recognition Task at JNLPBA. Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and Its Applications https://doi.org/10.3115/1567594.1567610. ↗
  14. Martin Krallinger, Florian Leitner, Obdulia Rabal, Miguel Vazquez, Julen Oyarzabal, and Alfonso Valencia. 2015. CHEMDNER: The drugs and chemical names extraction challenge. Journal of Cheminformatics 7(Suppl 1). https://doi.org/10.1186/1758-2946-7-S1-S1. — [doi:10.1186/1758-2946-7-s1-s1](https://doi.org/10.1186/1758-2946-7-s1-s1) ↗
  15. John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the 18th International Conference on Machine Learning. find
  16. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural Architectures for Named Entity Recognition. In Proceedings of NAACL-HLT 2016.
  17. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics 19. https://doi.org/10.1162/coli.2010.36.1.36100. ↗
  18. Tomáš Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. In Proceedings of the International Conference on Learning Representations (ICLR 2013). https://doi.org/10.1162/153244303322533223. ↗
  19. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent Neural Network based Language Model. Interspeech (September):1045–1048. find
  20. Andriy Mnih and Yee Whye Teh. 2012. A fast and simple algorithm for training neural probabilistic language models. In Neural Information Processing Systems (NIPS).
  21. Frederic Morin and Yoshua Bengio. 2005. Hierarchical probabilistic neural network language model. Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics https://doi.org/10.1109/JCDL.2003.1204852. — [doi:10.1109/jcdl.2003.1204852](https://doi.org/10.1109/jcdl.2003.1204852) ↗
  22. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 Shared Task on Grammatical Error Correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task. http://www.aclweb.org/ant… ↗
  23. Joakim Nivre, Željko Agić, Maria Jesus Aranzabe, Masayuki Asahara, Aitziber Atutxa, Miguel Ballesteros, John Bauer, Kepa Bengoetxea, Riyaz Ahmad Bhat, Cristina Bosco, Sam Bowman, et al. 2015. Universal dependencies 1.2. LIN-DAT/CLARIN digital library at the Institute of Formal and Applied Linguist… find
  24. Tomoko Ohta, Yuka Tateisi, and Jin-Dong Kim. 2002. The GENIA corpus: An annotated research abstract corpus in molecular biology domain. Proceedings of the second international conference on Human Language Technology Research http://portal.acm.org/citation.cfm?id=1289260. ↗
  25. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL-16). pages 412–418. http://arxiv.org/abs/16… ↗
  26. Marek Rei, Gamal K. O. Crichton, and Sampo Pyysalo. 2016. Attending to Characters in Neural Sequence Labeling Models. In Proceedings of the 26th International Conference on Computational Linguistics (COLING-2016). http://arxiv.org/abs/1611.04361. ↗
  27. Marek Rei and Helen Yannakoudakis. 2016. Compositional Sequence Labeling Models for Error Detection in Learner Writing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. https://aclweb.org/anthology/P/P16/P16-1112.pdf. ↗
  28. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout : A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research (JMLR) 15. https://doi.org/10.1214/12-AOS1000. ↗
  29. Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task: Chunking. Proceedings of the 2nd Workshop on Learning Language in Logic and the 4th Conference on Computational Natural Language Learning 7. https://doi.org/10.3115/1117601.1117631. ↗
  30. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003. http://arxiv.org/abs/cs/0306050. ↗
  31. Yoshimasa Tsuruoka, Yuka Tateishi, Jin Dong Kim, Tomoko Ohta, John McNaught, Sophia Ananiadou, and Jun’ichi Tsujii. 2005. Developing a robust part-of-speech tagger for biomedical text. In Proceedings of Panhellenic Conference on Informatics. find
  32. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A New Dataset and Method for Automatically Grading ESOL Texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. http://www.aclweb.org/anthology/P11-1019. ↗
  33. Matthew D. Zeiler. 2012. ADADELTA: An Adaptive Learning Rate Method. arXiv preprint arXiv:1212.5701 http://arxiv.org/abs/1212.5701. ↗
  34. GuoDong Zhou and Jian Su. 2004. Exploring Deep Knowledge Resources in Biomedical Name Recognition. Workshop on Natural Language Processing in Biomedicine and Its Applications at COLING . find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rei-2017-semi-supervised-multitask-learning-sequence/