Elamur Research
☰
home / papers / 2018 / Semi-Supervised Sequence Modeling with Cross-View Training

Semi-Supervised Sequence Modeling with Cross-View Training

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1217.pdf

Summary

Pre-trained representations such as word2vec or ELMo use unlabeled text in a phase that ignores the target task. Supervised training then relies on labeled data alone. Clark et al. propose Cross-View Training (CVT), a semi-supervised method in the self-training tradition that learns from labeled and unlabeled data together. The model is a shared two-layer CNN-BiLSTM encoder. On labeled batches it is trained with ordinary cross-entropy. On unlabeled batches, the full model's predictions act as fixed soft targets. Extra auxiliary prediction modules, each seeing only part of the input (for example, only the forward LSTM, so no right context), are trained to match those targets. This forces the shared representations to stay informative when context is missing, which also helps the primary predictor. Unlike noise-based consistency methods, CVT never alters the discrete text input. The authors report state-of-the-art results on sequence tagging, dependency parsing and English–Vietnamese machine translation. Multi-task CVT is both more accurate and faster to train than separate models.

Key points

  • Evaluated on seven tasks: CCG supertagging, NER, POS tagging, text chunking, English dependency parsing, English→Vietnamese translation, and one further tagging task.
  • The unlabeled-data loss is the KL divergence between primary and auxiliary outputs. Gradients do not flow through the primary module's targets.
  • Auxiliary modules are used only during training, so inference cost and final parameter count are unchanged.
  • A joint model for six tasks (all except MT) beats a multi-task ELMo baseline and needs about half the total training time of single-task models. It turns unlabeled sentences into artificial examples labeled for every task.
  • CVT is compared with virtual adversarial training (embedding noise). Code was planned for release under tensorflow/models/research/cvt_text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

123 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 114 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1217

@inproceedings{clark2018semi,
  title = {{Semi-Supervised Sequence Modeling with Cross-View Training}},
  author = {Clark, Kevin and Luong, Minh-Thang and Manning, Christopher D. and Le, Quoc},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1217},
  url = {https://doi.org/10.18653/v1/d18-1217}
}

References

68 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Philip Bachman, Ouais Alsharif, and Doina Precup. 2014. Learning with pseudo-ensembles. In NIPS. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  3. Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In COLT. ACM. find our card
  4. Rich Caruana. 1997. Multitask learning. Machine Learning, 28:41–75. find
  5. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2015. find
  6. The IWSLT 2015 evaluation campaign. In International Workshop on Spoken Language Translation. find
  7. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling. In INTERSPEECH. find our card
  8. Jason PC Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics. find our card
  9. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML. find
  10. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research. find our card
  11. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP. find our card
  12. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NIPS. find
  13. Carl Doersch and Andrew Zisserman. 2017. Multi-task self-supervised visual learning. arXiv preprint arXiv:1708.07860. — [arXiv:1708.07860](https://arxiv.org/abs/1708.07860) find
  14. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In ICLR. find our card
  15. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5):602–610. find
  16. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple nlp tasks. In EMNLP. find our card
  17. Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In HLT-NAACL. find our card
  18. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. find our card
  19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  20. Julia Hockenmaier and Mark Steedman. 2007. CCGbank: a corpus of CCG derivations and dependency structures extracted from the Penn treebank. Computational Linguistics, 33(3):355–396. find our card
  21. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: the 90% solution. In HLT-NAACL. find
  22. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. find
  23. Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. 2017. Reinforcement learning with unsupervised auxiliary tasks. In ICLR. find
  24. Kevin Jarrett, Koray Kavukcuoglu, Yann LeCun, et al. 2009. What is the best multi-stage architecture for object recognition? In IEEE Conference on Computer Vision. find
  25. Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In EMNLP. find our card
  26. James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural netwo… find
  27. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302. find
  28. Samuli Laine and Timo Aila. 2017. Temporal ensembling for semi-supervised learning. In ICLR. find
  29. Yann LeCun, Koray Kavukcuoglu, and Clément Farabet. 2010. Convolutional networks and applications in vision. In ISCAS. IEEE. find
  30. Zhizhong Li and Derek Hoiem. 2016. Learning without forgetting. In ECCV. find
  31. Minh-Thang Luong, Eugene Brevdo, and Rui Zhao. 2017. Neural machine translation (seq2seq) tutorial. https://github.com/tensorflow/nmt. ↗
  32. Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In ICLR. find our card
  33. Minh-Thang Luong and Christopher D. Manning. 2015. Stanford neural machine translation systems for spoken language domains. In IWSLT. find
  34. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. https://blog.openai.com/language-unsupervised. ↗
  35. Prajit Ramachandran, Peter J Liu, and Quoc V Le. 2017. Unsupervised pretraining for sequence to sequence learning. In EMNLP. find
  36. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP. find
  37. Marek Rei. 2017. Semi-supervised multitask learning for sequence labeling. In ACL. find
  38. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNN-CRF. In ACL. find
  39. Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In EMNLP. find our card
  40. Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stack-pointer networks for dependency parsing. In ACL. find
  41. Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098. find our card
  42. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The Penn treebank. Computational linguistics, 19(2):313–330. find our card
  43. Sebastian Ruder and Barbara Plank. 2018. Strong baselines for neural semi-supervised learning under domain shift. In ACL. find
  44. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS. find our card
  45. Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. 2016. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In NIPS. find
  46. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In ACL. find our card
  47. H Scudder. 1965. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363–371. find
  48. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS. find
  49. Vikas Sindhwani and Mikhail Belkin. 2005. A co-regularization approach to semi-supervised learning with multiple views. In ICML Workshop on Learning with Multiple Views. find
  50. Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In ICLR. find our card
  51. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In ACL. find our card
  52. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. 2016. Distributional smoothing with virtual adversarial training. In ICLR. find
  53. Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017. Fast and accurate sequence labeling with iterated dilated convolutions. In EMNLP. find
  54. Hao Peng, Sam Thomson, and Noah A. Smith. 2017. Deep multitask learning for semantic dependency parsing. In ACL. find
  55. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. In ICLR. find
  56. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP. find
  57. Matthew E Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In ACL. find our card
  58. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  59. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS. find
  60. Antti Tarvainen and Harri Valpola. 2017. Weight-averaged consistency targets improve semi-supervised deep learning results. In Workshop on Learning with Limited Labeled Data, NIPS. find
  61. Erik F Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task: Chunking. In CoNLL. find
  62. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In HLT-NAACL. find
  63. Xiang Wei, Zixia Liu, Liqiang Wang, and Boqing Gong. 2018. Improving the improved training of Wasserstein GANs. In ICLR. find
  64. Huijia Wu, Jiajun Zhang, and Chengqing Zong. 2017. Shortcut sequence tagging. arXiv preprint arXiv:1701.00576. — [arXiv:1701.00576](https://arxiv.org/abs/1701.00576) find
  65. Chang Xu, Dacheng Tao, and Chao Xu. 2013. A survey on multi-view learning. arXiv preprint arXiv:1304.5634. — [arXiv:1304.5634](https://arxiv.org/abs/1304.5634) find
  66. David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In ACL. find our card
  67. Yuan Zhang and David Weiss. 2016. Stack-propagation: Improved representation learning for syntax. In ACL. find
  68. Zhi-Hua Zhou and Ming Li. 2005. Tri-training: Exploiting unlabeled data using three classifiers. IEEE Transactions on knowledge and Data Engineering. find