Elamur Research⌕
☰
home / papers / 2017 / Unsupervised Pretraining for Sequence to Sequence Learning

Unsupervised Pretraining for Sequence to Sequence Learning

Summary

Seq2seq networks tend to overfit when labeled data is scarce. The authors propose a simple way to use unlabeled text to address this. They train two language models on monolingual data, one for the source language and one for the target language. These LMs supply the initial weights for the embeddings and first LSTM layer of the encoder and decoder, and for the decoder softmax. The full model is then fine-tuned on parallel or labeled data. During fine-tuning, the LM losses stay in the objective with equal weight, so the pretrained knowledge is not forgotten. Residual connections and attention over both the first and top layers give small additional gains.

On WMT English→German, using News Crawl monolingual data, the approach set a new state of the art. It outperformed phrase-based systems, supervised NMT and backtranslation. For CNN/Daily Mail abstractive summarization, ROUGE scores only matched a prior baseline, but human judges preferred the pretrained system and found less repetition. The work shows that generic LM pretraining can benefit full encoder-decoder models, not just classifiers.

Key points

  • MT setup: Parallel data came from WMT'14, filtered by language detection to about 4M sentence pairs, with subword units from 89500 merge operations (≈90k vocabulary). The LMs were 1-layer 4096-unit LSTMs projected to 1024 dimensions, trained for a week on 32 Tesla K40 GPUs over News Crawl corpora of more than a billion tokens each.
  • MT single model: It reached 24.0 BLEU on newstest2014 and 27.0 on newstest2015. That is 1.3 BLEU above the best single backtranslation system (22.7 / 25.7) and also beats the earlier 4-model ensemble.
  • MT ensemble: A 5-model ensemble scored 24.7 / 28.1, matching or exceeding a 12-model backtranslation ensemble (23.8 / 27.6 per the extracted table).
  • Ablations: In translation, pretraining only the encoder cost 1.6 BLEU, while pretraining only the decoder cost 1.0. In summarization the pattern reversed, with encoder pretraining mattering more, which the authors attribute to better optimization over long inputs. Dropping the LM objective hurt both tasks.
  • Human evaluation (200 CNN/Daily Mail documents): The pretrained system scored higher in 83 cases, tied in 88 and lost in 29 (sign test p < 0.0001). It also produced repeated phrases significantly less often (McNemar p < 0.0001).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

72 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 57 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1039 · arXiv 1611.02683

@inproceedings{ramachandran2017unsupervised,
  title = {{Unsupervised Pretraining for Sequence to Sequence Learning}},
  author = {Ramachandran, Prajit and Liu, Peter and Le, Quoc},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1039},
  eprint = {1611.02683},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d17-1039}
}

References

40 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Unsupervised pretraining has been intensively studied in the past years, most notably is the work by Dahl et al. (2012) who found that pretraining with deep belief networks improved feedforward \#\# References pretrained GloVe embedding vectors (Pennington et al., 2014) had more impact. Related to our… find
  2. Robert B. Allen. 1987. Several studies on natural language and back-propagation. IEEE First International Conference on Neural Networks. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  4. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 workshop on statistical machine translation. In Proc… find our card
  5. William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. 2015. Listen, attend and spell. arXiv preprint arXiv:1508.01211. — [arXiv:1508.01211](https://arxiv.org/abs/1508.01211) find
  6. Jianshu Chen, Po-Sen Huang, Xiaodong He, Jianfeng Gao, and Li Deng. 2016. Unsupervised learning of predictors from unpaired input-output samples. abs/1606.04646. — [arXiv:1606.04646](https://arxiv.org/abs/1606.04646) find
  7. Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation. arXiv preprint arXiv:1606.04596. find
  8. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP. find
  9. G. E. Dahl, D. Yu, L. Deng, and A. Acero. 2012. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):30–42. find
  10. Andrew M. Dai and Quoc V. Le. 2015. supervised sequence learning. In NIPS. \#\# Conclusion Semi-Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman-Vural, and Kyunghyun Cho. 2016. Zero-resource translation with multilingual neural machine translation. arXiv preprint arXiv:1606.04164. — [arXiv:1606.04164](https://arxiv.org/abs/1606.04164) find
  11. We presented a novel unsupervised pretraining method to improve sequence to sequence learning. The method can aid in both generalization and optimization. Our scheme involves pretraining two language models in the source and target domain, and initializing the embeddings, first LSTM layers, and soft… find
  12. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211. find
  13. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535. find our card
  14. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS. find our card
  15. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. find
  16. Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. Montreal neural machine translation systems for WMT’15. In Proceedings of the Tenth Workshop on Statistical Machine Translation. find
  17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. find
  18. Felix Stahlberg, Eva Hasler, and Bill Byrne. 2016. The edit distance transducer in action: The university of cambridge english-german system at wmt16. In Proceedings of the First Conference on Machine Translation, pages 377–384, Berlin, Germany. Association for Computational Linguistics. find
  19. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. find
  20. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP. find our card
  21. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS. find
  22. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  23. Subhashini Venugopalan, Lisa Anne Hendricks, Raymond Mooney, and Kate Saenko. 2016. Improving LSTM-based video description with linguistic knowledge mined from text. arXiv preprint arXiv:1604.01729. find
  24. Chin-Yew Lin. 2004. ROUGE: a package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004). find
  25. Philip Williams, Rico Sennrich, Maria Nadejde, Matthias Huck, Barry Haddow, and Ondřej Bojar. 2016. Edinburgh’s statistical machine translation systems for wmt16. In Proceedings of the First Conference on Machine Translation, pages 399– 410, Berlin, Germany. Association for Computational Linguistic… find
  26. Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015. Multi-task sequence to sequence learning. In ICLR. find our card
  27. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS. find
  28. Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Mike Seltzer, Andreas Stolcke, Dong Yu, and Geoffrey Zweig. 2016. Achieving human parity in conversational speech recognition. abs/1610.05256. — [arXiv:1610.05256](https://arxiv.org/abs/1610.05256) find
  29. Ramesh Nallapati, Bing Xiang, and Bowen Zhou. 2016. Sequence-to-sequence RNNs for text summarization. arXiv preprint arXiv:1602.06023. find
  30. Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction. ACL. find our card
  31. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329. find our card
  32. Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In EMNLP. find our card
  33. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In ACL. find our card
  34. Yu Zhang, William Chan, and Navdeep Jaitly. 2016. Very deep convolutional networks for end-to-end speech recognition. abs/1610.03022. — [arXiv:1610.03022](https://arxiv.org/abs/1610.03022) find
  35. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. ICML. find
  36. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In EMNLP. find
  37. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. find
  38. Ramon P. Ñeco and Mikel L. Forcada. 1997. Asynchronous translations with recurrent neural nets. Neural Networks. find
  39. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444. find our card
  40. Hasim Sak, Andrew W. Senior, and Françoise Beaufays. 2014. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ramachandran-2017-unsupervised-pretraining-sequence-sequence-learning/