Elamur Research
☰
home / papers / 2015 / Multi-task Sequence to Sequence Learning

Multi-task Sequence to Sequence Learning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1511.06114

Summary

Sequence-to-sequence (seq2seq) models had mostly been trained for one task at a time. The authors ask whether training them jointly on several related tasks helps. They define three ways to share parameters. In one-to-many, a single encoder feeds several task-specific decoders, such as translation and constituency parsing. In many-to-one, several encoders share one decoder, such as translation and image captioning. In many-to-many, several encoders and decoders are shared, which lets translation be combined with unsupervised objectives on monolingual text.

Training switches between tasks, choosing each mini-batch's task at random according to mixing ratios α. The experiments focus on WMT'15 English⇆German and use LSTM models without attention. Adding fairly small parsing and captioning datasets improves translation by up to 1.5 BLEU over single-task baselines. Translation data also helps parsing, and an ensemble reaches a new best constituent parsing score of 93.0 F1.

The paper matters as an early systematic look at when parameter sharing pays off across different seq2seq tasks. It also compares how two unsupervised objectives behave in this setting.

Key points

  • Data: about 4.5M WMT'15 sentence pairs per translation direction, with 50K-word vocabularies shared across tasks. The parsing output vocabulary has 104 tags. Testing uses newstest2014 (English→German) and newstest2015 (German→English).
  • Auxiliary corpora: Penn Tree Bank parsing (40K), a High-Confidence Corpus for parsing (11.0M), image captioning data (596K), and monolingual English and German text (12.1M and 13.8M).
  • Unsupervised objectives: sequence autoencoders and skip-thought. Because sentences are unordered, skip-thought is adapted by predicting one half of a sentence from the other half. Compared with skip-thought, the autoencoder improves perplexity less but improves BLEU more.
  • Limitation: none of the models use attention, and the authors leave multi-task learning with attention to future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

257 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 236 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.06114 · arXiv 1511.06114

@misc{luong2015multi,
  title = {{Multi-task Sequence to Sequence Learning}},
  author = {Luong, Minh-Thang and Le, Quoc V. and Sutskever, Ilya and Vinyals, Oriol and Kaiser, Lukasz},
  year = {2015},
  doi = {10.48550/arxiv.1511.06114},
  eprint = {1511.06114},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.06114}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ando, Rie Kubota and Zhang, Tong. A framework for learning predictive structures from multiple tasks and unlabeled data. JMLR, 6:1817–1853, 2005. find
  2. Argyriou, Andreas, Evgeniou, Theodoros, and Pontil, Massimiliano. Multi-task feature learning. In NIPS, 2007. find
  3. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. find our card
  4. Bojar, Ondřej, Chatterjee, Rajen, Federmann, Christian, Haddow, Barry, Huck, Matthias, Hokamp, Chris, Koehn, Philipp, Logacheva, Varvara, Monz, Christof, Negri, Matteo, Post, Matt, Scarton, Carolina, Specia, Lucia, and Turchi, Marco. Findings of the 2015 workshop on statistical machine translation.… find our card
  5. Caruana, Rich. Multitask learning. Machine Learning, 28(1):41–75, 1997. find
  6. Cho, Kyunghyun, van Merrienboer, Bart, Gulcehre, Caglar, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, 2014. find
  7. Dai, Andrew M. and Le, Quoc V. Semi-supervised sequence learning. In NIPS, 2015. find
  8. Donahue, Jeff, Jia, Yangqing, Vinyals, Oriol, Hoffman, Judy, Zhang, Ning, Tzeng, Eric, and Darrell, Trevor. DeCAF: A deep convolutional activation feature for generic visual recognition, 2014. find
  9. Dong, Daxiang, Wu, Hua, He, Wei, Yu, Dianhai, and Wang, Haifeng. Multi-task learning for multiple language translation. In ACL, 2015. find our card
  10. Evgeniou, Theodoros and Pontil, Massimiliano. Regularized multi–task learning. In SIGKDD, 2004. find
  11. Gulcehre, Caglar, Firat, Orhan, Xu, Kelvin, Cho, Kyunghyun, Barrault, Loic, Lin, Huei-Chi, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535, 2015. find our card
  12. Heigold, Georg, Vanhoucke, Vincent, Senior, Alan, Nguyen, Patrick, Ranzato, Marc’Aurelio, Devin, Matthieu, and Dean, Jeffrey. Multilingual acoustic models using distributed deep neural networks. In ICASSP, 2013. find
  13. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural Computation, 9(8): 1735–1780, 1997. find
  14. Huang, Jui-Ting, Li, Jinyu, Yu, Dong, Deng, Li, and Gong, Yifan. Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers. In ICASSP, 2013. find
  15. Jean, Sébastien, Cho, Kyunghyun, Memisevic, Roland, and Bengio, Yoshua. On using very large target vocabulary for neural machine translation. In ACL, 2015a. find our card
  16. Jean, Sébastien, Firat, Orhan, Cho, Kyunghyun, Memisevic, Roland, and Bengio, Yoshua. Montreal neural machine translation systems for WMT’15. In WMT, 2015b. find
  17. Jozefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., and Wu, Y. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. find
  18. Kalchbrenner, Nal and Blunsom, Phil. Recurrent continuous translation models. In EMNLP, 2013. find
  19. Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan, Zemel, Richard S., Torralba, Antonio, Urtasun, Raquel, and Fidler, Sanja. Skip-thought vectors. In NIPS, 2015. find
  20. Kumar, Abhishek and III, Hal Daumé. Learning task grouping and overlap in multi-task learning. In ICML, 2012. find
  21. Liu, Xiaodong, Gao, Jianfeng, He, Xiaodong, Deng, Li, Duh, Kevin, and Wang, Ye-Yi. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In NAACL, 2015. find
  22. Luong, Minh-Thang and Manning, Christopher D. Stanford neural machine translation systems for spoken language domain. In IWSLT, 2015. find
  23. Luong, Minh-Thang, Pham, Hieu, and Manning, Christopher D. Effective approaches to attention-based neural machine translation. In EMNLP, 2015a. find
  24. Luong, Minh-Thang, Sutskever, Ilya, Le, Quoc V., Vinyals, Oriol, and Zaremba, Wojciech. Addressing the rare word problem in neural machine translation. In ACL, 2015b. find our card
  25. Marcus, Mitchell P., Marcinkiewicz, Mary Ann, and Santorini, Beatrice. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2):313–330, 1993. find our card
  26. Papineni, Kishore, Roukos, Salim, Ward, Todd, and jing Zhu, Wei. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002. find our card
  27. Pham, Vu, Bluche, Théodore, Kermorvant, Christopher, and Louradour, Jérôme. Dropout improves recurrent neural networks for handwriting recognition. In Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on, pp. 285–290. find
  28. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. Sequence to sequence learning with neural networks. In NIPS, 2014. find
  29. Thrun, Sebastian. Is learning the n-th thing any easier than learning the first? In NIPS, 1996. find
  30. Vinyals, Oriol, Kaiser, Lukasz, Koo, Terry, Petrov, Slav, Sutskever, Ilya, and Hinton, Geoffrey. Grammar as a foreign language. In NIPS, 2015a. find
  31. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. In CVPR, 2015b. find our card
  32. Xu, Kelvin, Ba, Jimmy, Kiros, Ryan, Cho, Kyunghyun, Courville, Aaron C., Salakhutdinov, Ruslan, Zemel, Richard S., and Bengio, Yoshua. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015. find our card