Elamur Research
☰
home / papers / 2015 / Multi-Task Learning for Multiple Language Translation

Multi-Task Learning for Multiple Language Translation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P15-1166.pdf

Summary

Translating one source language into several target languages usually means training a separate system for each pair. Statistical phrase-table systems cannot easily share knowledge across pairs, and quality drops sharply for pairs with little parallel data. Dong et al. (Baidu) recast this as multi-task learning on top of the attention-based RNN encoder-decoder of Bahdanau et al. (2014). A single bidirectional-RNN encoder for English is shared across all translation directions, while each target language gets its own decoder with its own attention. Training maximizes the sum of per-pair log-likelihoods, each normalized by the size of that pair's corpus.

The authors tested the approach with English as the source and French, Spanish, Dutch and Portuguese as targets. They used Europarl and WMT 2013 data, in both a large-data setting and a setting where some pairs were artificially reduced. The joint model beat separately trained NMT models in every configuration and also outperformed Moses on WMT 2013. It converged faster and produced better source word embeddings. These results suggest that sharing the encoder lets low-resource pairs benefit from data in high-resource pairs without a pivot language.

Key points

  • Architecture: one shared English encoder plus a separate decoder (with its own attention) for each target language. The authors describe this as the first end-to-end NMT system trained in a multi-task framework.
  • WMT 2013 test set (BLEU):
    • En-Fr rose from 23.89 (single NMT) to 26.02 with the fully trained multi-task model.
    • En-Es rose from 23.28 to 25.31, and reached 25.83 in the partial-data variant.
    • Moses scored 23.83 and 23.58, roughly the level of the single NMT models.
  • Low-resource setting: En-Nl and En-Pt were sub-sampled to 15% of their bilingual data. Multi-task gains over single models were:
    • En-Es: +1.64
    • En-Fr: +0.67
    • En-Nl: +1.26
    • En-Pt: +1.06
  • Convergence: on the Europarl common test set, learning curves for the resource-poor pairs improved more quickly under multi-task training. The joint model overtook the single models within three training iterations.
  • Embedding quality: nearest neighbours of English words from the shared encoder were far more sensible than those from the low-resource single English-Portuguese model. For example, "provide" was closest to "deliver" in the shared encoder but to "though" in the single model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

226 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 206 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-1166

@inproceedings{dong2015multi,
  title = {{Multi-Task Learning for Multiple Language Translation}},
  author = {Dong, Daxiang and Wu, Hua and He, Wei and Yu, Dianhai and Wang, Haifeng},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-1166},
  url = {https://doi.org/10.3115/v1/p15-1166}
}

References

22 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817–1853. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. find our card
  3. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, David Warde-Farley, and Yoshua Bengio. 2012. Theano: new features and speed improvements. CoRR, abs/1211.5590. find
  4. Léon Bottou. 1991. Stochastic gradient learning in neural networks. In Proceedings of Neuro-Nı̂mes 91, Nimes, France. EC2. find
  5. KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. CoRR, abs/1409.1259. find our card
  6. Trevor Cohn and Mirella Lapata. 2007. Machine translation by triangulation: Making effective use of multi-parallel corpora. In Proc. ACL, pages 728– 735. find
  7. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537. find our card
  8. Lei Cui, Xilun Chen, Dongdong Zhang, Shujie Liu, Mu Li, and Ming Zhou. 2013. Multi-domain adaptation for SMT using multi-task learning. In Proc. EMNLP, pages 1055–1065. find
  9. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard M. Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proc. ACL, pages 1370–1380. find
  10. Jianfeng Gao, Xiaodong He, Wen-tau Yih, and Li Deng. 2014. Learning continuous phrase representations for translation modeling. In Proc. ACL, pages 699–709. find
  11. Jun Hatori, Takuya Matsuzaki, Yusuke Miyao, and Jun’ichi Tsujii. 2012. Incremental joint approach to word segmentation, POS tagging, and dependency parsing in chinese. In Proc. ACL, pages 1045–1053. find
  12. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proc. EMNLP, pages 1700–1709. find
  13. Philipp Koehn. 2004. Pharaoh: A beam search decoder for phrase-based statistical machine translation models. In Machine Translation: From Real Users to Research, 6th Conference of the Association for Machine Translation in the Americas, AMTA 2004, Washington, DC, USA, September 28-October 2, 2004, P… find
  14. Zhenghua Li, Min Zhang, Wanxiang Che, Ting Liu, and Wenliang Chen. 2014. Joint optimization for chinese POS tagging and dependency parsing. IEEE/ACM Transactions on Audio, Speech & Language Processing, 22(1):274–286. find
  15. Shujie Liu, Nan Yang, Mu Li, and Ming Zhou. 2014. A recursive recurrent neural network for statistical machine translation. In Proc. ACL, pages 1491– 1500. find
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proc. find our card
  17. ACL, ACL 2002, pages 311–318, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  18. Rico Sennrich, Holger Schwenk, and Walid Aransa. 2013. A multi-domain translation model framework for statistical machine translation. In Proc. ACL, pages 832–840. find
  19. Martin Sundermeyer, Tamer Alkhouli, Joern Wuebker, and Hermann Ney. 2014. Translation modeling with bidirectional recurrent neural networks. In Proc. EMNLP, pages 14–25. find
  20. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104–3112. find
  21. Hua Wu and Haifeng Wang. 2007. Pivot language approach for phrase-based statistical machine translation. In Proc. ACL, pages 165–181. find
  22. Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701. find our card