Elamur Research
☰
home / papers / 2016 / Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism

Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N16-1101.pdf

Summary

Attention-based neural MT systems are usually built for a single language pair. Their attention (alignment) component was assumed to be pair-specific, so earlier multilingual NMT work either dropped attention or supported only one-to-many translation. Firat, Cho and Bengio design one network with a separate encoder for each source language, a separate decoder for each target language, and a single attention module reused by every pair. Since only encoders and decoders are added per language, model size scales linearly with the number of languages, rather than quadratically as with a collection of bilingual systems. The model is trained jointly on WMT'15 data for English, French, Czech, German, Russian and Finnish (ten translation directions). It is also tested in simulated low-resource settings where the target parallel corpus is artificially reduced. The multilingual model outperforms single-pair baselines in all low-resource cases and matches or beats them on most large-scale directions while using roughly half the parameters. This shows that attention can be shared across languages and that multilinguality especially benefits data-poor pairs.

Key points

  • Architecture: N encoders and M decoders connected through one shared attention mechanism. Parameter count grows as O(L), versus O(L²) for separate single-pair models.
  • Data: all ten WMT'15 pairs/directions (En↔Fr, Cs, De, Ru, Fi). Low-resource conditions were created by artificially limiting the size of the target parallel corpus.
  • In low-resource tests, the multilingual model ("Multi") beats both "Single" and "Single+DF", which adds target-side monolingual data. The advantage widens as parallel data shrinks, so the gain is not just a result of seeing more target text.
  • At large scale, the multilingual model is better than or comparable to single-pair models in both BLEU and log-probability, with the clearest gains when translating into English. The authors attribute this to English appearing in every training corpus. Adding monolingual data via autoencoder-style reconstruction led to token copying and worse results.
  • Training splits language pairs across GPUs (NVidia Titan X, 12GB) and synchronises only the attention parameters, which gives near-linear speed-up. Code is released at github.com/nyu-dl/dl4mt-multi.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

259 papers of our library cite it (of the 140673 whose bibliographies we read); 22 of them have a page here, 237 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1101 · arXiv 1601.01073

@inproceedings{firat2016multi,
  title = {{Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism}},
  author = {Firat, Orhan and Cho, Kyunghyun and Bengio, Yoshua},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1101},
  eprint = {1601.01073},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n16-1101}
}

References

14 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In ICLR 2015. Léon Bottou, Yoshua Bengio, and Yann Le Cun. 1997. Global training of document processing systems using graph transformer networks. In Computer Vision and P… find our card
  2. IEEE Computer Society Conference on, pages 489– 494. IEEE. Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41–75. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the properties of neural machine translation: Encoder–Decoder approaches. In Eighth W… find
  3. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-based models for speech recognition. In Advances in Neural Information Processing Systems, pages 577–585. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Ku… find our card
  4. Weiguang Ding, Ruoyan Wang, Fei Mao, and Graham W. Taylor. 2014. Theano-based large-scale visual recognition with multiple gpus. arXiv:1412.2302. — [arXiv:1412.2302](https://arxiv.org/abs/1412.2302) find
  5. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. ACL. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neu… find our card
  6. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term me… find our card
  7. Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. Montreal neural machine translation systems for wmt’15. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 134–140, Lisbon, Portugal, September. Association for Computational Linguisti… find
  8. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. The International Conference on Learning Representations (ICLR). find
  9. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015a. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114. find our card
  10. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015b. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025. find
  11. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063. find
  12. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. find
  13. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. find
  14. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find