Elamur Research
☰
home / papers / 2017 / Unsupervised Neural Machine Translation

Unsupervised Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1710.11041

Summary

Neural MT usually needs large parallel corpora, which most language pairs lack. This applies to low-resource languages such as Basque and to pairings of major languages such as German-Russian. Artetxe, Labaka, Agirre and Cho show that an NMT system can be trained from monolingual text alone. Their model is an attentional encoder-decoder with three changes. It handles both translation directions together. A single encoder serves both languages, while each language keeps its own decoder. The encoder's word vectors are cross-lingual embeddings, learned without supervision and frozen during training. Training alternates two objectives. In denoising, the model rebuilds a sentence whose tokens were randomly swapped. In on-the-fly back-translation, the current model translates a sentence and then learns to recover the original from that output. On WMT 2014 the system reaches 15.56 BLEU for French→English and 10.21 for German→English, and a small parallel corpus lifts these scores further. The work was an early proof that fully unsupervised NMT is possible, which matters for pairs with no parallel data.

Key points

  • Architecture: a two-layer bidirectional GRU encoder and a two-layer GRU decoder, with 600 hidden units and 300-dimensional embeddings. Attention follows Luong et al.'s global method with the general alignment function.
  • Embeddings: the encoder relies on fixed cross-lingual embeddings, obtained through the unsupervised mapping techniques that Artetxe et al. (2017) and, separately, Zhang et al. (2017) proposed. Each language has its own vocabulary in a shared space, so French "chair" and English "chair" get different vectors.
  • Unsupervised results (WMT 2014): 15.56 BLEU for French→English and 10.21 for German→English.
  • Semi-supervised results: adding 100,000 parallel sentences raises these to 21.81 and 15.24. By contrast, He et al. (2016) needed about 1.2M parallel sentences just for a warm start.
  • Code and analysis: the implementation is open source as undreamt (github.com/artetxem/undreamt). A manual analysis indicates the model learns translation patterns beyond word-by-word substitution.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cited by

330 papers of our library cite it (of the 140673 whose bibliographies we read); 23 of them have a page here, 307 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1710.11041 · arXiv 1710.11041

@misc{artetxe2017unsupervised,
  title = {{Unsupervised Neural Machine Translation}},
  author = {Artetxe, Mikel and Labaka, Gorka and Agirre, Eneko and Cho, Kyunghyun},
  year = {2017},
  doi = {10.48550/arxiv.1710.11041},
  eprint = {1710.11041},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1710.11041}
}

References

36 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2289–2294, Austin, Texas, November 2016. Association for Co… find our card
  2. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 451–462, Vancouver, Canada, July 2017. Association for Computatio… find our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proceedings of the 2014 International Conference on Learning Representations, 2014. find our card
  4. Yun Chen, Yang Liu, Yong Cheng, and Victor O.K. Li. A teacher-student framework for zero-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1925–1935, Vancouver, Canada, July 2017. Association f… find
  5. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Pro… find
  6. Anna Currey, Antonio Valerio Miceli Barone, and Kenneth Heafield. Copied monolingual data improves low-resource neural machine translation. In Proceedings of the Second Conference on Machine Translation, pp. 148–156, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. URL… find
  7. Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems 28, pp. 3079–3087. 2015. URL http://papers.nips.cc/paper/5949-semi-supervised-sequence-learning.pdf. ↗
  8. Qing Dou and Kevin Knight. Large scale decipherment for out-of-domain machine translation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 266–275, Jeju Island, Korea, July 2012. Association for Computat… find
  9. Qing Dou and Kevin Knight. Dependency-based decipherment for resource-limited machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1668–1676, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL http://w… find
  10. Qing Dou, Ashish Vaswani, Kevin Knight, and Chris Dyer. Unifying bayesian inference and vector space models for improved decipherment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing… find
  11. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 866–875, San Die… find our card
  12. Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. Zero-resource translation with multi-lingual neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 268–277, Austin, Texas, November 2016b.… find
  13. Stephan Gouws, Yoshua Bengio, and Greg Corrado. BilBOWA: Fast bilingual distributed representations without word alignments. In Proceedings of the 32nd International Conference on Machine Learning, pp. 748–756, 2015. find
  14. Thanh-Le Ha, Jan Niehues, and Alexander Waibel. Toward multilingual neural machine translation with universal encoder and decoder. arXiv preprint arXiv:1611.04798, 2016. find our card
  15. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, and Wei-Ying Ma. Dual learning for machine translation. In Advances in Neural Information Processing Systems 29, pp. 820–828. 2016. URL http://papers.nips.cc/paper/6469-dual-learning-for-machine-translation.pdf. ↗
  16. Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1367– 1377, San Diego, Californ… find our card
  17. Melvin Johnson, Mike Schuster, Quoc Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernand a Vià c gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Associa… find
  18. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference for Learning Representations, 2015. find
  19. Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pp. 28–39, Vancouver, August 2017. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/W17-3204. ↗
  20. Angeliki Lazaridou, Georgiana Dinu, and Marco Baroni. Hubness and pollution: Delving into cross-space mapping for zero-shot learning. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (… find
  21. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics, 5: 365–378, 2017. ISSN 2307-387X. URL https://transacl.org/ojs/index.php/tacl/article/view/1051. ↗ our card
  22. Thang Luong, Hieu Pham, and Christopher D. Manning. Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pp. 151–159, Denver, Colorado, June 2015a. Association for Computational Linguistics. URL… find our card
  23. Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412–1421, Lisbon, Portugal, September 2015b. Association for Computational Linguisti… find
  24. Antonio Valerio Miceli Barone. Towards cross-lingual distributed representations without parallel text trained with adversarial autoencoders. In Proceedings of the 1st Workshop on Representation Learning for NLP, pp. 121–126, Berlin, Germany, August 2016. Association for Computational Linguistics. U… find
  25. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168, 2013a. find
  26. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pp. 3111–3119. 2013b. find
  27. Prajit Ramachandran, Peter Liu, and Quoc Le. Unsupervised pretraining for sequence to sequence learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 383–391, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. URL https:/… find
  28. Sujith Ravi and Kevin Knight. Deciphering foreign language. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 12–21, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/antho… ↗
  29. Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 86–96, Berlin, Germany, August 2016a. Association for Computati… find
  30. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725, Berlin, Germany, August 2016b. Association for Computation… find
  31. Noah A. Smith and Jason Eisner. Contrastive estimation: Training log-linear models on unlabeled data. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pp. 354–362, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. doi: 10.3… find
  32. Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In 5th International Conference on Learning Representations (ICLR 2017), 2017. find our card
  33. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27, pp. 3104–3112. 2014. URL http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf. ↗
  34. Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11(Dec):3371–3408, 2010. find
  35. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  36. Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. Adversarial training for unsupervised bilingual lexicon induction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1959–1970, Vancouver, Canada, July 2017. Association for Comp… find