Elamur Research
☰
home / papers / 2017 / Unsupervised Machine Translation Using Monolingual Corpora Only

Unsupervised Machine Translation Using Monolingual Corpora Only

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1711.00043

Summary

Neural machine translation usually needs millions of aligned sentence pairs, which most low-resource language pairs lack. This paper tests whether a translator can be trained with no parallel data, using only one monolingual corpus per language. The authors use a single attentional LSTM encoder–decoder for both languages. Only the word embeddings are language-specific, so sentences from either language are encoded into a common latent space. Training combines two reconstruction tasks. In the first, the model restores a sentence from a corrupted copy of itself (denoising auto-encoding). In the second, it recovers a sentence from the translation produced by the previous model iteration (back-translation). An adversarial discriminator pushes the latent codes of both languages toward the same distribution. Training starts from word-by-word translation with a bilingual dictionary that was also learned without supervision. On Multi30k and WMT, the English–French system reaches 32.8 and 15.1 BLEU. The work shows that usable translation can be learned from monolingual text alone, which makes it relevant to languages with little or no parallel data.

Key points

  • Results: 32.76 BLEU on English–French Multi30k-Task1 (above 22 for every language pair there) and 15.1 BLEU on WMT English–French, with no parallel sentences used in training.
  • On WMT, the unsupervised model performs about as well as a comparable supervised system trained on 100,000 sentence pairs.
  • Architecture: 3-layer bidirectional LSTM encoder and 3-layer LSTM decoder with attention (Bahdanau et al.). LSTM layers and attention weights are shared across languages, embedding and hidden sizes are 300, and output uses greedy decoding.
  • Noise for denoising: words are randomly dropped with probability p_wd, and word order is lightly shuffled so that no token moves more than k positions.
  • The first translation model comes from an unsupervised bilingual lexicon (Conneau et al., 2017, "Word translation without parallel data"). Each later iteration retrains on fresh back-translations from the previous model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

477 papers of our library cite it (of the 140673 whose bibliographies we read); 26 of them have a page here, 451 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1711.00043 · arXiv 1711.00043

@misc{lample2017unsupervised,
  title = {{Unsupervised Machine Translation Using Monolingual Corpora Only}},
  author = {Lample, Guillaume and Conneau, Alexis and Denoyer, Ludovic and Ranzato, Marc'Aurelio},
  year = {2017},
  doi = {10.48550/arxiv.1711.00043},
  eprint = {1711.00043},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1711.00043}
}

References

41 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. find our card
  2. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5: 135–146, 2017. find
  3. Y. Chen, Y. Liu, Y. Cheng, and V.O.K. Li. A teacher-student framework for zero-resource neural machine translation. In ACL, 2017. find
  4. Y. Cheng, W. Xu, Z. He, W. He, H. Wu, M. Sun, and Y. Liu. Semi-supervised learning for neural machine translation. arXiv:1606.04596, 2016. find
  5. A. Conneau, G. Lample, M. Ranzato, L. Denoyer, and H. Jégou. Word translation without parallel data. arXiv:1710.04087, 2017. find
  6. D.L. Donoho. Compressed sensing. IEEE Transactions on Information Theory, 2006. find
  7. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. Multi30k: Multilingual english-german image descriptions. arXiv preprint arXiv:1605.00459, 2016. find our card
  8. O. Firat, B. Sankaran, Y. Al-Onaizan, F.T.Y. Vural, and K. Cho. Zero-resource translation with multi-lingual neural machine translation, 2016. find
  9. Y. Ganin, E. Ustinova, h. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Domain-adversarial training of neural networks. JMLR, 2016. find
  10. Ian Goodfellow. Nips 2016 tutorial: arXiv:1701.00160, 2016. — [arXiv:1701.00160](https://arxiv.org/abs/1701.00160) find
  11. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680, 2014. find
  12. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535, 2015. find our card
  13. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, and Wei-Ying Ma. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pp. 820–828, 2016. find
  14. Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. arXiv preprint arXiv:1602.03483, 2016. find our card
  15. Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Controllable text generation. arXiv preprint arXiv:1703.00955, 2017. find our card
  16. A. Irvine. Combining bilingual and comparable corpora for low resource machine translation. 2013. find
  17. A. Irvine and C. Callison-Burch. End-to-end statistical machine translation with zero or small parallel texts. Natural Language Engineering, 1(1), 2015. find
  18. M. Johnson, M. Schuster, Q.V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. Vigas, M. Wattenberg, G. Corrado, M. Hughes, and J. Dean. Googles multilingual neural machine translation system: Enabling zero-shot translation, 2016. find
  19. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. find
  20. Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013. find our card
  21. Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. In Advances In Neural Information Processing Systems, pp. 4601–4609, 2016. find
  22. Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic DENOYER, et al. Fader networks: Manipulating images by sliding attributes. In Advances in Neural Information Processing Systems, pp. 5963–5972, 2017. find
  23. J. Lee, K. Cho, J. Weston, and D. Kiela. Emergent translation in multi-agent communication. arXiv:1710.06922, 2017. — [arXiv:1710.06922](https://arxiv.org/abs/1710.06922) find
  24. Ming-Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In Advances in neural information processing systems, pp. 469–477, 2016. find
  25. Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. arXiv preprint arXiv:1703.00848, 2017. — [arXiv:1703.00848](https://arxiv.org/abs/1703.00848) find
  26. D.S. Munteanu, A. Fraser, and D. Marcu. Improved machine translation performance via parallel sentence extraction from comparable corpora. In ACL, 2004. find
  27. H. Nakayama and N. Nishida. Zero-resource machine translation by multimodal encoder-decoder network with multimedia pivot. arXiv:1611.04503, 2017. — [arXiv:1611.04503](https://arxiv.org/abs/1611.04503) find
  28. K. Papineni, S. Roukos, T. Ward, and W.J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting on Association for Computational Linguistics, 2002. find our card
  29. N. Pourdamghani and K. Knight. Deciphering related languages. In EMNLP, 2017. find
  30. S. Ravi and K. Knight. Deciphering foreign language. In ACL, 2011. find
  31. Generative adversarial networks. arXiv preprint Auto-encoding variational bayes. arXiv preprint Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709, 2015a. find
  32. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015b. find
  33. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. arXiv preprint arXiv:1705.09655, 2017. find our card
  34. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104–3112, 2014. find
  35. Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. arXiv preprint arXiv:1611.02200, 2016. — [arXiv:1611.02200](https://arxiv.org/abs/1611.02200) find
  36. T. Tieleman and G. Hinton. Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012. find
  37. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096–1103. ACM, 2008. find
  38. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. find
  39. Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. arXiv preprint arXiv:1705.11122, 2017. — [arXiv:1705.11122](https://arxiv.org/abs/1705.11122) find
  40. H. Zheng, Y. Cheng, and Y. Liu. Maximum expected likelihood estimation for zero-resource neural machine translation. In IJCAI, 2017. find
  41. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593, 2017. — [arXiv:1703.10593](https://arxiv.org/abs/1703.10593) find