Elamur Research⌕
☰
home / papers / 2019 / An Effective Approach to Unsupervised Machine Translation

An Effective Approach to Unsupervised Machine Translation

Summary

Machine translation systems normally need large parallel corpora, which exist for only a few major language pairs. This paper improves unsupervised phrase-based SMT, which learns from monolingual text alone. The authors first build a phrase-table by training n-gram embeddings with phrase2vec and aligning them across languages with VecMap. They then fix three weaknesses of earlier systems. Character-level similarity scores are added to the phrase-table, which helps with named entities and numerals. Ad-hoc weight setting is replaced by an unsupervised tuning objective that is guaranteed to converge. Finally, models for both translation directions are refined together. The resulting SMT system initializes a dual NMT model, which is further trained with on-the-fly back-translation. On WMT 2014/2016 French-English and German-English, the approach beats the previous best unsupervised results by 5–7 BLEU in every direction. For English-to-German, it even outperforms the supervised WMT 2014 winner. This suggests that translation without parallel data is becoming practical, which matters for under-resourced language pairs.

Key points

  • English→German WMT 2014 reaches 22.5 BLEU. This is 5.5 points above the prior best unsupervised system and 0.5 above the supervised 2014 shared task winner. In the other directions, the system trails that winner by roughly 2 BLEU.
  • Phrase-table induction uses phrase2vec embeddings with vocabularies capped at 200,000 unigrams, 400,000 bigrams and 400,000 trigrams. VecMap maps them with identical initialization, and each source phrase gets 100 nearest-neighbor translation candidates.
  • The subword scores rely on Levenshtein distance normalized by word length, with a floor (ε = 0.3) so that dissimilar candidates are not overly penalized.
  • The new tuning objective replaces MERT on a parallel dev set, which is impossible in unsupervised settings. It includes a length penalty on round-trip translations.
  • Code is released as monoses (github.com/artetxem/monoses). Qualitative French→English examples show that earlier errors, such as "Sunday Telegraph" becoming "The Times of London" and 34 becoming 32, no longer occur.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

76 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 75 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1019

@inproceedings{artetxe2019effective,
  title = {{An Effective Approach to Unsupervised Machine Translation}},
  author = {Artetxe, Mikel and Labaka, Gorka and Agirre, Eneko},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1019},
  url = {https://doi.org/10.18653/v1/p19-1019}
}

References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462, Vancouver, Canada. Association for Computational… find our card
  2. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics. find
  3. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–798. Association for Com… find our card
  4. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. 2018. Achieving human parity on automatic chinese to english news translation. arXiv preprint arXiv:1803.05567. find our card
  5. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018b. Unsupervised statistical machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3632–3642, Brussels, Belgium. Association for Computational Linguistics. find our card
  6. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in Neural Information Processing Systems 29, pages 820–828. find
  7. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified kneser-ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 690–696, Sofia, Bulgaria. Association for… find our card
  8. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018c. Unsupervised neural machine translation. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018). find our card
  9. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018). find
  10. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  11. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018). find our card
  12. Qing Dou and Kevin Knight. 2012. Large scale decipherment for out-of-domain machine translation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 266–275, Jeju Island, Korea. Association for Computation… find
  13. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018b. find
  14. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039–5049, Brussels, Belgium. Association for Computational Linguistics. find
  15. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010. find
  16. Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707–710. find
  17. Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. 2018. Unsupervised neural machine translation with weight sharing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 46–55. Association for Computational Linguistics. find
  18. Benjamin Marie and Atsushi Fujita. 2018. Unsupervised neural machine translation initialized by unsupervised statistical machine translation. arXiv preprint arXiv:1810.12703. — [arXiv:1810.12703](https://arxiv.org/abs/1810.12703) find
  19. Omar Zaidan. 2009. Z-mert: A fully configurable open source tool for minimum error rate training of machine translation systems. The Prague Bulletin of Mathematical Linguistics, 91:79–88. find
  20. Andrew McCallum, Kedar Bellare, and Fernando Pereira. 2005. A conditional random field for discriminatively-trained finite-state string edit distance. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence, pages 388–395. find
  21. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In The IEEE International Conference on Computer Vision (ICCV). find
  22. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111–3119. find
  23. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 160–167, Sapporo, Japan. Association for Computational Linguistics. find
  24. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1–9, Belgium, Brussels. Association for Computational Linguistics. find our card
  25. Matt Post. 2018. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. find our card
  26. Sujith Ravi and Kevin Knight. 2011. Deciphering foreign language. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 12– 21, Portland, Oregon, USA. Association for Computational Linguistics. find
  27. Shuo Ren, Zhirui Zhang, Shujie Liu, Ming Zhou, and Shuai Ma. 2019. Unsupervised neural machine translation with smt as posterior regularization. arXiv preprint arXiv:1901.04112. — [arXiv:1901.04112](https://arxiv.org/abs/1901.04112) find
  28. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational L… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/artetxe-2019-effective-approach-unsupervised-machine-translation/