Elamur Research⌕
☰
home / papers / 2019 / Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations

Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations

Paper · 2019 · Jiatao Gu, Yong Wang, Kyunghyun Cho et al.

Summary

Multilingual NMT systems can translate between language pairs never seen together in training (zero-shot translation), but this ability is fragile. It depends heavily on hyper-parameters and usually falls well short of pivoting through a third language such as English. The authors trace the failure to a spurious dependency: with English-centric training data, maximum-likelihood training lets the decoder link the source language to the output language. The decoder then disregards the requested target-language ID and often produces English. They quantify this with an estimate of how much information the source-language ID and the language of the decoded output share, i.e. their mutual information. To weaken that dependency, they adapt two known techniques, decoder (language model) pre-training and back-translation, to the zero-shot setting. They test Transformer models on Europarl, IWSLT and MultiUN. Both techniques lift vanilla zero-shot quality substantially and stabilise training, and they match or beat pivot-based translation. This makes direct zero-shot translation a practical option for language pairs without parallel data.

Key points

  • Vanilla zero-shot NMT is highly sensitive to training settings. In preliminary Europarl experiments (En, Fr, Es, De), a layer-wise attention variant failed completely on zero-shot pairs, while parallel directions were barely affected.
  • The degeneracy is measured through the estimated mutual information I(i; z) between source language ID and output language. As this value grows, the model drifts toward outputting the wrong language, typically English.
  • Decoder (LM) pre-training and back-translation (BTZS) improve vanilla zero-shot results by roughly 4–22 BLEU. On some pairs they beat pivoting by about 2–3 BLEU.
  • Evaluation covers 4 languages from Europarl, 5 from IWSLT and 4 from MultiUN. The base model is a Transformer (d_model 512, 6 layers, 8 heads) with 40,000 shared BPE operations.
  • In an IWSLT setup with chained pairs (De-En, En-It, It-Ro, Ro-Nl), pivot translation must pass through several languages and degrades sharply. Zero-shot translation avoids this chain and gains clearly over pivoting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States158 works · 211,459 citations🎓New York UniversityNew York · United States136 works · 86,390 citations🎓University of Hong KongPok Fu Lam · China44 works · 10,936 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Machine translation · 958 works

Cite

Cite as Jiatao Gu, Yong Wang, Kyunghyun Cho and Victor O.K. Li. 2019. Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1121

@inproceedings{gu2019improved,
  title = {{Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations}},
  author = {Jiatao Gu, Yong Wang, Kyunghyun Cho and Victor O.K. Li},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1121},
  url = {https://doi.org/10.18653/v1/p19-1121}
}

References

25 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2018. The missing ingredient in zero-shot neural machine translation. find
  2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In Proceedings of International Conference on Learning Representations (ICLR), Vancouver, Canada. our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. our card
  4. Yun Chen, Yang Liu, Yong Cheng, and Victor OK Li. 2017. A teacher-student framework for zero-resource neural machine translation. arXiv preprint arXiv:1705.00753. find
  5. Yun Chen, Yang Liu, and Victor OK Li. 2018. Zero-resource neural machine translation with multi-agent communication game. arXiv preprint arXiv:1802.03116. — [arXiv:1802.03116](https://arxiv.org/abs/1802.03116) find
  6. Yong Cheng, Yang Liu, Qian Yang, Maosong Sun, and Wei Xu. 2016. Neural machine translation with pivot languages. arXiv preprint arXiv:1611.04928. — [arXiv:1611.04928](https://arxiv.org/abs/1611.04928) find
  7. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–Decoder approaches. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. our card
  8. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016a. Multi-way, multilingual neural machine translation with a shared attention mechanism. In NAACL. our card
  9. Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T Yarman Vural, and Kyunghyun Cho. 2016b. Zero-resource translation with multi-lingual neural machine translation. In EMNLP. our card
  10. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of International Conference on Machine Learning (ICML). our card
  11. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. find
  12. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018a. Non-autoregressive neural machine translation. ICLR. our card
  13. Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor OK Li. 2018b. Universal neural machine translation for extremely low resource languages. arXiv preprint arXiv:1802.05368. our card
  14. Jiatao Gu, Yong Wang, Yun Chen, Kyunghyun Cho, and Victor OK Li. 2018c. Meta-learning for low-resource neural machine translation. arXiv preprint arXiv:1808.08437. our card
  15. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the As… find
  16. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 48–54. Association for Computational Linguis… find
  17. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  18. Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In Proceedings of International Conference on Learning Representations (ICLR), Vancouver, Canada. our card
  19. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018b. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039–5049. Association for Computational Linguistics. find
  20. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2016. Fully character-level neural machine translation without explicit segmentation. arXiv preprint arXiv:1610.03017. our card
  21. Jason Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela. 2017. Emergent translation in multi-agent communication. arXiv preprint arXiv:1710.06922. — [arXiv:1710.06922](https://arxiv.org/abs/1710.06922) find
  22. Marco Lui and Timothy Baldwin. 2012. langid. py: An off-the-shelf language identification tool. In Proceedings of the ACL 2012 system demonstrations, pages 25–30. Association for Computational Linguistics. machine translation. In Human Language Technology Conference of the North American Chapter of… our card
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS). our card
  24. Hao Zheng, Yong Cheng, and Yang Liu. 2017. Maximum expected likelihood estimation for zero-resource neural machine translation. IJCAI. find
  25. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gu-2019-improved-zero-shot-neural-machine/