Elamur Research⌕
☰
home / papers / 2018 / Contextual Parameter Generation for Universal Neural Machine Translation

Contextual Parameter Generation for Universal Neural Machine Translation

Paper · 2018 · Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig et al.

Summary

The paper addresses a trade-off in multilingual neural machine translation. Universal models share all parameters across languages and struggle when training data is scarce or the languages differ a lot. Per-language encoder-decoder setups share nothing between languages, so they end up overparameterized. The authors propose a contextual parameter generator (CPG) that takes learned embeddings of the source and target languages and produces the weights of the encoder and decoder, respectively. The underlying architecture stays a standard NMT model. It is shared across all language pairs, and only its parameters change depending on the language context. In the experiments, the generator is a linear function, which amounts to a low-rank constraint on the parameters. The same framework also covers earlier pairwise, per-language and universal approaches. The system:

  • can learn from monolingual data,
  • supports zero-shot translation,
  • surpasses state-of-the-art results on IWSLT-15 and IWSLT-17.

This matters because it gives a cheap, architecture-agnostic way to control how much languages share. It needs no back-translation, pivoting or multilingual word embeddings.

Key points

  • Parameter counts: pairwise NMT needs O(ST) parameter sets, per-language models need O(S+T), and the CPG adds controllable sharing on top of the per-language design.
  • Language embeddings live in their own space, separate from word embeddings, unlike Johnson et al. (2017).
  • The setup uses BPE subword vocabularies kept separate per language, so the decoder's output projection is language-specific.
  • Results surpass state-of-the-art on IWSLT-15 and IWSLT-17. The authors note that their IWSLT-17 numbers are not comparable to the official challenge report.
  • Cosine distances between the learned language embeddings mirror known similarities, such as German–Dutch. The code is released as symphony-mt (github.com/eaplatanios/symphony-mt).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig and Tom Mitchell. 2018. Contextual Parameter Generation for Universal Neural Machine Translation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1039

@inproceedings{platanios2018contextual,
  title = {{Contextual Parameter Generation for Universal Neural Machine Translation}},
  author = {Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig and Tom Mitchell},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1039},
  url = {https://doi.org/10.18653/v1/d18-1039}
}

References

36 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Lon…
  2. Maruan Al-Shedivat, Avinava Dubey, and Eric P Xing. 2017. Contextual Explanation Networks. CoRR, abs/1705.10301. — [arXiv:1705.10301](https://arxiv.org/abs/1705.10301) find
  3. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised Neural Machine Translation. In International Conference on Learning Representations.
  4. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016a. Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism. In Proceedings of NAACL-HLT, pages 866–875.
  5. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In International Conference on Learning Representations.
  6. Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman-Vural, and Kyunghyun Cho. 2016b. Zero-Resource Translation with Multi-Lingual Neural Machine Translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
  7. Ozan Caglayan, Walid Aransa, Yaxing Wang, Marc Masana, Mercedes Garcı́a-Martı́nez, Fethi Bougares, Loı̈c Barrault, and Joost van de Weijer. 2016. Does Multimodality Help Human and Machine for Translation and Image Captioning? In Proceedings of the First Conference on Machine Translation, volume 2. find
  8. Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O K Li. 2018. Universal Neural Machine Translation for Extremely Low Resource Languages. In Proceedings of NAACL-HLT, pages 344–354.
  9. David Ha, Andrew Dai, and Quoc V Le. 2018. Hyper-Networks. In International Conference on Learning Representations. find
  10. Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuhito Sudoh, Koichi Yoshino, and Christian Federmann. 2017. Overview of the IWSLT 2017 Evaluation Campaign. In Proceedings of the 14th International Workshop on Spoken Language Translation.
  11. Thanh-Le Ha, Jan Niehues, and Alexander Waibel. 2016. Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder. In Proceedings of the 13th International Workshop on Spoken Language Translation.
  12. Yun Chen, Yang Liu, Yong Cheng, and Victor O K Li. 2017. A Teacher-Student Framework for Zero-Resource Neural Machine Translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1925–1935. find
  13. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, and Wei-Ying Ma. 2016. Dual Learning for Machine Translation. In Advances in Neural Information Processing Systems, pages 820–828. find
  14. Yong Cheng, Yang Liu, Qian Yang, Maosong Sun, and Wei Xu. 2016. Neural Machine Translation with Pivot Languages. CoRR, abs/1611.04928. — [arXiv:1611.04928](https://arxiv.org/abs/1611.04928) find
  15. Po-Sen Huang, Chong Wang, Sitao Huang, Dengyong Zhou, and Li Deng. 2018. Towards Neural Phrase-Based Machine Translation. In International Conference on Learning Representations. find
  16. Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder– Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural… find
  17. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google’s Multilingual Neural Machine Translation System:Enabling Zero-Shot Translation. In Transactions of the Association for Computationa… find
  18. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Pro… find
  19. Josep Maria Crego, Jungi Kim, Guillaume Klein, Anabel Rebollo, Kathy Yang, Jean Senellart, Egor Akhanov, Patrice Brunelle, Aurelien Coquard, Yongchao Deng, Satoshi Enoue, Chiyo Geiss, Joshua Johanson, Ardas Khalsa, Raoum Khiari, Byeongil Ko, Catherine Kobus, Jean Lorieux, Leidiana Martins, Dang-Chua… — [arXiv:1610.05540](https://arxiv.org/abs/1610.05540) find
  20. Surafel M Lakew, Quintino F Lotito, Negri Matteo, Turchi Marco, and Federico Marcello. 2017. Improving Zero-Shot Translation of Low-Resource Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting…
  21. Languages. In Proceedings of the 14th International Workshop on Spoken Language Translation, pages 113–119. find
  22. Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Unsupervised Machine Translation Using Monolingual Corpora Only. In International Conference on Learning Representations.
  23. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems, pages 3104–3112. find
  24. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully Character-Level Neural Machine Translation without Explicit Segmentation. 5:365–378.
  25. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and Composing Robust Features with Denoising Autoencoders. In Proceedings of the 25th International Conference on Machine Learning, pages 1096–1103. find
  26. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task Sequence to Sequence Learning. In International Conference on Learning Representations.
  27. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  28. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421.
  29. David G. Marr. 1981. Language and Literacy. In Vietnamese Tradition on Trial, 1920-1945, pages 136– 189. University of California Press. find
  30. Paul Michel and Graham Neubig. 2018. Extreme Adaptation for Personalized Neural Machine Translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Short Papers), pages 312–318.
  31. Barret Zoph and Kevin Knight. 2016. Multi-Source Neural Translation. In Proceedings of NAACL-HLT, pages 30–34.
  32. Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, Bryan Kisiel, Jayant Krishnamurthy, Ni Lao, Kathryn Mazaitis, Thahir P. Mohamed, Ndapa Nakashole, Emmanouil A. Platanios, Alan Ritter, Mehdi Samadi, Burr Settl… find
  33. Emmanouil A. Platanios. 2018. TensorFlow Scala. https://github.com/eaplatanios/ tensorflow_scala. ↗
  34. Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. 2018. On the Convergence of Adam and Beyond. In International Conference on Learning Representations. find
  35. Richard H Richens. 1958. Interlingual Machine Translation. The Computer Journal, 1(3):144–147. find
  36. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 86–96. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/platanios-2018-contextual-parameter-generation-universal-neural/