Elamur Research
☰
home / papers / 2018 / Meta-Learning for Low-Resource Neural Machine Translation

Meta-Learning for Low-Resource Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1398.pdf

Summary

Neural machine translation performs poorly when parallel data is scarce. Earlier remedies used monolingual corpora or multilingual and transfer learning. Gu et al. treat each language pair as a separate task and apply model-agnostic meta-learning (MAML). The goal is a parameter initialization that a new low-resource pair can adapt from in only a few updates. During meta-training, the system repeatedly samples one high-resource pair and runs a short simulated fine-tuning on one subset of its data. It then updates the shared starting point based on how well the adapted model does on a second subset. Languages do not share a vocabulary, so the authors add the universal lexical representation (ULR) to align inputs and outputs across languages. The resulting system, MetaNMT, learns from 18 European source languages and is tested on five target languages, all translating into English. It beat the multilingual transfer-learning baseline on every target pair, and its advantage widened as training data shrank. This makes usable NMT more realistic for languages with very small parallel corpora.

Key points

  • Data: The source tasks are 17 Europarl languages plus Russian from WMT. The target tasks are Ro, Lv, Fi, Tr and Ko into English, with at most 160k tokens of training data per target.
  • Headline result: MetaNMT reached 22.04 BLEU on Romanian-English WMT'16 after fine-tuning on only about 16,000 English tokens (roughly 600 sentence pairs).
  • Comparison and training: MetaNMT outperformed the multilingual transfer approach of Zoph et al. (2016) on all five targets. That baseline saturated and then degraded during training, while MetaNMT kept improving.
  • Effect of source tasks: Moving from one to 18 source tasks gave up to a 2× gain on Lv-En, Fi-En, Tr-En and Ko-En. Which source languages helped most depended on the target: Ro-En benefited more from {Es, Fr, It, Pt}, while the other targets benefited more from {De, Ru}.
  • Method details: The meta-gradient can be simplified by dropping the second-order term. The choice of validation task for early stopping (e.g., Ro-En vs. Tr-En) noticeably affected final quality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

126 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 123 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1398

@inproceedings{gu2018meta,
  title = {{Meta-Learning for Low-Resource Neural Machine Translation}},
  author = {Gu, Jiatao and Wang, Yong and Chen, Yun and Cho, Kyunghyun and Li, Victor O.K.},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1398},
  url = {https://doi.org/10.18653/v1/d18-1398}
}

References

43 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. 2016. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pages 3981–3989. find
  2. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017a. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 451–462. find our card
  3. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017b. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041. find our card
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606. find
  6. Yun Chen, Yang Liu, Yong Cheng, and Victor OK Li. 2017. A teacher-student framework for zero-resource neural machine translation. arXiv preprint arXiv:1705.00753. find
  7. Yun Chen, Yang Liu, and Victor OK Li. 2018. Zero-resource neural machine translation with multi-agent communication game. arXiv preprint arXiv:1802.03116. — [arXiv:1802.03116](https://arxiv.org/abs/1802.03116) find
  8. Yong Cheng, Yang Liu, Qian Yang, Maosong Sun, and Wei Xu. 2016. Neural machine translation with pivot languages. arXiv preprint arXiv:1611.04928. — [arXiv:1611.04928](https://arxiv.org/abs/1611.04928) find
  9. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–Decoder approaches. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. find our card
  10. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. International Conference on Learning Representations. find
  11. Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400. find
  12. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016a. Multi-way, multilingual neural machine translation with a shared attention mechanism. In NAACL. find our card
  13. Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T Yarman Vural, and Kyunghyun Cho. 2016b. Zero-resource translation with multi-lingual neural machine translation. In EMNLP. find
  14. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. 2017. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122. find our card
  15. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018a. Non-autoregressive neural machine translation. ICLR. find our card
  16. Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor OK Li. 2018b. Universal neural machine translation for extremely low resource languages. arXiv preprint arXiv:1802.05368. find our card
  17. Caglar Gulcehre, Sarath Chandar, Kyunghyun Cho, and Yoshua Bengio. 2018. Dynamic neural turing machine with continuous and discrete addressing schemes. Neural computation, 30(4):857–884. find
  18. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535. find our card
  19. David Ha, Andrew Dai, and Quoc V Le. 2016a. Hypernetworks. arXiv preprint arXiv:1609.09106. — [arXiv:1609.09106](https://arxiv.org/abs/1609.09106) find
  20. Thanh-Le Ha, Jan Niehues, and Alexander Waibel. 2016b. Toward multilingual neural machine translation with universal encoder and decoder. arXiv preprint arXiv:1611.04798. find our card
  21. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pages 820–828. find
  22. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2016. Google’s multilingual neural machine translation system: enabling zero-shot translation. arXiv preprint arXiv:1611.04558. find
  23. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  24. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. arXiv preprint arXiv:1706.03872. find
  25. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 48–54. Association for Computational Linguis… find
  26. Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338. find
  27. Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043. find our card
  28. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2016. Fully character-level neural machine translation without explicit segmentation. arXiv preprint arXiv:1610.03017. find our card
  29. Jason Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela. 2017. Emergent translation in multi-agent communication. arXiv preprint arXiv:1710.06922. — [arXiv:1710.06922](https://arxiv.org/abs/1710.06922) find
  30. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126. find
  31. Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. 2017. Meta-learning with temporal convolutions. arXiv preprint arXiv:1707.03141. find
  32. Herbert Robbins and Sutton Monro. 1951. A stochastic approximation method. The annals of mathematical statistics, pages 400–407. find
  33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. find
  34. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation systems for wmt 16. arXiv preprint arXiv:1606.02891. find our card
  35. Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. arXiv preprint arXiv:1702.03859. find our card
  36. Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pages 4080–4090. find
  37. Ilya Sutskever, Oriol Vinyals, and Quôc Lê. 2014. Sequence to sequence learning with neural networks. In NIPS. find
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. find
  39. Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, pages 3630–3638. find
  40. Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. 2018. Unsupervised neural machine translation with weight sharing. arXiv preprint arXiv:1804.09057. find
  41. Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1535–1545. find our card
  42. Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Earth mover’s distance minimization for unsupervised bilingual lexicon induction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1934– 1945. Association for Computational Linguistics. find
  43. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. arXiv preprint arXiv:1604.02201. find