Elamur Research⌕
☰
home / papers / 2018 / Trivial Transfer Learning for Low-Resource Neural Machine Translation

Trivial Transfer Learning for Low-Resource Neural Machine Translation

Summary

Neural MT degrades sharply when parallel data are scarce. Earlier transfer-learning recipes for this case relied on a shared target language, related languages, or custom training schedules. Kocmi and Bojar show that a much plainer procedure works. A Transformer (Tensor2Tensor 1.4.2) is first trained to convergence on a high-resource "parent" pair. Then only the training corpus is replaced with the low-resource "child" pair, and the learning rate and other hyperparameters are left untouched. The one requirement is a joint wordpiece vocabulary built from equal-sized samples of the parent and child corpora. The experiments use WMT and UN corpus data. Estonian and Slovak serve as low-resource languages, while Finnish, Czech, Russian, Arabic, French and Spanish appear in parent pairs. The resulting children outperform child-only baselines. This holds even when the shared English sits on different sides in parent and child, and even when the parent shares neither language nor script with the child. The approach matters because it needs no changes to existing NMT toolkits and gives under-resourced languages a cheap quality boost.

Key points

  • The shared vocabulary targets about 32k wordpiece types, with actual sizes between 26.1k and 34.8k. Training ran for 1M steps on a single GTX 1080 Ti. Resetting the learning rate at the switch erased the gains inherited from the parent.
  • The English→Estonian child rose from 17.03 BLEU (baseline) to 19.74 with an EN→FI parent and to 20.41 with an EN→CS parent. Estonian→English with an FI→EN parent improved by 2.44 BLEU (21.74 → 24.18).
  • Gains are largest with tiny child data. With only 10k EN–ET training sentences, the transfer model scored 12.46 BLEU, versus 1.95 for the baseline.
  • When the child pair was actually the larger one (Slovak parent → Czech child), transfer underperformed the baseline. An AR–RU parent overlapping only 6.2% in subwords with its child still improved BLEU.
  • Cased metrics (chrF3, nCharacTER, nTER and others) confirm the improvements, which are not explained by output length, punctuation, numbers or named entities. The authors conclude that parent corpus size matters more than vocabulary overlap.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

96 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 95 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6325

@inproceedings{kocmi2018trivial,
  title = {{Trivial Transfer Learning for Low-Resource Neural Machine Translation}},
  author = {Kocmi, Tom and Bojar, Ondřej},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Research Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6325},
  url = {https://doi.org/10.18653/v1/w18-6325}
}

References

32 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fernand a Vigas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339–351. find
  2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In Proceedings of the Sixth International Conference on Learning Representations. find our card
  3. Alina Karakanta, Jon Dehdari, and Josef van Genabith. 2018. Neural machine translation for low-resource languages without parallel corpora. Machine Translation, 32(1):167–189. find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR, abs/1409.0473. find our card
  5. Tom Kocmi and Ondřej Bojar. 2017. Curriculum Learning and Minibatch Bucketing in Neural Machine Translation. In Recent Advances in Natural Language Processing 2017. find
  6. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48. ACM. find our card
  7. Tom Kocmi, oman Sudarikov, and Ondřej Bojar. 2018. CUNI Submissions in WMT18. In Proceedings of the 3rd Conference on Machine Translation (WMT), Brussels, Belgium. find
  8. Ondřej Bojar, Ondřej Dušek, Tom Kocmi, Jindřich Libovický, Michal Novák, Martin Popel, Roman Sudarikov, and Dušan Variš. 2016. Czeng 1.6: enlarged czech-english parallel corpus with processing tools dockered. In International Conference on Text, Speech, and Dialogue, pages 231–238. Springer. find
  9. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of EMNLP, volume 4, pages 388–395. find
  10. Anna Currey, Antonio Valerio Miceli Barone, and Kenneth Heafield. 2017. Copied monolingual data improves low-resource neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 148–156. find our card
  11. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver. Association for Computational Linguistics. find
  12. Raj Dabre, Tetsuji Nakagawa, and Hideto Kazawa. 2017. An empirical study of language relatedness for transfer learning in neural machine translation. In Proceedings of the 31st Pacific Asia Conference on Language, Information and Computation, pages 282–286. find
  13. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  14. Ahmed El Kholy, Nizar Habash, Gregor Leusch, Evgeny Matusov, and Hassan Sawaf. 2013. Language independent connectivity strength features for phrase pivot statistical machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Paper… find
  15. Preslav Nakov and Hwee Tou Ng. 2012. Improving statistical machine translation for a resource-poor language using related resource-rich languages. Journal of Artificial Intelligence Research, 44:179– 222. find
  16. Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource, related languages for neural machine translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 296–301. Asian Federation of Natural Language Pro… find our card
  17. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 866–875,… find our card
  18. Xing Niu, Michael Denkowski, and Marine Carpuat. 2018. Bi-directional neural machine translation with synthetic parallel data. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 84–91, Melbourne, Australia. Association for Computational Linguistics. find
  19. Petra Galušcáková and Ondrej Bojar. 2012. Improving smt by using parallel data of a closely related language. In Proc. of HLT, pages 58–65. find
  20. Duc Tam Hoang and Ondrej Bojar. 2016. Pivoting methods and data for czech-vietnamese translation via english. Baltic Journal of Modern Computing, 4(2):190–202. find
  21. Robert Östling, Yves Scherrer, Jörg Tiedemann, Gongbo Tang, and Tommi Nieminen. 2017. The helsinki neural machine translation system. In Proceedings of the Second Conference on Machine Translation, pages 338–347, Copenhagen, Denmark. Association for Computational Linguistics. find
  22. Melvin Johnson, Mike Schuster, Quoc Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, human and machine translation. arXiv:1609.08144. find
  23. Martin Popel and Ondej Bojar. 2018. Training Tips for the Transformer Model. The Prague Bulletin of Mathematical Linguistics, 110(1):43–70. arXiv preprint Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1. 0. In LREC. find
  24. Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, and Philip Williams. 2017. The university of edinburgh’s neural mt systems for wmt17. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Pape… find our card
  25. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575, Austin, Texas. Association for Computational Linguistics. find
  26. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational… find
  27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational L… find
  28. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  29. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2Tensor for Neural Machine Translation. In Proceedings of the 13th Conference of the Assoc… find our card
  30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Process… find
  31. Hua Wu and Haifeng Wang. 2007. Pivot language approach for phrase-based statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 856–863, Prague, Czech Republic. Association for Computational Linguistics. find
  32. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kocmi-2018-trivial-transfer-learning-low-resource/