Elamur Research
☰
home / papers / 2018 / Universal Neural Machine Translation for Extremely Low Resource Languages

Universal Neural Machine Translation for Extremely Low Resource Languages

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N18-1032.pdf

Summary

Neural machine translation usually depends on millions of sentence pairs, so it performs poorly for languages with only a few thousand parallel sentences or none at all. Gu et al. build on the multilingual NMT setup of Johnson et al. (2017), where many source languages are translated into one target language (English), and add two sharing mechanisms.

  • Universal Lexical Representation (ULR) turns every source word into a soft mixture of English-based "universal tokens." Monolingual embeddings for each language are first aligned to a common space using a small seed dictionary. Similarity scores then select universal-token embeddings, which are learned during NMT training. As a result, words never seen in parallel data still receive useful representations, and the method does not depend on shared spelling or scripts.
  • Mixture of Language Experts (MoLE) lets a low-resource language reuse the sentence-level encoders of related high-resource languages.

On Romanian-English WMT2016 with just 6k parallel sentences, the system reaches 23 BLEU, compared with 18 for a strong multilingual baseline that also uses back-translation. The work shows that cross-lingual transfer can make NMT practical for languages with very scarce parallel data.

Key points

  • Main result: 23 BLEU on Romanian-English WMT2016 with 6k sentence pairs, versus 18 BLEU for a multilingual baseline with back-translation.
  • In ULR, a query–key matching step links language-specific monolingual embeddings (E_Q) to universal-token embeddings (E_K). Only the NMT embeddings E_U and a shared transformation matrix A (initialized as identity) are trained. Per-language projections are orthogonal and learned from seed word pairs.
  • Auxiliary languages included Es, Pt, Fr, It, Cs, El, De and Fi. MoLE activations roughly followed relatedness to Romanian (Es ≈ Pt > Fr ≈ It > Cs ≈ El > De > Fi). The authors attribute Czech's strong influence to geographic and phrasal contact rather than shared language family.
  • Fine-tuning experiment:
    • A Multi-NMT+ULR model was trained only on {Es, Fr, It, Pt}-En.
    • Zero-shot, it scored about 10.7 BLEU on Ro-En.
    • Three epochs on the 6k Ro-En sentences raised this to nearly 20 BLEU, about 6 BLEU above the baseline, in under 2 minutes of training.
  • The work is related to unsupervised NMT approaches (Artetxe et al., 2018; Lample et al., 2018). The authors name meta-learning for fine-tuning as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

128 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 120 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1032

@inproceedings{gu2018universal,
  title = {{Universal Neural Machine Translation for Extremely Low Resource Languages}},
  author = {Gu, Jiatao and Hassan, Hany and Devlin, Jacob and Li, Victor O.K.},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1032},
  url = {https://doi.org/10.18653/v1/n18-1032}
}

References

25 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In ACL. find our card
  2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In Proceedings of International Conference on Learning Representations (ICLR). Vancouver, Canada. find our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of International Conference on Learning Representations (ICLR). find our card
  4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5:135–146. find
  5. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In ICLR. find
  6. Jacob Devlin. 2017. Sharp models on dull hardware: Fast and accurate neural machine translation decoding on the cpu. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, pages 2810–2815. find
  7. Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (ICML). find
  8. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL). find our card
  9. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of International Conference on Machine Learning (ICML). find our card
  10. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zh… find our card
  11. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735– 1780. find
  12. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Ass… find
  13. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. AAAI Press, AAAI’16, pages 2741–2749. find our card
  14. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . find
  15. Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Unsupervised machine translation using monolingual corpora only. In Proceedings of International Conference on Learning Representations (ICLR). Vancouver, Canada. find our card
  16. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. TACL 5:365– 378. find our card
  17. Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pa… find our card
  18. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for wmt 16. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 371– 376. find our card
  19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 1715–1725. find
  20. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In Proceedings of International Conference on Learning Representations (ICLR). find
  21. Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proceedings of International Conference on Learning Representations (ICLR). find our card
  22. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS). find
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS). find
  24. Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, Ł. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corra… find
  25. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 1568–1575. find