Elamur Research
☰
home / papers / 2018 / When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?

When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N18-2084.pdf

Summary

Neural MT performs poorly when little parallel data is available. Pre-trained word vectors are widely used in other NLP tasks, but it was unclear when and why they help translation. The authors ran five controlled experiments on a multilingual TED talks corpus. It contains translations into English from three language pairs, each with one low-resource language and one closely related higher-resource language: Galician/Portuguese, Azerbaijani/Turkish and Belarusian/Russian. They used a 1-layer attentional encoder-decoder in xnmt and initialized its embeddings with 300-dimensional fastText vectors trained on Wikipedia. The experiments varied the side where pre-training is applied, training set size, language similarity, cross-lingual alignment of embeddings, and bilingual versus multilingual training.

Pre-training helps mostly on the source side. It works best when the baseline is weak but still trainable. Aligning embedding spaces is unnecessary for bilingual systems but useful when one encoder is shared across similar languages. The gains concentrate on rare words. These results give practical guidance for low-resource MT, for example for Ukrainian-related language pairs.

Key points

  • Higher-resource pairs (PT, TR, RU→EN) gain about 3 BLEU from pre-training. Galician→EN jumps from 2.2 to 13.2 BLEU with pre-trained source embeddings. Gains for AZ and BE are small.
  • Down-sampling experiments show a "sweet spot": the largest gains appear when the baseline BLEU is roughly 3–4.
  • With Portuguese as the target and 40,000 training sentences, gains were ES +7.0, FR +5.7 and IT +4.7. The distant languages RU (+6.2) and HE (+8.9) gained more because their baselines were very low.
  • Orthogonal alignment (Smith et al., 2017) had no meaningful effect in bilingual systems. In multilingual GL+PT training it raised Galician to 22.4 BLEU, compared with 2.2 for the bilingual baseline, which is a gain of about 20 BLEU. BE+RU did not benefit from pre-training.
  • Code and data are available at github.com/neulab/word-embeddings-for-nmt. Analyses with compare-mt.py and frequency-bucketed F-measure show better handling of low-frequency words, names and phrases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

174 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 159 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-2084

@inproceedings{qi2018when,
  title = {{When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?}},
  author = {Qi, Ye and Sachan, Devendra and Felix, Matthieu and Padmanabhan, Sarguna and Neubig, Graham},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-2084},
  url = {https://doi.org/10.18653/v1/n18-2084}
}

References

26 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In HLT-NAACL. find
  2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041 . find our card
  3. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 1064–1074. http: //www… find our card
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv e-prints abs/1409.0473. https://arxiv.org/abs/1409.0473. ↗ our card
  5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606 . find
  6. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics 19(2):313–330. find our card
  7. Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation. arXiv preprint arXiv:1606.04596 . find
  8. P.H. Matthews. 1997. The Concise Oxford Dictionary of Linguistics.. Oxford Paperback Reference / Oxford University Press, Oxford. Oxford University Press, Incorporated. https://books.google.com/ books?id=aYoYAAAAIAAJ. ↗
  9. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087 . find
  10. Greville Corbett and Bernard Comrie. 2003. Slavonic Languages. Routledge. find
  11. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2017. Advances in pre-training distributed word representations . find our card
  12. The Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. pages 3111–3119. find
  13. Michael Denkowski and Graham Neubig. 2017. Stronger baselines for trustable results in neural machine translation. arXiv preprint arXiv:1706.09733 . find
  14. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. arXiv preprint arXiv:1601.01073 . find our card
  15. Masato Neishi, Jin Sakuma, Satoshi Tohda, Shonosuke Ishiwatari, Naoki Yoshinaga, and Masashi Toyoda. 2017. A bag of useful tricks for practical neural machine translation: Embedding layer initialization and large batch size. In Proceedings of the 4th Workshop on Asian Translation (WAT2017). Asian Fe… find
  16. Mattia Antonino Di Gangi and Marcello Federico. 2017. Monolingual embeddings for low resourced neural machine translation. In International Workshop on Spoken Language Translation (IWSLT). find
  17. Graham Neubig, Matthias Sperber, Xinyi Wang, Matthieu Felix, Austin Matthews, Sarguna Padmanabhan, Ye Qi, Devendra Singh Sachan, Philip Arthur, Pierre Godard, John Hewitt, Rachid Riad, and Liming Wang. 2018. XNMT: The extensible neural machine translation toolkit. In Conference of the Association fo… find
  18. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. pages 249–256. find
  19. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in Neural Information Processing Systems. pages 820–828. find
  20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 311–318. find our card
  21. Melvin Johnson et al. 2016. Google’s multilingual neural machine translation system: Enabling zero-shot translation. arXiv preprint arXiv:1611.04558 . find
  22. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In In EMNLP. Citeseer. find our card
  23. Prajit Ramachandran, Peter J Liu, and Quoc V Le. 2016. Unsupervised pretraining for sequence to sequence learning. arXiv preprint arXiv:1611.02683 . find
  24. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . find
  25. Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. arXiv preprint arXiv:1702.03859 . find our card
  26. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find