Elamur Research⌕
☰
home / papers / 2019 / Investigating Multilingual NMT Representations at Scale

Investigating Multilingual NMT Representations at Scale

Summary

Multilingual NMT models transfer well to low-resource languages and downstream tasks, but it is unclear how their internal representations make this possible. The authors examine a single Transformer-Big model trained jointly on 103 languages. Training uses a web-crawled, English-centric in-house corpus that is heavily imbalanced across language pairs. They compare the model's hidden states across languages, layers and fine-tuned variants with SVCCA. For this they average activations over time steps and use a multi-way parallel evaluation set, so every sentence has the same meaning in all languages. The analysis finds three things. Encoder representations group languages by family and branch. What the encoder produces for a source language is shaped by the target language, and the reverse also holds. After fine-tuning on one language pair, high-resource languages and relatives of the fine-tuned language change least. Smaller shifts in representation go together with smaller BLEU losses. These results help explain why transfer works better between related languages. They also suggest practical steps, such as adapting from a low-resource language's nearest neighbours or adding losses that keep distant languages from being distorted.

Key points

  • Training data: more than 25 billion sentence pairs covering 102 languages to and from English, which gives 204 direct pairs. Each pair has between 10^4 and 10^9 examples.
  • Model and evaluation set:
    • Transformer-Big with all parameters shared across languages.
    • A shared SentencePiece vocabulary of 64k tokens.
    • A <2xx> token added to each source sentence to mark the target language.
    • A multi-way aligned evaluation set of roughly 3k sentences per language.
  • Method: SVCCA keeps 99% of the variance and applies CCA to sentence-level representations obtained by mean-pooling. Language maps are drawn with Spectral Embeddings (Laplacian Eigenmaps).
  • Fine-tuning experiment: the model was fine-tuned for 40k steps on 12 X-En pairs from 6 sub-families, including uk-en, ru-en, sr-en, tr-en and ny-en.
    • Chinese, German, Russian and Italian stayed stable whichever pair was used.
    • Slavic, Turkic and Romance languages held up when fine-tuning used a related language.
    • Fine-tuning on ny-en degraded all languages to a similar degree.
  • Embeddings: input embeddings barely changed under fine-tuning (ρ̄ > 0.98). They also overlap across languages less than the final encoder outputs do.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

79 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 75 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1167

@inproceedings{kudugunta2019investigating,
  title = {{Investigating Multilingual NMT Representations at Scale}},
  author = {Kudugunta, Sneha and Bapna, Ankur and Caswell, Isaac and Firat, Orhan},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1167},
  url = {https://doi.org/10.18653/v1/d19-1167}
}

References

47 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. arXiv preprint arXiv:1903.00089. find our card
  2. Maruan Al-Shedivat and Ankur P. Parikh. 2019. Consistency by agreement in zero-shot neural machine translation. CoRR, abs/1904.02338. find
  3. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2019a. The missing ingredient in zero-shot neural machine translation. arXiv preprint arXiv:1903.07091. — [arXiv:1903.07091](https://arxiv.org/abs/1903.07091) find
  4. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2018. Identifying and controlling important neurons in neural machine translation. arXiv preprint arXiv:1811.01157. find
  5. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? arXiv preprint arXiv:1704.03471. find our card
  6. Yonatan Belinkov, Lluı́s Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2018. Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks. arXiv preprint arXiv:1801.07772. find our card
  7. Mikhail Belkin and Partha Niyogi. 2003. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1373– 1396. find
  8. Wayles Browne and Vyacheslav Vsevolodovich Ivanov. Slavic languages’ family tree. Encyclopdia Britannica. find
  9. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in ne… find our card
  10. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. find
  11. Sandagomi Coperahawa. 2007. Language Contact and Linguistic Area: the Sinhala - Tamil Contact Situation. Journal of the Royal Asiatic Society of Sri Lanka, 53:133 – 152. find
  12. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019b. Massively multilingual neural machine translation in the wild: Findings and challenges. find our card
  13. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, D Anthony Bau, and James Glass. 2019. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 7. find
  14. Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. arXiv preprint arXiv:1812.10464. find our card
  15. Akiko Eriguchi, Melvin Johnson, Orhan Firat, Hideto Kazawa, and Wolfgang Macherey. 2018. Zero-shot cross-lingual classification using multilingual neural machine translation. arXiv preprint arXiv:1809.04686. — [arXiv:1809.04686](https://arxiv.org/abs/1809.04686) find
  16. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations. find our card
  17. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. arXiv preprint arXiv:1601.01073. find our card
  18. Harald Hammarström, Robert Forkel, and Martin Haspelmath. 2017. Hindustani. find
  19. David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. 2004. Canonical correlation analysis: An overview with application to learning methods. Neural computation, 16(12):2639–2664. find
  20. Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource, related languages for neural machine translation. In Proc. IJCNLP, volume 2, pages 296–301. find our card
  21. Marvin Irving Herzog. West Germanic Languages. find
  22. Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. 2018. The building blocks of interpretability. Distill, 3(3):e10. find
  23. Pavle Ivić. 2011. Encyclopaedia Britannica. find
  24. A. Phillips and M Davis. 2009. Tags for Identifying Languages. RFC 5646, RFC Editor. find
  25. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Googles multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational L… find
  26. Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287–297. find our card
  27. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521… find
  28. Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1318–1326. Association for Computational Linguistics. find our card
  29. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226. find our card
  30. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  31. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109– 165. Elsevier. find
  32. Etukoori Balaraama Moorti. 2011. Proto Dravidian. Study of Dravidian Linguistics and Civilization. find
  33. Ari S. Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, NIPS’18, pages 5732–5741, USA. Curran Associates Inc. find
  34. Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. arXiv preprint arXiv:1808.04189. find our card
  35. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, pages 6076– 6085. find
  36. Hichem Sahbi. 2018. Learning cca representations for misaligned data. In Proceedings of the European Conference on Computer Vision (ECCV), pages 0– 0. find
  37. Naomi Saphra and Adam Lopez. 2018. Understanding learning dynamics of language models with svcca. arXiv preprint arXiv:1811.00225. find
  38. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235. find
  39. Mohammad Tahsin Siddiqi. 1994. Hindustani-English code-mixing in modern literary texts. University of Wisconsin. find
  40. Roland Sussex and Paul Cubberley. The Slavic Languages. find
  41. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. find
  42. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316. find our card
  43. Jakob Uszkoreit, Jay M Ponte, Ashok C Popat, and Moshe Dubiner. 2010. Large scale parallel document mining for machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1101–1109. Association for Computational Linguistics. find
  44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  45. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. CoRR, abs/1904.09077. find our card
  46. Kelly W Zhang and Samuel R Bowman. 2018. Language modeling teaches you more syntax than translation does: Lessons learned through auxiliary task analysis. arXiv preprint arXiv:1809.10040. find
  47. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kudugunta-2019-investigating-multilingual-nmt-representations-scale/