Elamur Research⌕
☰
home / papers / 2020 / Monolingual Adapters for Zero-Shot Neural Machine Translation

Monolingual Adapters for Zero-Shot Neural Machine Translation

Paper · 2020 · Jerin Philip, Alexandre Berard, Matthias Gallé et al.

Summary

Multilingual NMT systems can, in principle, translate between language pairs never observed during training, but this zero-shot quality is usually poor. The authors address this by changing the adapter approach of Bapna and Firat (2019). Instead of one adapter per language direction, they attach one lightweight bottleneck adapter per language. Each source language gets an adapter in the encoder, and each target language gets one in the decoder. At inference, any source adapter can be paired with any target adapter, so the number of adapter sets grows linearly with the number of languages rather than quadratically. They first train a 20-language Transformer parent in fairseq on English-centric TED talks data. They then freeze it and train only the adapters. Monolingual adapters with a 1024-dimensional bottleneck beat the parent on unseen non-English pairs, with a median gain of +2.77 BLEU. They also match or exceed bilingual adapters while adding fewer parameters. This shows that compositional, language-specific modules are a cheap way to make multilingual models generalize to new translation directions.

Key points

  • Data: TED talks multiparallel corpus restricted to its 20 largest languages, with roughly 108k–214k parallel sentences per corpus.
    • The English-centric condition uses 38 directions.
    • Zero-shot evaluation covers the 342 pairs that exclude English.
  • Parameter cost: The parent model has 68M parameters. Total model growth across all 380 directions is:
    • ×380 for full fine-tuning;
    • ×4.53 for bilingual-64 adapters;
    • ×3.73 for mono-1024;
    • ×1.47 for mono-64.
  • Zero-shot results:
    • Mono-64 gives a median gain of +1.26 BLEU (worst pair −0.14).
    • Mono-1024 gives +2.77 BLEU (worst pair +0.30).
    • Bilingual adapters chained through English slightly beat mono-64 but not mono-1024.
  • Adaptation results: When adapters are trained on all 380 multiparallel directions, mono-1024 improves on the parent by a median of +5.59 BLEU, with gains ranging from −0.22 to +14.43.
  • Ablation:
    • In zero-shot translation, using only encoder adapters still helps more (+1.69) than using only decoder adapters (+0.63).
    • In the adaptation setting, encoder-only use collapses performance (−7.00), while decoder-only use gives +1.03.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢NaverSeongnam-si · South Korea15 works · 11,528 citations🎓Université Grenoble AlpesSaint-Martin-d'Hères · France7 works · 6,195 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Jerin Philip, Alexandre Berard, Matthias Gallé and Laurent Besacier. 2020. Monolingual Adapters for Zero-Shot Neural Machine Translation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.361

@inproceedings{philip2020monolingual,
  title = {{Monolingual Adapters for Zero-Shot Neural Machine Translation}},
  author = {Jerin Philip, Alexandre Berard, Matthias Gallé and Laurent Besacier},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.361},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.361}
}

References

25 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of NAACL-HLT.
  2. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019.
  3. Ankur Bapna and Orhan Firat. 2019. Simple, scalable adaptation for neural machine translation. In Proceedings of EMNLP-IJCNLP.
  4. Alexandre Berard, Ioan Calapodescu, and Claude Roux. 2019. Naver Labs Europe’s systems for the WMT19 machine translation robustness task. In Proceedings of WMT: Shared Task Papers. find
  5. Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. 2020. A comprehensive survey of multilingual neural machine translation. arXiv preprint arXiv:2001.01115. find
  6. Carlos Escolano, Marta R Costa-jussà, José AR Fonollosa, and Mikel Artetxe. 2020. Multilingual machine translation: Closing the gap between shared and language-specific encoder-decoders. arXiv preprint arXiv:2004.06575. find
  7. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of ICML. find
  8. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. TACL. find
  9. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR. find
  10. Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019. Investigating multilingual NMT representations at scale. In Proceedings of EMNLP-IJCNLP.
  11. Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. In Proceedings of EMNLP.
  12. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT: Demonstrations.
  13. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling Neural Machine Translation. In Proceedings of WMT: Research Papers.
  14. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07779.
  15. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Se-MAD-X: An adapter-bastian Ruder. 2020b. based framework for multi-task cross-lingual transfer. arXiv preprint arXiv:2005.00052. — [arXiv:2005.00052](https://arxiv.org/abs/2005.00052) find
  16. Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In Proceedings of NAACL-HLT.
  17. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In Proceedings of NeurIPS.
  18. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2018. Efficient parametrization of multi-domain deep neural networks. In Proceedings of CVPR. find
  19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of ACL.
  20. Asa Cooper Stickland, Xian Li, and Marjan Recipes for adapting Ghazvininejad. 2020. pre-trained monolingual and multilingual models to machine translation. find
  21. Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. In Proceedings of ICLR. find
  22. Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. 2020. UDapter: Language adaptation for truly universal dependency parsing. arXiv preprint arXiv:2004.14327. find
  23. Dušan Variš and Ondřej Bojar. 2019. Unsupervised pre-training for neural machine translation using elastic weight consolidation. In Proceedings of ACL: Student Research Workshop. find
  24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS.
  25. Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of ACL.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/philip-2020-monolingual-adapters-zero-shot-neural/