Elamur Research⌕
☰
home / papers / 2020 / From English To Foreign Languages: Transferring Pre-trained Language Models

From English To Foreign Languages: Transferring Pre-trained Language Models

Paper · 2020 · Ke Tran

Summary

The paper asks whether an English pretrained encoder can be reused for other languages on a small compute budget, instead of training a multilingual model from scratch. Tran builds bilingual models, called RAMEN, from BERT and RoBERTa in two stages. First, every foreign subword receives an initial embedding that is a sparse weighted mix of English embeddings. The weights come from word-alignment probabilities (fast-align on parallel corpora) or from fastText vectors aligned across languages and passed through sparsemax. Second, the model gets separate foreign embedding and output layers around the shared English encoder, and all parameters are fine-tuned on Wikipedia text in both languages. Tests cover six languages (fr, ru, ar, zh, hi, vi) on zero-shot XNLI and UD v2.4 parsing. On a single GPU, a BASE model is ready in about 20 hours and a LARGE model in about 46. RAMEN matches or beats mBERT, and stronger English source models give stronger bilingual models. Cheap cross-lingual transfer therefore becomes realistic for smaller teams.

Key points

  • Hardware cost: one Tesla V100 16GB. RAMEN BASE takes about 20 hours and RAMEN LARGE about 46 hours. After roughly 40,000 updates (around 7 hours), RAMEN BASE+BERT already beats mBERT on both zero-shot tasks.
  • Data:
    • Parallel corpora: UN Parallel Corpus (en-fr/ru/ar/zh), IIT Bombay (en-hi) and OpenSubtitles 2018 (en-vi).
    • Monolingual route: pretrained fastText vectors aligned with orthogonal Procrustes, using identical strings as the only supervision.
    • Target vocabulary: fastBPE, truncated to 32k (BERT) or 50k (RoBERTa).
  • Initialization ablation: random foreign embeddings lower average zero-shot scores by 10.3 points on XNLI and 11.6 LAS on UD. The SOV languages ar and hi are hit hardest.
  • Comparison with training from scratch: a bilingual BERT trained from scratch for 400 GPU hours (2.3M updates) is outperformed by RAMEN after only 20,000 updates (about 3.5 GPU hours).
  • Language similarity: zero-shot LAS depends strongly on closeness to English. French reaches 78.0, while zh, ar and hi stay roughly between 30 and 40.
  • Supervised UD parsing (average LAS): mBERT 84.6, RAMEN BASE+BERT 85.0, RAMEN LARGE+RoBERTa 86.5.

Interesting ideas

  • I1. An unseen item can get its starting vector as a sparse, normalized blend of vectors for known items it resembles; this makes training converge faster and gives usable zero-shot behaviour from the first step.
  • I2. Sparsemax applied to similarity scores keeps only a handful of plausible correspondences with nonzero weight, so the mapping stays readable and is less blurred than a softmax over everything.
  • I3. During adaptation, the model keeps training on the original source data alongside the new data, so it does not lose its existing skills and can still transfer across languages.
  • I4. The method is judged against two reference points: a random-initialization variant and a model trained from scratch with about 16 times more compute, which shows how much each component contributes.
  • I5. Downstream metrics (accuracy and LAS) are tracked at each checkpoint, which reveals how early the adapted model overtakes the baseline.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): When gparse meets rare or new subwords, their embeddings could start as a sparse mix of embeddings for known lemmas linked by morphology or WordNet synonyms, rather than random noise.
  • I2 (world): Linking a word to WordNet/FrameNet/Wikidata concepts could produce a short sparse candidate list with explicit weights, which fits the explainability requirement better than a dense ranking.
  • I3 (learn): Retraining the parser on silver or domain-specific trees could always mix in gold English UD treebank batches, so the new skill does not erode baseline LAS.
  • I4 (learn): Every new trick, such as repair rules or teacher tiers, could be reported against a random or ablated control and a brute-force baseline, so its real gain is measured honestly.
  • I5 (learn): Logging LAS and answer accuracy per checkpoint during gparse training would show when to stop training and whether extra compute is still paying off.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AmazonSeattle · United States112 works · 40,565 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Ke Tran. 2020. From English To Foreign Languages: Transferring Pre-trained Language Models. arXiv:2002.07306.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2002.07306 · arXiv 2002.07306

@misc{tran2020english,
  title = {{From English To Foreign Languages: Transferring Pre-trained Language Models}},
  author = {Ke Tran},
  year = {2020},
  doi = {10.48550/arxiv.2002.07306},
  eprint = {2002.07306},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2002.07306}
}

References

29 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joakim Nivre et al. 2019. Universal dependencies 2.4. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  2. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2289–2294, Austin, Texas. Association for Computational Linguistics.
  3. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. On the Cross-lingual Transferability of Monolingual Representations. arXiv e-prints, page arXiv:1910.11856. ↗
  4. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. ↗
  5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗
  6. Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263– 311. find
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. find
  8. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics. ↗
  9. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In ACL. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  11. Timothy Dozat and Christopher D. Manning. 2016. Deep biaffine attention for neural dependency parsing. In ICLR.
  12. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia. Association for Computational Linguistics. find
  13. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 866–875, San Diego, California. Association for Computational Linguistics.
  14. Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. 2018. Loss in translation: Learning bilingual word mapping with a retrieval criterion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2979–2984, Brussels, Belgium. Association for Computational Linguistics.
  15. Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual {bert}: An empirical study. In International Conference on Learning Representations. ↗
  16. Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2018. The IIT Bombay English-Hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Languages Resources Association (ELRA).
  17. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. NeurIPS. find
  18. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations. ↗
  19. Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018b. Word translation without parallel data. In International Conference on Learning Representations. ↗
  20. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics, 5:365–378. ↗
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  22. André F. T. Martins and Ramón F. Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, pages 1614–1623. JMLR.org. ↗
  23. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  24. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.
  26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ↗
  27. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong, China. Association for Computational Linguistics. ↗
  28. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. NeurIPS. ↗
  29. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. European Language Resources Association (ELRA).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tran-2020-english-foreign-languages-transferring-pre/