Elamur Research⌕
☰
home / papers / 2020 / Zero-Shot Cross-Lingual Transfer with Meta Learning

Zero-Shot Cross-Lingual Transfer with Meta Learning

Paper · 2020 · Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva et al.

Summary

Most of the world's languages have little labelled data for natural language understanding, so models trained only on English often transfer poorly to other languages. The authors propose X-MAML, a cross-lingual adaptation of Model Agnostic Meta-Learning (MAML). It works in three stages:

  1. The model is first trained on plentiful English data.
  2. Small task batches drawn from the development sets of one or two auxiliary languages drive gradient-based meta-updates. These updates search for parameters that adapt quickly to new languages.
  3. The resulting model is either tested directly on target languages (zero-shot) or briefly fine-tuned on their development data (few-shot).

The approach is evaluated on cross-genre MultiNLI, XNLI and MLQA, using Multilingual BERT and XLM-RoBERTa as base models and covering 15 languages. X-MAML yields consistent improvements and exceeds previous state-of-the-art results for zero-shot XNLI and MLQA. Gains appear even between typologically distant languages. An analysis based on WALS ties the benefits partly to shared morphosyntactic properties. The work shows that a small amount of auxiliary-language data can make transfer to under-resourced languages noticeably stronger.

Key points

  • Datasets:
    • MultiNLI: 433k sentence pairs in 10 genres, only 5 of which appear in training.
    • XNLI: 5000 test and 2500 development English pairs, translated into 14 languages.
    • MLQA: 7 languages, with over 12k English instances and about 5k for each other language.
  • Training setup: implemented with the higher library and the Adam optimiser, batch size 32, α = 1e−4, β = 1e−5, K = Q = 16. 100 meta-learning iterations worked best, and results are averaged over 10 runs. The code is released at github.com/copenlu/X-MAML.
  • Zero-shot XNLI: nearly every auxiliary language improved results, with Swahili as the only exception. Russian → Hindi was among the strongest pairings.
  • Typology analysis: a logistic regression was trained per WALS feature and compared to baselines with a paired t-test using Bonferroni correction (p = 0.00025). Two features stood out:
    • 67A The Future Tense matters in few-shot transfer.
    • 25A Locus of Marking matters in zero-shot transfer; dependent-marking languages (en, de, ru, zh) gained most from each other.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CopenhagenCopenhagen · Denmark78 works · 14,228 citations🎓University of ZurichZurich · Switzerland50 works · 11,138 citations🎓University of OsloOslo · Norway29 works · 3,700 citations🎓KU LeuvenLeuven · Belgium25 works · 5,716 citations🎓Vrije Universiteit BrusselBrussels · Belgium10 works · 695 citations🎓Aalborg UniversityAalborg · Denmark6 works · 695 citations

Also on the byline: IMEC.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 467 works

Cite

Cite as Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva and Isabelle Augenstein. 2020. Zero-Shot Cross-Lingual Transfer with Meta Learning. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.368

@inproceedings{nooralahzadeh2020zero,
  title = {{Zero-Shot Cross-Lingual Transfer with Meta Learning}},
  author = {Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva and Isabelle Augenstein},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.368},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.368}
}

References

46 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mostafa Abdou, Cezar Sas, Rahul Aralikatte, Isabelle Augenstein, and Anders Søgaard. 2019. X-WikiRE: A Large, Multilingual Resource for Relation Extraction as Machine Comprehension. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 265–274, Hong… find
  2. Zeljko Agic and Natalie Schluter. 2018. Baselines and Test Data for Cross-Lingual Inference. In LREC. European Language Resources Association (ELRA). find
  3. Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. 2016. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pages 3981–3989. find
  4. Emily M Bender. 2019. The \#BenderRule: On Naming the Languages We Study and Why It Matters. find
  5. Johannes Bjerva. 2017. One Model to Rule them all – Multitask and Multilingual Modelling for Lexical Analysis. Ph.D. thesis, University of Groningen. find
  6. Johannes Bjerva and Isabelle Augenstein. 2018a. From Phonology to Syntax: Unsupervised Linguistic Typology at Different Levels with Language Embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  7. Johannes Bjerva and Isabelle Augenstein. 2018b. Tracking Typological Traits of Uralic Languages in Distributed Language Representations. In Proceedings of the Fourth International Workshop on Computational Linguistics of Uralic Languages, pages 76–86, Helsinki, Finland. Association for Computational… find
  8. Johannes Bjerva, Yova Kementchedjhieva, Ryan Cotterell, and Isabelle Augenstein. 2019a. A Probabilistic Generative Model of Linguistic Typology. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1… find
  9. Johannes Bjerva, Yova Kementchedjhieva, Ryan Cotterell, and Isabelle Augenstein. 2019b. Uncovering Probabilistic Implications in Typological Knowledge Bases. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3924–3930, Florence, Italy. Association for… find
  10. Johannes Bjerva, Robert Östling, Maria Han Veiga, Jörg Tiedemann, and Isabelle Augenstein. 2019c. What Do Language Representations Really Represent? Computational Linguistics, 45(2):381–389. find
  11. David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2019. Ethnologue: Languages of the World. https://www.ethnologue.com/ statistics/size . Accessed: 2019-05-25. ↗
  12. Manaal Faruqui and Shankar Kumar. 2015. Multilingual Open Relation Extraction Using Cross-lingual Projection. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1351–1356. Association for Computatio… find
  13. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, and Hui Jiang. 2016. Enhancing and Combining Sequential and Tree LSTM for Natural Language Inference. CoRR, abs/1609.06038. find
  14. Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1126–1135. JMLR. org. find
  15. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. Proceedings of the 58th Annual Meeting of the Association for… find
  16. Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018. Meta-Learning for Low-Resource Neural Machine Translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3622–3631, Brussels, Belgium. Association for Computational Linguistic…
  17. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual Language Model Pretraining. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 7057–7067. find
  18. Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803–480… find
  19. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating Cross-lingual Sentence Representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475– 2485. Association… find
  20. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  21. Zi-Yi Dou, Keyi Yu, and Antonios Anastasopoulos. 2019. Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Languag…
  22. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  23. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization. CoRR, abs/2003.11080.
  24. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp…
  25. Yova Kementchedjhieva, Johannes Bjerva, and Isabelle Augenstein. 2018. Copenhagen at CoNLL– SIGMORPHON 2018: Multilingual Inflection in Context with Explicit Morphosyntactic Decoding. In Proceedings of the CoNLL–SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection, pages 93–98, Brussels… find
  26. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980. find
  27. Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015. Siamese Neural Networks for One-shot Image Recognition. In ICML deep learning workshop. find
  28. Souvik Kundu and Hwee Tou Ng. 2018. A Nil-Aware Answer Extraction Framework for Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4243–4252, Brussels, Belgium. Association for Computational Linguistics. find
  29. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-Shot Relation Extraction via Reading Comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342, Vancouver, Canada. Association for Computational Linguistics. find
  30. Nils Rethmeier and Isabelle Augenstein. 2020. Long-Tail Zero and Few-Shot Learning via Contrastive Pretraining on and for Small Data. arXiv preprint arXiv:2010.01061. — [arXiv:2010.01061](https://arxiv.org/abs/2010.01061) find
  31. Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy P. Lillicrap. 2016. Meta-Learning with Memory-Augmented Neural Networks. In ICML, volume 48 of JMLR Workshop and Conference Proceedings, pages 1842– 1850. JMLR.org. find
  32. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating Cross-lingual Extractive Question Answering. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
  33. Patrick Verga, David Belanger, Emma Strubell, Benjamin Roth, and Andrew McCallum. 2016. Multilingual Relation Extraction using Compositional Universal Schema. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… find
  34. Miryam de Lhoneux, Johannes Bjerva, Isabelle Augenstein, and Anders Søgaard. 2018. Parameter sharing between dependency parsers for related languages. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4992–4997, Brussels, Belgium. Association for Comput… find
  35. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. 2016. Matching Networks for One Shot Learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 3630–3638. Curran Associat… find
  36. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Bruce Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, and Ming Zhou. 2020. XGLUE: A New Benchmark Datase…
  37. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
  38. Alex Nichol, Joshua Achiam, and John Schulman. 2018. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999. find
  39. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  40. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 412–41…
  41. Kun Qian and Zhou Yu. 2019. Domain Adaptive Dialog Generation via Meta Learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2639–2649, Florence, Italy. Association for Computational Linguistics. find
  42. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100, 000+ Questions for Machine Comprehension of Text. CoRR, abs/1606.05250.
  43. Sachin Ravi and Hugo Larochelle. 2017. Optimization as a model for few-shot learning. In International Conference on Learning Representations (ICLR). OpenReview.net. find
  44. Shijie Wu and Mark Dredze. 2019. Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT. CoRR, abs/1904.09077.
  45. Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse Few-Shot Text Classification with Multiple Metrics. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: H…
  46. Wei Zhao, Steffen Eger, Johannes Bjerva, and Isabelle Augenstein. 2020. Inducing Language-Agnostic Multilingual Representations. arXiv preprint arXiv:2008.09112. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nooralahzadeh-2020-zero-shot-cross-lingual-transfer/