Elamur Research⌕
☰
home / papers / 2020 / Extending Multilingual BERT to Low-Resource Languages

Extending Multilingual BERT to Low-Resource Languages

Paper · 2020 · Zihan Wang, Karthikeyan K, Stephen Mayhew et al.

Summary

Multilingual BERT (M-BERT) transfers well across languages in zero-shot settings, but its pre-training covers only the 104 largest Wikipedia languages. Many widely spoken low-resource languages, such as Oromo, Hausa or Uyghur, are left out. Rebuilding a multilingual model from scratch is costly, and a bilingual BERT (B-BERT) cannot use labelled data from several languages at once. The authors propose Extend. It builds a 30,000-word-piece vocabulary from target-language monolingual text and drops pieces M-BERT already has. The remaining pieces are appended to the embedding matrix and its tied output layer with fresh initialization, and M-BERT is then further pre-trained on the target-language corpus. The resulting E-MBERT is tested on zero-shot NER transferred from English, using LORELEI text and annotations for 27 languages, 11 of which M-BERT does not cover. On average, E-MBERT outperforms plain M-BERT by about 6 F1 points on covered languages and about 23 on new ones. It also usually beats B-BERT while needing far fewer training steps. This offers an inexpensive way to add a language to an existing multilingual encoder.

Key points

  • Setup: 27 languages (16 inside M-BERT, 11 outside), with corpora and NER data from LORELEI. The NER tagger is a Bi-LSTM-CRF built in AllenNLP, and F1 is averaged over five random seeds. Supervised M-BERT, estimated with cross-validation, serves as an upper-bound reference.
  • Training recipe: Extend continues pre-training for 500k iterations with batch size 32 and learning rate 2e-5. B-BERT needs 2M iterations at learning rate 1e-4. On Hindi and Sinhala, E-MBERT converges within about 100k steps, while B-BERT needs over 1M.
  • Gains without extra data: Extending with Wikipedia text (data M-BERT already saw) works as well as using LORELEI text. Thai rises from 22.46 to 38.35 F1 and Hindi from 48.31 to 62.77, while Russian stays roughly unchanged (about 56).
  • Versus B-BERT: Extend wins on 7 of 11 new languages. The largest gains are Uyghur (21.94 → 42.98) and Sinhala (16.93 → 33.97), which the authors attribute to related languages already in M-BERT (Turkish, Tamil). B-BERT is better on Hausa, Zulu, Kinyarwanda and, marginally, Wolof.
  • Trade-off and outlook: Specializing the model on one target language lowers its performance on other languages. Proposed future work includes extending to several languages at once, initializing new embeddings with MUSE or VecMap, and applying the approach to RoBERTa-style models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States233 works · 65,583 citations🎓University of Illinois Urbana-ChampaignUrbana · United States209 works · 71,394 citations🎓Indian Institute of Technology KanpurKanpur · India15 works · 799 citations🏢DuolingoPittsburgh · United States3 works · 616 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 worksLow-resource languages · 287 works

Cite

Cite as Zihan Wang, Karthikeyan K, Stephen Mayhew and Dan Roth. 2020. Extending Multilingual BERT to Low-Resource Languages. arXiv:2004.13640.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2004.13640 · arXiv 2004.13640

@misc{wang2020extending,
  title = {{Extending Multilingual BERT to Low-Resource Languages}},
  author = {Zihan Wang, Karthikeyan K, Stephen Mayhew and Dan Roth},
  year = {2020},
  doi = {10.48550/arxiv.2004.13640},
  eprint = {2004.13640},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2004.13640}
}

References

17 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–798, Melbourne, Australia…
  2. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Trans. Assoc. Comput. Linguistics, 7:597–610.
  3. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 7057–7067. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Multilingual bert - r. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  6. Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 462–471, Gothenburg, Sweden. Association for Computational Linguistic…
  7. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1– 6, Melbourne,…
  8. K Karthikeyan, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. find
  9. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, page…
  10. Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Herv Jgou. 2018. Word translation without parallel data. In International Conference on Learning Representations. find
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  12. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064–1074, Berlin, Germany. Association for Computational Linguistics.
  13. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics.
  14. Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1
  15. (Long and Short Papers), pages 3795–3805, Minneapolis, Minnesota. Association for Computational Linguistics. find
  16. Stephanie Strassel and Jennifer Tracey. 2016. Lorelei language packs: Data, tools, and resources for technology development in low resource languages. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 3273–3280.
  17. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844,…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2020-extending-multilingual-bert-low-resource/