Elamur Research
☰
home / papers / 2019 / How Multilingual is Multilingual BERT?

How Multilingual is Multilingual BERT?

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1493.pdf

Summary

Most earlier probing work on contextual encoders looked only at what English models learn about English. This paper tests how well Multilingual BERT (M-BERT) carries knowledge from one language to another. M-BERT is a single 12-layer transformer pre-trained on Wikipedia in 104 languages, with a shared word-piece vocabulary and no explicit cross-lingual signal. The authors fine-tune it for NER (CoNLL-2002/2003 plus an in-house 16-language set) or POS tagging (UD, 41 languages) in one language. They then evaluate it in another language, with no target-language labels. Further probes relate transfer quality to entity word-piece overlap, to WALS word-order features, to code-switched and transliterated Hindi/English, and to translation retrieval with sentence vectors on WMT16. Zero-shot transfer is strong and depends little on shared vocabulary. It works even between languages in different scripts, which points to real shared multilingual representations. However, it weakens when word order differs and breaks down on transliterated input. For lower-resource languages such as Ukrainian, this helps predict when labeled data from a related language is worth borrowing.

Key points

  • Little dependence on shared vocabulary: Even when language pairs share almost no entity word pieces, M-BERT still gets roughly 40–70% NER F1. English-only BERT drops to near zero across scripts.
  • Transfer across scripts: After fine-tuning on Urdu POS data (Arabic script), M-BERT tags Hindi (Devanagari) with 91.1% accuracy. English→Bulgarian reaches 87.1%, but English→Japanese reaches only 49.4%.
  • Word order matters: Average POS accuracy is 81.55 for SVO→SVO transfer but 63.98 for SVO→SOV. More shared WALS features means better transfer.
  • Code-switching vs. transliteration: On the Hindi/English UD corpus of Bhat et al. (2018), training only on monolingual data gives 86.59% when Hindi is in Devanagari. Training on code-switched data gives 90.56%. With Hindi in Latin script, the monolingual-trained model falls to 50.41%.
  • Translation retrieval without fine-tuning: Shifting a sentence vector by one average offset per language pair finds the correct EN–DE translation more than 50% of the time in most layers. The first and last layers do worse.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

806 papers of our library cite it (of the 140673 whose bibliographies we read); 32 of them have a page here, 774 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1493 · arXiv 1906.01502

@inproceedings{pires2019how,
  title = {{How Multilingual is Multilingual BERT?}},
  author = {Pires, Telmo and Schlinger, Eva and Garrette, Dan},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1493},
  eprint = {1906.01502},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1493}
}

References

15 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. arXiv preprint arXiv:1812.10464. find our card
  2. Kelsey Ball and Dan Garrette. 2018. Part-of-speech tagging for code-switched, transliterated texts without explicit language identification. In Proceedings of EMNLP. find
  3. Irshad Bhat, Riyaz A. Bhat, Manish Shrivastava, and Dipti Sharma. 2018. Universal dependency parsing for Hindi-English code-switching. In Proceedings of NAACL. find
  4. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 metrics shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers. find our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL. find
  6. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. https://wals.info/. ↗
  7. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. find
  8. Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpradit, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strnadová, Esha Banerjee, Ruli Manurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo Mendonca, Tatiana Lando, Rattima Nitisaroj, and Josie Li… find
  9. CoNLL 2017 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of CoNLL. find our card
  10. Neural architectures for named entity recognition. In Proceedings of NAACL. find
  11. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  12. Tahira Naseem, Regina Barzilay, and Amir Globerson. 2012. Selective sharing for multilingual dependency parsing. In Proceedings of ACL. find
  13. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan T. McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of LREC. find our card
  14. Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018a. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of EMNLP. find our card
  15. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018b. Deep contextualized word representations. In Proceedings of NAACL. find