Elamur Research⌕
☰
home / papers / 2022 / Investigating Language Relationships in Multilingual Sentence Encoders Through the Lens of Linguistic Typology

Investigating Language Relationships in Multilingual Sentence Encoders Through the Lens of Linguistic Typology

Paper · 2022 · Rochelle Choenni and Ekaterina Shutova

Summary

Multilingual sentence encoders support cross-lingual transfer, but it is unclear which properties of individual languages they capture and how they represent typological similarity and difference between languages. Choenni and Shutova study four encoders that differ in architecture and pretraining objective: LASER, M-BERT, XLM, and XLM-R. Earlier work showed that encoder representations split into a language-neutral part and a language-specific part. The authors isolate the language-specific part and probe it, layer by layer, with 25 language-level tasks based on WALS features covering lexicon, morphology, and syntax. They also propose "cross-neutralizing": they strip one language's specific information and check whether probe accuracy drops for other languages with the same feature value. All encoders capture word order, negation, and pronoun features, while M-BERT and XLM-R do better on several lexical and morphological properties. Cross-lingual objectives push models toward a shared interlingua, whereas monolingual objectives preserve language-specific subspaces. Typological properties turn out to be encoded jointly across similar languages. The study shows how pretraining choices shape what multilingual models retain about individual languages, which matters for transfer to languages such as Ukrainian.

Key points

  • Probing setup: 25 typological probing tasks built from the World Atlas of Language Structures (WALS) were applied to LASER, M-BERT, XLM, and XLM-R. Bulgarian, French, Marathi, Polish, Spanish, Swedish, and Ukrainian served as test languages.
  • Where the information sits: In M-BERT and XLM-R, typological information is spread across all layers. In LASER and XLM, it is concentrated in the lower layers. t-SNE plots show LASER and XLM merging languages in their top layers, while M-BERT and XLM-R separate them again.
  • Main trade-off: The more universal a model's representations are, the less language-specific information it keeps, and this holds regardless of architecture.
  • Not just similar-language matching: When the closest training language was swapped for German, the XLM-based probe still mostly succeeded. For Ukrainian it worked in 6 of 7 tasks with 99.5% average accuracy, but it failed in all 3 Marathi tasks.
  • Unseen languages: XLM was not trained on Ukrainian, Polish, or Marathi, yet it reached macro-F1 scores on these languages that were comparable to, and sometimes higher than, LASER's.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Rochelle Choenni and Ekaterina Shutova. 2022. Investigating Language Relationships in Multilingual Sentence Encoders Through the Lens of Linguistic Typology. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00444

@article{choenni2022investigating,
  title = {{Investigating Language Relationships in Multilingual Sentence Encoders Through the Lens of Linguistic Typology}},
  author = {Rochelle Choenni and Ekaterina Shutova},
  journal = {Computational Linguistics},
  year = {2022},
  doi = {10.1162/coli\_a\_00444},
  url = {https://doi.org/10.1162/coli_a_00444}
}

References

32 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agić, Željko, Jörg Tiedemann, Kaja Dobrovoljc, Simon Krek, Danijela Merkler, and Sara Može. 2014. Cross-lingual dependency parsing of related languages with rich morphosyntactic tagsets. In EMNLP 2014 Workshop on Language Technology for Closely Related Languages and Language Variants, pages 13–24.… find
  2. Ammar, Waleed, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016a. Many languages, one parser. Transactions of the Association for Computational Linguistics, 4:431–444. https://doi.org/10.1162/tacl_a_00109 ↗
  3. Ammar, Waleed, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A. Smith. 2016b. Massively multilingual word embeddings. arXiv preprint arXiv:1602.01925.
  4. Artetxe, Mikel and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. https://doi.org/10.1162/tacl_a_00288 Banea, Carmen, Rada Mihalcea, Janyce Wiebe, and Samer H… ↗
  5. Beinborn, Lisa and Rochelle Choenni. 2020. Semantic drift in multilingual representations. Computational Linguistics, 46(3):571–603. https://doi.org/10 .1162/coli_a_00382 Bjerva, Johannes, Robert Östling, Maria Han Veiga, Jörg Tiedemann, and Isabelle Augenstein. 2019. What do language representation…
  6. Blevins, Terra, Omer Levy, and Luke Zettlemoyer. 2018. Deep RNNs Encode Soft Hierarchical Syntax. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 14–19. https://doi.org/10 .18653/v1/P18-2003 Chen, Xilun and Claire Cardie. 201…
  7. Conneau, Alexis, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018b. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485. https://doi… find
  8. Dryer, Matthew S. 2013. Order of subject, object and verb. In Matthew S. Dryer and Martin Haspelmath, editors, The World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. Dryer, Matthew S. and Martin Haspelmath, editors. 2013. WALS Online, Max Planck I… find
  9. Duong, Long, Trevor Cohn, Steven Bird, and Paul Cook. 2015a. Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural L… find
  10. Ganchev, Kuzman, Jennifer Gillenwater, and Ben Taskar. 2009. Dependency grammar induction via bitext projection constraints. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-V…
  11. Hewitt, John and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Volume 1 (Long and Short Pa…
  12. Huang, Haoyang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internation…
  13. Khapra, Mitesh M., Salil Joshi, Arindam Chatterjee, and Pushpak Bhattacharyya. 2011. Together we can: Bilingual bootstrapping for WSD. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 561–569. find
  14. Kim, Taeuk, Bowen Li, and Sang-goo Lee. 2021. Multilingual chart-based constituency parse extraction from pre-trained language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 454–463. https://doi.org/10.18653/v1/2021.findings-emnlp.41 Kingma, Diederik P. and… ↗
  15. Lample, Guillaume and Alexis Conneau. 2019. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems (NIPS), pages 7059–7069. Lauscher, Anne, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfe…
  16. Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  17. Naseem, Tahira, Regina Barzilay, and Amir Globerson. 2012. Selective sharing for multilingual dependency parsing. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 629–637.
  18. Navigli, Roberto and Simone Paolo Ponzetto. 2012. Joining forces pays off: Multilingual joint word sense disambiguation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1399–1410. find
  19. Nivre, Joakim, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on…
  20. Nozza, Debora, Federico Bianchi, and Dirk Hovy. 2020. What the [MASK]? Making sense of language-specific BERT models. arXiv preprint arXiv:2003.02912. O’Horan, Helen, Yevgeni Berzak, Ivan Vulić, Roi Reichart, and Anna Korhonen. 2016. Survey on the use of typological information in natural language… — [arXiv:2003.02912](https://arxiv.org/abs/2003.02912) find
  21. Pappas, Nikolaos and Andrei Popescu-Belis. 2017. Multilingual hierarchical attention networks for document classification. In IJCNLP (1), pages 1015–1025.
  22. Pennington, Jeffrey, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. https://doi.org/10.3115/v1/D14-1162 ↗
  23. Peters, Matthew, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018a. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499–1509. https://doi.org/10.18653/v1/D18-1179 ↗
  24. Peters, Matthew E., Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018b. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237. https://doi.org/10 .18653/v1/N18-1202 Pires, Telmo, Eva Schlinger, and Dan Garrette. 2019.…
  25. Ravishankar, Vinit, Lilja Øvrelid, and Erik Velldal. 2019. Probing multilingual sentence representations with X-P ROBE . In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 156–168. https://doi.org/10.18653/v1/W19-4318 Şahin, Gözde Gül, Clara Vania, Ilia Kuz… ↗
  26. Tenney, Ian, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601. https://doi.org/10.18653/v1/P19-1452 Tenney, Ian, Patrick Xia, Berlin Chen, et al. 2019. What… ↗
  27. Tiedemann, Jörg. 2015. Cross-lingual dependency parsing with universal dependencies and predicted POS labels. In Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015), pages 340–349. Tiedemann, Jörg, Željko Agić, and Joakim Nivre. 2014. Treebank translation for…
  28. Wu, Shijie and Mark Dredze. 2019. Beto, Bentz, Becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844.…
  29. Zeman, Daniel and Philip Resnik. 2008. Cross-language parser adaptation between related languages. In Proceedings of the IJCNLP-08 Workshop on NLP for Less Privileged Languages, pages 35–42.
  30. Zhang, Yuan, David Gaddy, Regina Barzilay, and Tommi Jaakkola. 2016. Ten pairs to tag–multilingual POS tagging via coarse mapping between embeddings. In Proceedings of NAACL-HLT, pages 1307–1317. https://doi.org/10 .18653/v1/N16-1156 Zhang, Yuan, Roi Reichart, Regina Barzilay, and Amir Globerson. 20… find
  31. Zhou, Guangyou, Tingting He, Jun Zhao, and Wensheng Wu. 2015. A subspace learning framework for cross-lingual sentiment classification with partial parallel data. In Twenty-Fourth International Joint Conference on Artificial Intelligence, pages 1426–1432. find
  32. Zuccon, Guido, Bevan Koopman, Peter Bruza, and Leif Azzopardi. 2015. Integrating and evaluating neural word embeddings in information retrieval. In Proceedings of the 20th Australasian Document Computing Symposium, pages 1–8. https://doi.org/10.1145/2838931.2838936 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/choenni-2022-investigating-language-relationships-multilingual-sentence/