Elamur Research⌕
☰
home / papers / 2019 / What Do Language Representations Really Represent?

What Do Language Representations Really Represent?

Paper · 2019 · Johannes Bjerva, Robert Östling, Maria Han Veiga et al.

Summary

Multilingual neural models can learn a vector for each language. Earlier work, notably Rabinovich, Ordan, and Wintner (2017), judged such vectors mainly by how well they recover family trees. Bjerva et al. ask what kind of similarity these vectors actually capture. They use English translations from the Europarl corpus, each labelled with its original EU source language. On this data they train a 2-layer LSTM language model that receives a language vector at every time step. Inputs come at several abstraction levels: raw text, function words, POS, phrase structure and dependency relations. The learned language distances are compared with three references: genetic distance, geographical distance between speaker communities, and a new structural distance. Spearman correlations show the vectors align best with structural similarity, not genealogy. A causal analysis supports this and suggests genetic relatedness acts as a confound. The study warns against validating language embeddings with one convenient benchmark and shows how NLP and typology can inform each other.

Key points

  • Data: the same Europarl-derived English translations used by Rabinovich et al. (2017), with the faint source-language signal acting as the training cue.
  • Five representation types are compared: Raw, Func, POS, Phrase and Deprel. Genetic distances are taken from Serva and Petroni (2008) phylogenetic trees.
  • The new structural measure profiles each language by dependency relations together with head and dependent POS. This yields 8,607-dimensional normalized vectors, richer than the approach of Chen and Gerdes (2017).
  • Ward clustering on the structural vectors departs from genealogy in several ways. Romanian is set apart from the other Romance languages, English groups with North Germanic, and Latvian/Lithuanian cluster with Estonian/Finnish.
  • In the causal network built with IC* (p < 0.001), the only link between similarity measures and representations connects structural similarity to the raw model. No link reaches genetic similarity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CopenhagenCopenhagen · Denmark115 works · 15,292 citations🎓University of ZurichZurich · Switzerland78 works · 12,260 citations🎓University of HelsinkiHelsinki · Finland50 works · 5,318 citations🎓Stockholm UniversityStockholm · Sweden21 works · 1,832 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

Cite

Cite as Johannes Bjerva, Robert Östling, Maria Han Veiga, Jörg Tiedemann and Isabelle Augenstein. 2019. What Do Language Representations Really Represent?. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00351

@article{bjerva2019what,
  title = {{What Do Language Representations Really Represent?}},
  author = {Johannes Bjerva, Robert Östling, Maria Han Veiga, Jörg Tiedemann and Isabelle Augenstein},
  journal = {Computational Linguistics},
  year = {2019},
  doi = {10.1162/coli\_a\_00351},
  url = {https://doi.org/10.1162/coli_a_00351}
}

References

17 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Malaviya, Chaitanya, Graham Neubig, and Patrick Littell. 2017. Learning language representations for typology prediction. In EMNLP, pages 2519–2525.
  2. Nivre, Joakim, et al. 2017. Universal Dependencies 2.1. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Charles University. Östling, Robert. 2015. Word order typology through multilingual word alignment. In ACL-IJCNLP, pages 205–211. Östling, Robert and Jo… find
  3. Pearl, Judea. 2009. Causality, Cambridge University Press. Ponti, Edoardo Maria, Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2018. Modeling language variation and universals: A survey on typological linguistics for natural language… — [arXiv:1807.00914](https://arxiv.org/abs/1807.00914) find
  4. Rabinovich, Ella, Noam Ordan, and Shuly Wintner. 2017. Found in translation: Reconstructing phylogenetic language trees from translations. In ACL. Roberts, Seán and James Winters. 2013. Linguistic diversity and traffic accidents: Lessons from statistical studies of cultural traits. PloS One, 8(8):e… find
  5. Serva, Maurizio and Filippo Petroni. 2008. Indo-European languages tree by Levenshtein distance. EPL, 81(6):68005. Straka, Milan, Jan Hajic, and Jana Straková. 2016. UD-Pipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In… find
  6. Velupillai, Viveka. 2012. An Introduction to Linguistic Typology, John Benjamins Publishing. find
  7. Verma, Thomas and Judea Pearl. 1992. An algorithm for deciding if a set of observed independencies has a causal explanation. In Proceedings of the 8th Conference on Uncertainty and Artificial Intelligence, pages 323–330. find
  8. Vinyals, Oriol, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In NIPS. find
  9. Ammar, Waleed, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah Smith. 2016. Many languages, one parser. TACL, 4:431–444.
  10. Bjerva, Johannes and Isabelle Augenstein. 2018. From phonology to syntax: Unsupervised linguistic typology at different levels with language embeddings. In NAACL-HLT. find
  11. Chen, Xinying and Kim Gerdes. 2017. Classifying languages by dependency structure. Typologies of delexicalized universal dependency treebanks. In DepLing, pages 54–63. find
  12. Cotterell, Ryan and Jason Eisner. 2017. Probabilistic typology: Deep generative models of vowel inventories. In ACL. find
  13. Dunn, Michael, Simon J. Greenhill, Stephen C. Levinson, and Russell D. Gray. 2011. Evolved structure of language shows lineage-specific trends in word-order universals. Nature, 473(7345):79–82. find
  14. Gellerstam, Martin. 1986. Translationese in Swedish novels translated from English. Translation Studies in Scandinavia, 1:88–95. find
  15. Gerz, Daniela, Ivan Vulic, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. On the relation between linguistic typology and (limitations of) multilingual language modeling. In EMNLP. Hammarström, Harald, Robert Forkel, and Martin Haspelmath. 2017. Glottolog 3.0. Jena: Max Planck Institut…
  16. Haspelmath, Martin. 2001. Language Typology and Language Universals: An International Handbook, volume 20, Walter de Gruyter. Johnson, Melvin, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeff… find
  17. Koehn, Philipp. 2005. Europarl: A parallel corpus for statistical machine translation. In MT Summit X. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bjerva-2019-what-do-language-representations-really/