Elamur Research⌕
☰
home / papers / 2020 / Multi-SimLex: A Large-Scale Evaluation of Multilingual and Crosslingual Lexical Semantic Similarity

Multi-SimLex: A Large-Scale Evaluation of Multilingual and Crosslingual Lexical Semantic Similarity

Paper · 2020 · Ivan Vulić, Simon Baker, Edoardo Maria Ponti et al.

Summary

Datasets that score how alike two words are in meaning exist for only a handful of mostly well-resourced languages. Here "alike" means sharing features (lion–cat), not just being associated (lion–zoo). The existing sets also differ in size, rating scales and construction methods, so they cannot be compared across languages. To fix this, the authors first built a new English set of 1,888 concept pairs. It is balanced for part of speech, frequency, concreteness and similarity range. They then translated and re-rated it in 11 further languages using a single language-agnostic protocol. Because the pairs are aligned across languages, the monolingual sets can be combined into 66 crosslingual benchmarks. Human agreement is high in every language. The authors evaluate static embeddings, pretrained encoders, knowledge-injection methods and crosslingual embeddings, and every system falls well short of human performance. The resource matters because it gives lexical semantics a shared, extensible standard in the spirit of UD. It includes low-resource languages such as Welsh, Kiswahili and Yue Chinese, and its public guidelines invite the community to add more languages.

Key points

  • Each of the 12 languages has 1,888 aligned pairs covering nouns, verbs, adjectives and adverbs. This is the first such resource to include adverbs. Pairing the languages yields 66 crosslingual sets.
  • Mean inter-annotator agreement is Spearman's ρ = 0.740 on average, ranging from 0.667 (Russian) to 0.812 (French).
  • Massively multilingual encoders (M-BERT, XLM-100) perform poorly on the benchmark. Single-language BERT-style encoders, even when given words without context, sometimes rival fastText or word2vec, which is consistent with the "curse of multilinguality."
  • Models match human judgments best on nouns. The amount of raw training text predicts embedding quality better than language family, geographic area or typological features. Unsupervised post-processing always helps, but the best combination depends on the language.
  • Data, guidelines and baselines are available at multisimlex.com, and Arabic has been added since publication.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🎓Technion – Israel Institute of TechnologyHaifa · Israel49 works · 5,306 citations🔬École Normale Supérieure - PSLParis · France23 works · 4,357 citations🎓Université Paris Sciences et LettresParis · France21 works · 3,290 citations🎓Université Sorbonne NouvelleParis · France9 works · 569 citations🔬Langues, Textes, Traitements Informatiques, CognitionMontrouge · France7 works · 397 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksLexical semantics & word embeddings · 63 works

Cite

Cite as Ivan Vulić, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, Roi Reichart and Anna Korhonen. 2020. Multi-SimLex: A Large-Scale Evaluation of Multilingual and Crosslingual Lexical Semantic Similarity. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00391

@article{vulic2020multi,
  title = {{Multi-SimLex: A Large-Scale Evaluation of Multilingual and Crosslingual Lexical Semantic Similarity}},
  author = {Ivan Vulić, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, Roi Reichart and Anna Korhonen},
  journal = {Computational Linguistics},
  year = {2020},
  doi = {10.1162/coli\_a\_00391},
  url = {https://doi.org/10.1162/coli_a_00391}
}

References

77 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adams, Oliver, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017. Cross-lingual word embeddings for low-resource language modeling. In Proceedings of EACL, pages 937–947. Valencia. DOI: https://doi.org/10 .18653/v1/E17-1088 Agirre, Eneko, Enrique Alfonseca, Keith Hall, Jana Kravalova… find
  2. Aldarmaki, Hanan and Mona Diab. 2019. Context-aware cross-lingual mapping. In Proceedings of NAACL-HLT, pages 3906–3911. Minneapolis, MN. DOI: https://doi.org/10.18653/v1/N19-1391 ↗
  3. Alvarez-Melis, David and Tommi Jaakkola. 2018. Gromov-Wasserstein alignment of word embedding spaces. In Proceedings of EMNLP, pages 1881–1890. Brussels. DOI: https://doi.org/10.18653/v1/D18-1214 ↗
  4. Ammar, Waleed, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah Smith. 2016. Many languages, one parser. Transactions of the ACL, 4:431–444. DOI: https://doi.org/10.1162/tacl a 00109 Artetxe, Mikel, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bi… ↗
  5. New Orleans, LA. Artetxe, Mikel, Gorka Labaka, and Eneko Agirre. 2018b. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of ACL, pages 789–798. Melbourne. DOI: https://doi.org/10.18653/v1/P18-1073 Artetxe, Mikel, Gorka Labaka, and Eneko… ↗
  6. Artetxe, Mikel, Sebastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of monolingual representations. CoRR, abs/1910.11856. DOI: https://doi.org/10.18653/v1/2020.acl-main.421 Baker, Collin F., Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Pr… ↗
  7. Baker, Simon, Roi Reichart, and Anna Korhonen. 2014. An unsupervised model for instance level subcategorization acquisition. In Proceedings of EMNLP, pages 278–289. Doha. DOI: https://doi.org/10.3115/v1/D14-1034, PMID: 25178667 Bapna, Ankur and Orhan Firat. 2019. Simple, scalable adaptation for neu… ↗
  8. Miyazaki. Bjerva, Johannes and Isabelle Augenstein. 2018. From phonology to syntax: Unsupervised linguistic typology at different levels with language embeddings. In Proceedings of NAACL-HLT, pages 907–916. New Orleans, LA. DOI: https://doi.org/10.18653/v1/N18-1083 Bjerva, Johannes, Robert Östling… ↗
  9. Bruni, Elia, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1–47. DOI: https://doi.org/10.1613/jair.4135 Budanitsky, Alexander and Graeme Hirst. 2006. Evaluating WordNet-based measures of lexical semantic relatedness. Com… ↗
  10. Chen, Danqi and Christopher D. Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of EMNLP, pages 740–750. Doha. DOI: https://doi.org/10 .3115/v1/D14-1082 Chen, Xilun and Claire Cardie. 2018. Unsupervised multilingual word embeddings. In Proceedings of EMNLP,…
  11. Chen, Xinying and Kim Gerdes. 2017. Classifying languages by dependency structure. Typologies of delexicalized universal dependency treebanks. In Proceedings of the 4th International Conference on Dependency Linguistics (DepLing), pages 54–63. find
  12. Pisa. Cimiano, Philipp, Andreas Hotho, and Steffen Staab. 2005. Learning concept hierarchies from text corpora using formal concept analysis. Journal of Artificial Intelligence Research, 24:305–339. DOI: https://doi.org/10.1613/jair.1648 — [doi:10.1613/jair.1648](https://doi.org/10.1613/jair.1648) ↗
  13. Clark, Jonathan H., Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the ACL, 8:454–470. DOI: https://doi.org/10.1162/tacl a… ↗
  14. Vancouver. Conneau, Alexis, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018b. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP, pages 2475–2485. Brussels. DOI: https://doi.org/10.18653/v1/D18-1269 ↗
  15. Coseriu, Eugenio. 1967. Lexikalische solidaritäten. Poetica. 1:293–303. find
  16. Lexical Semantics, Cambridge University Press. De Deyne, Simon and Gert Storms. 2008. Word associations: Network and semantic properties. Behavior Research Methods, 40(1):213–231. DOI: https://doi.org/10.3758/BRM.40.1.213, PMID: 18411545 Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Touta… — [doi:10.3758/brm.40.1.213](https://doi.org/10.3758/brm.40.1.213) ↗
  17. Doitch, Amichay, Ram Yazdi, Tamir Hazan, and Roi Reichart. 2019. Perturbation based learning for structured NLP tasks with application to dependency parsing. Transactions of the ACL, 7:643–659. DOI: https://doi.org/10.1162/tacl a 00291 Doval, Yerai, Jose Camacho-Collados, Luis Espinosa-Anke, and Ste… ↗
  18. Dryer, Matthew S. 2013. Order of subject, object and verb. In Dryer, Matthew S. and Martin Haspelmath, editors, The World Atlas of Language Structures Online, Max Planck Institute for Evolutionary Anthropology, Leipzig, pages 330–333. Ercan, Gökhan and Olcay Taner Yıldız. 2018. AnlamVer: Semantic m… find
  19. Santa Fe, NM. Ethayarajh, Kawin. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of EMNLP, pages 55–65. Hong Kong. DOI: https://doi.org/10.18653/v1/D19-1006 Evans, Nicholas. 2011. Semantic Typology. In The Oxfo… ↗
  20. Firth, John R. 1957. A synopsis of linguistic theory, 1930-1955. Studies in Linguistic Analysis. Blackwell, pages 1–32. François, Alexandre. 2008. Semantic maps and the typology of colexification. In Martine Vanhove, editor, From Polysemy to Semantic Change: Towards a Typology of Lexical Semantic A…
  21. Grave, Edouard, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of LREC, pages 3483–3487.
  22. Miyazaki. Gruber, Jeffrey. 1976. Lexical Structures in Syntax and Semantics, volume 25, North-Holland. find
  23. Harris, Zellig S. 1951. Methods in Structural Linguistics, University of Chicago Press. find
  24. Hill, Felix, KyungHyun Cho, Anna Korhonen, and Yoshua Bengio. 2016. Learning to understand phrases by embedding the dictionary. Transactions of the ACL, 4:17–30. DOI: https://doi.org/10.1162/tacl a 00080 ↗
  25. Hill, Felix, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. DOI: https://doi.org/10 .1162/tacl a 00080 Hoshen, Yedid and Lior Wolf. 2018. Non-adversarial unsupervised word translation. In P…
  26. Huang, Junjie, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, and Maosong Sun. 2019. COS960: A Chinese word similarity dataset of 960 word pairs. CoRR, abs/1906.00247. — [arXiv:1906.00247](https://arxiv.org/abs/1906.00247) find
  27. Joshi, Pratik, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of ACL, pages 6282–6293. Online. DOI: https://doi.org/10 .18653/v1/2020.acl-main.560 Joulin, Armand, Piotr Bojanowski, To…
  28. Kamholz, David, Jonathan Pool, and Susan M. Colowick. 2014. PanLex: Building a resource for panlingual lexical translation. In Proceedings of LREC, pages 3145–3150. find
  29. Reykjavik. Kay, Paul and Luisa Maffi. 2013. Green and blue. In Dryer, Matthew S. and Martin Haspelmath, editors, The World Atlas of Language Structures Online, Max Planck Institute for Evolutionary Anthropology, Leipzig, pages 540–541. find
  30. Kiela, Douwe and Stephen Clark. 2014. A systematic study of semantic vector space model parameters. In Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC), pages 21–30. Gothenburg. DOI: https://doi.org/10.3115/v1/W14-1503 Kiela, Douwe, Felix Hill, and… ↗
  31. Kim, Joo-Kyung, Marie-Catherine de Marneffe, and Eric Fosler-Lussier. 2016. Adjusting word embeddings with semantic intensity orders. In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 62–69. Berlin. DOI: https://doi.org/10.18653/v1/W16-1607 ↗
  32. Kim, Joo-Kyung, Gokhan Tur, Asli Celikyilmaz, Bin Cao, and Ye-Yi Wang. 2016. Intent detection using semantically enriched word embeddings. In SLT, pages 414–419. San Diego, CA. DOI: https://doi.org/10.1109/SLT.2016 .7846297 Kipper, Karin, Anna Korhonen, Neville Ryant, and Martha Palmer. 2008. A larg… ↗
  33. Lisbon. Kipper Schuler, Karin. 2005. VerbNet: A broad-coverage, comprehensive verb lexicon. Ph.D. thesis, University of Pennsylvania. find
  34. Kondratyuk, Dan and Milan Straka. 2019. 75 languages, 1 model: Parsing Universal Dependencies universally. In Proceedings of EMNLP-IJCNLP, pages 2779–2795. Hong Kong. DOI: https://doi.org/10 .18653/v1/D19-1279 Lan, Zhenzhong, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori…
  35. Lazaridou, Angeliki, Georgiana Dinu, and Marco Baroni. 2015. Hubness and pollution: Delving into cross-space mapping for zero-shot learning. In Proceedings of ACL, pages 270–280. Beijing. DOI: https://doi.org/10.3115/v1/P15-1027 Le, Hang, Loı̈c Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, B… ↗
  36. Leviant, Ira and Roi Reichart. 2015. Separated by an un-common language: Towards judgment language informed vector space modeling. CoRR, abs/1508.00106. — [arXiv:1508.00106](https://arxiv.org/abs/1508.00106) find
  37. Levin, Beth. 1993. English Verb Classes and Alternation, A Preliminary Investigation. The University of Chicago Press. find
  38. Levy, Omer and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of ACL, pages 302–308. Baltimore, MD. DOI: https://doi.org/10.3115/v1/P14-2050, PMID: 25270273 Lewis, Patrick S. H., Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. MLQA: Evaluating cross-ling… ↗
  39. Lin, Yu Hsiang, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In Proceedings of ACL, pages 3125–3135. Flor…
  40. Lucas, Margery. 2000. Semantic priming without association: A meta-analytic review. Psychonomic Bulletin & Review, 7(4):618–630. DOI: https://doi.org/10.3758/BF03212999, PMID: 11206202 Luong, Thang, Richard Socher, and Christopher Manning. 2013. Better word representations with recursive neural net… — [doi:10.3758/bf03212999](https://doi.org/10.3758/bf03212999) ↗
  41. Sofia. Lyons, John. 1977. Semantics, volume 2. Cambridge University Press. DOI: https://doi.org/10.1017/CBO9781139165693 Majid, Asifa, Melissa Bowerman, Miriam van Staden, and James S. Boster. 2007. The semantic categories of cutting and breaking events: A cross-linguistic perspective. Cognitive Li… ↗
  42. McKeown, Kathleen R., Regina Barzilay, David Evans, Vasileios Hatzivassiloglou, Judith L. Klavans, Ani Nenkova, Carl Sable, Barry Schiffman, and Sergey Sigelman. 2002. Tracking and summarizing news on a daily basis with Columbia’s newsblaster. In Proceedings of HLT, pages 280-285. San Diego, CA. DOI…
  43. Miyazaki. Mikolov, Tomas, Quoc V. Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv preprint, CoRR, abs/1309.4168. find
  44. Mikolov, Tomas, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NeurIPS, pages 3111–3119.
  45. South Lake Tahoe, NV. Miller, George A. 1995. WordNet: A lexical database for English. Communications of the ACM. pages 39–41. DOI: https://doi.org/10.1145/219717.219748 Mimno, David and Laure Thompson. 2017. The strange geometry of skip-gram with negative sampling. In Proceedings of EMNLP, pages 2… ↗
  46. Mykowiecka, Agnieszka, Małgorzata Marciniak, and Piotr Rychlik. 2018. SimLex-999 for Polish. In Proceedings of LREC, pages 2398–2402. find
  47. Miyazaki. Nelson, Douglas L., Cathy L. McEvoy, and Thomas A. Schreiber. 2004. The University of South Florida free association, rhyme, and word fragment norms. Behavior Research Methods, 36(3):402–407. DOI: https://doi.org/10.3758/BF03195588, PMID: 15641430 Netisopakul, Ponrudee, Gerhard Wohlgenannt… ↗
  48. Pearson, Karl. 1901. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559–572. DOI: https://doi.org/10 .1080/14786440109462720 Peters, Matthew, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher…
  49. Pilehvar, Mohammad Taher, Dimitri Kartsaklis, Victor Prokhorov, and Nigel Collier. 2018. Card-660: Cambridge rare word dataset - a reliable benchmark for infrequent word representation models. In Proceedings of EMNLP, pages 1391–1401. Brussels. DOI: https://doi.org/10.18653/v1/D18-1169 Pires, Telmo… ↗
  50. Ponti, Edoardo Maria, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning. arXiv preprint arXiv:2005.00333. Online. Ponti, Edoardo Maria, Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poib…
  51. Quinn, Naomi and Dorothy Holland. 1987. Culture and cognition. Cultural Models in Language and Thought, pages 3–40. DOI: https://doi.org/10.1017/CBO9780511607660.002, PMID: 3552892 Radovanović, Miloš, Alexandros Nanopoulos, and Mirjana Ivanović. 2010. Hubs in space: Popular nearest neighbors in… ↗
  52. Rasooli, Mohammad Sadegh and Michael Collins. 2017. Cross-lingual syntactic transfer with limited resources. Transactions of the ACL, 5:279–293. DOI: https://doi.org/10.1162/tacl a 00061 ↗
  53. Ren, Liliang, Kaige Xie, Lu Chen, and Kai Yu. 2018. Towards universal dialogue state tracking. In Proceedings of EMNLP, pages 2780–2786. Brussels. DOI: https://doi.org/10.18653/v1/D18 -1299 Roemmele, Melissa, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evalu… ↗
  54. Palo Alto, CA. Rotman, Guy and Roi Reichart. 2019. Deep contextualized self-training for low resource dependency parsing. Transactions of the ACL, 7:695–713. DOI: https://doi.org/10.1162/tacl a 00294 Ruder, Sebastian, Anders Søgaard, and Ivan Vulić. 2019. Unsupervised cross-lingual representation… ↗
  55. Rzymski, Christoph, Tiago Tresoldi, Simon J. Greenhill, Mei-Shin Wu, Nathanael E. Schweikhard, Maria Koptjevskaja-Tamm, Volker Gast, Timotheus A. Bodt, Abbie Hantgan, Gereon A. Kaiping, et al. 2020. The database of cross-linguistic colexifications, reproducible analysis of cross-linguistic polysemie… — [doi:10.1038/s41597-019-0341-x](https://doi.org/10.1038/s41597-019-0341-x) find
  56. Sakaizawa, Yuya and Mamoru Komachi. 2018. Construction of a Japanese word similarity dataset. In Proceedings of LREC, pages 948–951. Miyazaki. Schlechtweg, Dominik, Anna Hätty, Marco Del Tredici, and Sabine Schulte im Walde. 2019. A wind of change: Detecting and evaluating lexical semantic change a… find
  57. Toulon. Snyder, Benjamin and Regina Barzilay. 2010. Climbing the tower of Babel: Unsupervised multilingual learning. In Proceedings of ICML, pages 29–36. Haifa. Søgaard, Anders, Sebastian Ruder, and Ivan Vulić. 2018. On the limitations of unsupervised bilingual dictionary induction. In Proceedings… — [doi:10.18653/v1/p18](https://doi.org/10.18653/v1/p18)
  58. Seattle, WA. Tang, Shuai, Mahta Mousavi, and Virginia R. de Sa. 2019. An empirical study on post-processing methods for word embeddings. CoRR, abs/1905.10971. — [arXiv:1905.10971](https://arxiv.org/abs/1905.10971) find
  59. Trier, Jost. 1931. Der Deutsche Wortschatz im Sinnbezirk des Verstandes: Die Geschichte eines sprachlichen Feldes. 1. Von den Anfängen bis zum Beginn des 13. Jahrhunderts. Ph.D. thesis, University of Bonn. find
  60. Tsvetkov, Yulia, Sunayana Sitaram, Manaal Faruqui, Guillaume Lample, Patrick Littell, David Mortensen, Alan W. Black, Lori Levin, and Chris Dyer. 2016. Polyglot neural language models: A case study in cross-lingual phonetic representation learning. In Proceedings of NAACL-HLT, pages 1357–1366. San D… find
  61. Turney, Peter D. 2012. Domain and function: A dual-space model of semantic relations and compositions. Journal of Artificial Intelligence Research, 44:533–585. DOI: https://doi.org/10.1613/jair.3640 — [doi:10.1613/jair.3640](https://doi.org/10.1613/jair.3640) ↗
  62. Turney, Peter D. and Patrick Pantel. 2010. From frequency to meaning: vector space models of semantics. Journal of Artifical Intelligence Research, 37(1):141–188. DOI: https://doi.org/10.1613/jair.2934 ↗
  63. Upadhyay, Shyam, Manaal Faruqui, Chris Dyer, and Dan Roth. 2016. Cross-lingual models of word embeddings: An empirical comparison. In Proceedings of ACL, pages 1661–1670. Berlin. DOI: https://doi.org/10.18653/v1/P16-1157 van den Berg, Robert A., Huub C. J. Hoefsloot, Johan A. Westerhuis, Age K. Smil… ↗
  64. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS, pages 6000–6010.
  65. Long Beach, CA. Vejdemo, Susanne. 2018. Lexical change often begins and ends in semantic peripheries: Evidence from color linguistics. Pragmatics & Cognition, 25(1):50–85. DOI: https://doi.org/10.1075/pc.00005.vej — [doi:10.1075/pc.00005.vej](https://doi.org/10.1075/pc.00005.vej) ↗
  66. Venekoski, Viljami and Jouko Vankka. 2017. Finnish resources for evaluating language model semantics. In Proceedings of NODALIDA, pages 231–236. find
  67. Gothenburg. Virtanen, Antti, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. CoRR, abs/1912.07076. Vulić, Ivan, Daniela Gerz, Douwe Kiela, Felix Hill, and Anna Korhonen. 2017a. HyperLex: A…
  68. Wieting, John, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. From paraphrase database to compositional paraphrase model and back. Transactions of the ACL, 3:345–358. DOI: https://doi.org/10 .1162/tacl a 00143 Williams, Adina, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge…
  69. Wolf, Thomas, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
  70. Wu, Shijie, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Emerging cross-lingual structure in pretrained language models. CoRR, abs/1911.01464.
  71. Wu, Shijie and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of EMNLP, pages 833–844. Hong Kong. DOI: https://doi.org/10.18653/v1/D19-1077 ↗
  72. Wu, Shijie and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130. Online. DOI: https://doi.org/10.18653/v1/2020.repl4nlp-1.16 Wu, Zhibiao and Martha Palmer. 1994. Verb semantics and lexical s… ↗
  73. CoNLL 2018 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–21.
  74. Brussels. Zhang, Mozhi, Keyulu Xu, Ken-ichi Kawarabayashi, Stefanie Jegelka, and Jordan Boyd-Graber. 2019. Are girls neko or shōjo? Cross-lingual alignment of non-isomorphic embeddings with iterative normalization. In Proceedings of ACL, pages 3180–3189. Minneapolis, MN. DOI: https://doi.org/10.186… find
  75. Zhang, Yuan, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of NAACL-HLT, pages 1298–1308. Hong Kong.
  76. Zhu, Yi, Benjamin Heinzerling, Ivan Vulić, Michael Strube, Roi Reichart, and Anna Korhonen. 2019. On the importance of subword information for morphological tasks in truly low-resource languages. In Proceedings of CoNLL, pages 216–226. Hong Kong. DOI: https://doi.org/10.18653/v1/K19-1021 ↗
  77. Zhu, Yi, Ivan Vulić, and Anna Korhonen. 2019. A systematic study of leveraging subword information for learning word representations. In Proceedings of NAACL-HLT, pages 912–932. Brussels. DOI: https://doi.org/10.18653/v1/N19-1097 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vulic-2020-multi-simlex-large-scale-evaluation/