Elamur Research⌕
☰
home / papers / 2020 / LINSPECTOR: Multilingual Probing Tasks for Word Representations

LINSPECTOR: Multilingual Probing Tasks for Word Representations

Paper · 2020 · Gözde Gül Şahin, Clara Vania, Ilia Kuznetsov et al.

Summary

Most probing research tests English representations, often at the sentence level. English is typologically unusual, though, because many languages encode grammatical information inside words through morphology. To fill this gap, the authors present LINSPECTOR, a multilingual suite of word-level probes covering 24 languages. They design 15 context-independent, dictionary-derived type-level tasks. These range from surface traits (word length) to morphosyntax (case, gender, number, polarity, possession) and pseudoword detection, and are built from UniMorph, Wikipedia and Wuggy. A comparable set of context-dependent, treebank-based token-level tests complements them. Several multilingual embedding models are probed with simple classifiers, and the resulting scores are correlated with performance on POS tagging, DEP, SRL, NER and NLI. An end-to-end SRL model is also diagnosed as a case study. Many probes show significant positive correlation with downstream results, most strongly for morphologically rich languages. This gives researchers with limited compute a cheap way to compare embeddings or inspect black-box encoders before running expensive downstream experiments.

Key points

  • The suite has 15 type-level probing tasks across 24 languages, with language-specific subsets: Russian and Spanish are probed for gender, while Turkish is probed for polarity and possession.
  • The type-level and token-level variants mostly agree, but each has distinct weaknesses:
    • Token-level tests can absorb domain and majority-class biases.
    • Type-level tests suffer when lexical diversity is low or ambiguity is high.
  • The probed embeddings are MUSE, word2vec, GloVe-BPE, fastText and D-ELMo. D-ELMo generally ranks first and fastText second; on Hungarian Case, for example, D-ELMo scores 98.2 against a 30.06 majority baseline.
  • Results also depend on factors beyond architecture, including OOV rates, domain similarity, dataset size and ambiguity, typology and paradigm size. Finnish, with 57,642 UniMorph paradigms versus 13,989 for Hungarian, gets lower scores.
  • The probing data sets and evaluation suite are released at github.com/UKPLab/linspector.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States201 works · 100,180 citations🎓Technische Universität DarmstadtDarmstadt · Germany109 works · 35,920 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksInterpretability · 130 works

Cite

Cite as Gözde Gül Şahin, Clara Vania, Ilia Kuznetsov and Iryna Gurevych. 2020. LINSPECTOR: Multilingual Probing Tasks for Word Representations. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00376

@article{sahin2020linspector,
  title = {{LINSPECTOR: Multilingual Probing Tasks for Word Representations}},
  author = {Gözde Gül Şahin, Clara Vania, Ilia Kuznetsov and Iryna Gurevych},
  journal = {Computational Linguistics},
  year = {2020},
  doi = {10.1162/coli\_a\_00376},
  url = {https://doi.org/10.1162/coli_a_00376}
}

References

72 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adi, Yossi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representations, pages 1–13.
  2. Aggarwal, Rakesh and Priya Ranganathan. 2016. Common pitfalls in statistical analysis: The use of correlation techniques. Perspectives in Clinical Research, 7(4):187–190. find
  3. Ataman, Duygu and Marcello Federico. 2018. Compositional representation of morphologically-rich input for neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 305–311. find
  4. Belinkov, Yonatan, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861–872.
  5. Belinkov, Yonatan and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72.
  6. Bengtson, Eric and Dan Roth. 2008. Understanding the value of features for coreference resolution. In 2008 Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, Proceedings of the Conference, pages 294–303, Honolulu, HI.
  7. Benikova, Darina, Chris Biemann, Max Kisselew, and Sebastian Pado. 2014. find
  8. Germeval 2014 named entity recognition shared task: Companion paper. In Proceedings of the KONVENS GermEval Shared Task on Named Entity Recognition, pages 104–112, Hildesheim. find
  9. Bisazza, Arianna and Clara Tump. 2018. The lazy encoder: A fine-grained analysis of the role of morphology in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2871–2876. find
  10. Blake, Barry J. 2001. Case. Cambridge University Press. 119–120. find
  11. Bojanowski, Piotr, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146.
  12. Bowman, Samuel R., Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, pages 632–642, Lisbon.
  13. Bruni, Elia, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In the 50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8-14, 2012, pages 136–145, Jeju Island.
  14. Camacho-Collados, Jose, Mohammad Taher Pilehvar, Nigel Collier, and Roberto Navigli. 2017. Semeval-2017 task 2: Multilingual and cross-lingual semantic word similarity. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 15–26, Vancouver. find
  15. Che, Wanxiang, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 55–64, Brussels…
  16. Chen, Qian, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, pages 1657–1668, Vancouver. find
  17. Comrie, Bernard and Maria Polinsky. 1998. The great Daghestanian case hoax. In A. Siewierska and J. Jung Song, eds. Case, Typology and Grammar, pages 95–114. Conneau, Alexis, Germán Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018a. What you can cram into a single \\$&!\#* vector:… find
  18. Conneau, Alexis, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018b. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels… find
  19. Corbett, Greville G. 2013. Number of genders. In Dryer, Matthew S. and Martin Haspelmath, editors, The World Atlas of Language Structures Online, Max Planck Institute for Evolutionary Anthropology, Leipzig. Available at https://wals.info/chapter/30. ↗
  20. Cotterell, Ryan, Christo Kirov, Mans Hulden, and Jason Eisner. 2019. On the complexity and typology of inflectional morphological systems. Transactions of the Association for Computational Linguistics, 7:327–342.
  21. Cotterell, Ryan, Sebastian J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, Volume 2 (Short P…
  22. Dozat, Timothy and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Conference Track Proceedings, pages 1–8, Toulon. Eichler, Max, Gözde Gül Sahin, and Iryna Gurevych. 2019. LINSPECTOR WEB:…
  23. Erten, Begum, Cem Bozsahin, and Deniz Zeyrek. 2014. Turkish resources for visual word recognition. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, pages 2106–2110, Reykjavic. Eryigit, Gülsen, Joakim Nivre, and Kemal Oflazer. 2008. Dependency par… find
  24. Fares, Murhaf, Andrey Kutuzov, Stephan Oepen, and Erik Velldal. 2017. Word vectors, reuse, and replicability: Towards a community repository of large-text resources. In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 271–276, Gothenburg, Sweden. find
  25. Faruqui, Manaal, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages… find
  26. Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the Tenth International World Wide Web Conference, WWW 10, pages 406–414, Hong Kong. find
  27. Gage, Philip. 1994. A new algorithm for data compression. C Users Journal, 12(2):23–38. find
  28. Gardner, Matt, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1–6. Gerz, Dani…
  29. Ghaddar, Abbas and Phillippe Langlais. 2017. Winer: A Wikipedia annotated corpus for named entity recognition. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 413–422. Hajič, Jan, Massimiliano Ciaramita, Richard Johansson, Da… find
  30. Haverinen, Katri, Jenna Kanerva, Samuel Kohonen, Anna Missila, Stina Ojala, Timo Viljanen, Veronika Laippala, and Filip Ginter. 2015. The Finnish Proposition Bank. Language Resources and Evaluation, 49(4):907–926. find
  31. Heinzerling, Benjamin and Michael Strube. 2018. BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 2989–2993, Miyazaki.
  32. Hill, Felix, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695.
  33. Hohensee, Matt and Emily M. Bender. 2012. Getting more from morphology in multilingual dependency parsing. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 315–326, Montréal. find
  34. Huang, Eric, Richard Socher, Christopher Manning, and Andrew Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 873–882.
  35. Huang, Zhiheng, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. CoRR, arXiv preprint arXiv: 1508.01991.
  36. Iggesen, Oliver A. 2013. Number of cases. In Dryer, Matthew S. and Martin Haspelmath, editors, The World Atlas of Language Structures Online, Max Planck Institute for Evolutionary Anthropology, Leipzig. Isgüder, Gözde Gül and Esref Adali. 2014. Using morphosemantic information in construction of… find
  37. Keuleers, Emmanuel and Marc Brysbaert. 2010. Wuggy: A multilingual pseudoword generator. Behavior Research Methods, 42(3):627–633. find
  38. Kim, Yoon, Yacine Jernite, David A. Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 2741–2749, Phoenix, AZ. Kirov, Christo, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterin…
  39. Levy, Omer and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, Volume 2: Short Papers, pages 302–308, Baltimore, MD. find
  40. Ling, Wang, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Pr…
  41. Linzen, Tal. 2016. Issues in evaluating semantic spaces using word analogies. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 13–18. find
  42. Luong, Thang, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, CoNLL 2013, pages 104–113, Sophia.
  43. McCarthy, Arya D., Miikka Silfverberg, Ryan Cotterell, Mans Hulden, and David Yarowsky. 2018. Marrying universal dependencies and universal morphology. In Proceedings of the Second Workshop on Universal Dependencies, UDW@EMNLP 2018, pages 91–101, Brussels.
  44. McDonald, Ryan, Koby Crammer, and Fernando Pereira. 2005. Online large-margin training of dependency parsers. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 91–98, Ann Arbor, MI.
  45. Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Workshop Track Proceedings, pages 1–12.
  46. Scottsdale, AZ. Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13, pages 3111–3…
  47. Miller, George A. and Walter G. Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1–28. find
  48. Nangia, Nikita, Adina Williams, Angeliki Lazaridou, and Samuel Bowman. 2017. find
  49. The RepEval 2017 shared task: Multi-genre natural language inference with sentence representations. In Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP, pages 1–10.
  50. Nayak, Neha, Gabor Angeli, and Christopher D. Manning. 2016. Evaluating word embeddings using a representative suite of practical tasks. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval at ACL 2016, pages 19–23, Berlin. Nivre, Joakim, Mitchell Abrams, Ž… find
  51. Pennington, Jeffrey, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, pages 1532–1543, Doha.
  52. Peters, Matthew E., Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn…
  53. Qian, Peng, Xipeng Qiu, and Xuanjing Huang. 2016. Investigating language universal and specific properties in word embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1478–1488, Berlin. find
  54. Rogers, Anna, Shashwath Hosur Ananthakrishna, and Anna Rumshisky. 2018. What’s in your embedding, and how it predicts task performance. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2690–2703. find
  55. Rogers, Anna, Aleksandr Drozd, and Bofang Li. 2017. The (too many) problems of analogical reasoning with word vectors. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 135–148. find
  56. Rubenstein, Herbert and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  57. Ruder, Sebastian, Ivan Vulic, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569–631. find
  58. Sahin, H. Bahadir, Caglar Tirkaz, Eray Yildiz, Mustafa Tolga Eren, and Ozan Sonmez. 2017. Automatically annotated Turkish corpus for named entity recognition and text categorization using large-scale gazetteers. Available at http://arxiv.org/abs/1702.02363. — [arXiv:1702.02363](https://arxiv.org/abs/1702.02363) ↗
  59. Sang, Erik Tjong Kim and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147.
  60. Schnabel, Tobias, Igor Labutov, David M. Mimno, and Thorsten Joachims. 2015. Evaluation methods for unsupervised word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, pages 298–307, Lisbon.
  61. Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725.
  62. Shi, Xing, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT Learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526–1534.
  63. Sylak-Glassman, John. 2016. The composition and use of the universal morphological feature schema (UniMorph schema). Technical report, Center for Language and Speech Processing, Johns Hopkins University. find
  64. Sylak-Glassman, John, Christo Kirov, David Yarowsky, and Roger Que. 2015. A language-independent feature schema for inflectional morphology. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, ACL 2015, Volume 2: Short Papers, pages 674–680, Beijing.
  65. Tal Linzen, Tal, Grzegorz Chrupała, and Afra Alishahi. 2018. Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 1–18. find
  66. Tenney, Ian, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In International Conference on Le…
  67. Tsvetkov, Yulia, Manaal Faruqui, Wang Ling, Guillaume Lample, and Chris Dyer. 2015. Evaluation of word vector representations by subspace alignment. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2049–2054. find
  68. Vania, Clara, Andreas Grivas, and Adam Lopez. 2018. What do character-level models learn about morphology? The case of dependency parsing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2573–2583. find
  69. Vania, Clara and Adam Lopez. 2017. From characters to words to in between: Do we capture morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2016–2027. find
  70. Veldhoen, Sara, Dieuwke Hupkes, and Willem H. Zuidema. 2016. Diagnostic classifiers revealing how neural networks process hierarchical structure. In CoCo@NIPS, pages 69–77. find
  71. Williams, Adina, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long P…
  72. Yang, Wei, Wei Lu, and Vincent Zheng. 2017. A simple regularization-based algorithm for learning cross-domain word embeddings. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, pages 2898–2904, Copenhagen. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sahin-2020-linspector-multilingual-probing-tasks-word/