Elamur Research⌕
☰
home / papers / 2020 / XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization

XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization

Paper · 2020 · Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados et al.

Summary

Testing whether contextualized models separate the different meanings of a word usually relies on WSD benchmarks tied to sense inventories such as WordNet. This makes it hard to evaluate inventory-free models. The English Word-in-Context (WiC) task avoids that dependence by asking a yes/no question: is a target word used with the same sense in two sentences? XL-WiC extends that setup to 12 additional languages, from several families and with uneven resource levels. The data comes from example sentences in Multilingual WordNet versions and in Wiktionary. Pairs from the same sense become positive instances, and pairs from different senses become negative ones. Overly fine-grained WordNet sense pairs are removed. The Farsi set was assembled semi-automatically by a human annotator. The authors evaluated mBERT, XLM-R and language-specific BERT models. Systems trained only on English transfer reasonably well zero-shot, even to distant languages. However, when sizeable target-language training data exists, as in Wiktionary, models trained on that data perform clearly better. The benchmark supports both monolingual evaluation of lexical semantics and research on cross-lingual transfer.

Key points

  • Coverage and size: XL-WiC has more than 80K instances in 12 new languages: BG, ZH, HR, DA, NL, ET, FA, FR, DE, IT, JA and KO. Together with English WiC, it covers 13 languages.
  • Data sources and filtering:
    • Most languages draw on their WordNet versions.
    • German, French and Italian use Wiktionary, which yields larger training sets.
    • Farsi was built from FarsNet v3.0 (about 30K synsets and over 100K word entries), with an annotator grouping the example sentences into pairs.
    • Sense pairs were dropped if their senses are linked in WordNet or share one of the 44 supersenses (lexicographer files).
  • Model comparison:
    • XLMR-large performed best in the zero-shot setting.
    • On Wiktionary, zero-shot scores trailed target-language training by roughly 10–20 points.
    • In monolingual training, XLMR-large stayed close to language-specific models (L-BERT): 1.1 points higher for German, and 2 and 0.3 points lower for French and Italian.
  • Seen vs. unseen target words: Multilingual models scored 1–12 points lower on target words absent from training (OOV). CamemBERT-large was a notable exception, scoring 0.4 points higher on the OOV subset.
  • Few-shot training: With only 10 training examples, models performed around chance level (50%). With small training subsets (under 5% of the full data), models reached about 85% of their full-data performance. Japanese and Korean remain the hardest languages for cross-lingual transfer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom155 works · 115,562 citations🎓Sapienza University of RomeRome · Italy36 works · 8,842 citations🎓University of HelsinkiHelsinki · Finland33 works · 4,705 citations🎓Cardiff UniversityCardiff · United Kingdom10 works · 2,319 citations🎓Institut des Hautes Études de TunisTunis · Tunisia3 works · 284 citations🔬Israel Institute for Advanced StudiesJerusalem · Israel3 works · 284 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 685 worksMultilingual & cross-lingual NLP · 465 works

Cite

Cite as Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados and Mohammad Taher Pilehvar. 2020. XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.584 · arXiv 2010.06478

@inproceedings{raganato2020xl,
  title = {{XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization}},
  author = {Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados and Mohammad Taher Pilehvar},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.584},
  eprint = {2010.06478},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.584}
}

References

67 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. \#\# Acknowledgments Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre. 2020b. A call for more rigor in unsupervised cross-lingual learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7375–7388, Online. Association for… find
  2. We would like to thank Qincheng Zhang (Chinese), Janine Siewert (Danish), Houman Mehrafarin, Hossein Mohebbi, and Ali Modarresi (Farsi), Ángela Collados Aı́s (German), Asahi Ushio (Japanese), and Yunseo Joung (Korean) for their help with the manual evaluation of the datasets. Alessandro gratefully… find
  3. Tommaso gratefully acknowledges the support of the ERC Consolidator Grant MOUSSE No. 726487 under the European Union’s Horizon 2020 research and innovation programme. find
  4. Eneko Agirre, Oier Lopez de Lacalle, and Aitor Soroa. 2014. Random walks for knowledge-based word sense disambiguation. Computational Linguistics, 40(1):57–84.
  5. Antonios Anastasopoulos and Graham Neubig. 2020. Should all cross-lingual embeddings speak English? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8658–8679, Online. Association for Computational Linguistics. find
  6. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020a. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
  7. Jordi Atserias, Luı́s Villarejo, and German Rigau. 2004. Spanish WordNet 1.6: Porting the Spanish Wordnet across Princeton versions. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal. European Language Resources Association (ELRA). find
  8. Siamak Barzegar, Brian Davis, Manel Zarrouk, Siegfried Handschuh, and Andre Freitas. 2018. SemR-11: A multi-lingual gold-standard for semantic similarity and relatedness for eleven languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Mi… find
  9. Laura Benı́tez, Sergi Cervell, Gerard Escudero, Mònica López, German Rigau, and Mariona Taulé. 1998. Methods and tools for building the catalan wordnet. Proceedings of ELRA Workshop on Language Resources for European Minority Languages. find
  10. Michele Bevilacqua and Roberto Navigli. 2020. Breaking through the 80% Glass Ceiling: Raising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information. In Proc. of ACL, pages 2854–2864.
  11. Davis, and Martha Palmer. 2007. Criteria for the manual grouping of verb senses. In Proceedings of the Linguistic Annotation Workshop, pages 49–52. find
  12. Terra Blevins and Luke Zettlemoyer. 2020. Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders. In Proc. of ACL, pages 1006–1017.
  13. Philip Edmonds and Scott Cotton. 2001. SENSEVAL-2: Overview. In Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems, pages 1–5, Toulouse, France. Association for Computational Linguistics.
  14. Francis Bond and Kyonghee Paik. 2012. A survey of wordnets and their licenses. In GWC 2012 6th International Global Wordnet Conference, volume 8, pages 64–71. find
  15. Mehrdad Farahani, Mohammad Gharachorloo, Marzieh Farahani, and Mohammad Manthouri. 2020. Parsbert: Transformer-based model for persian language understanding. find
  16. Jose Camacho-Collados, Mohammad Taher Pilehvar, Nigel Collier, and Roberto Navigli. 2017. SemEval-2017 task 2: Multilingual and cross-lingual semantic word similarity. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 15–26, Vancouver, Canada. Association… find
  17. Christiane Fellbaum, editor. 1998. WordNet: An Electronic Database. MIT Press, Cambridge, MA. find
  18. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.…
  19. Christos Christodouloupoulos and Mark Steedman. 2015. A massively parallel corpus: the bible in 100 languages. Language resources and evaluation, 49(2):375–395.
  20. Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactionsof the Association of Computational Linguistics.
  21. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… find
  22. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  23. Cecily Jill Duffield, Jena D Hwang, Susan Windisch Brown, Dmitriy Dligach, Sarah Vieweg, Jenny Darja Fišer, Jernej Novak, and Tomaž Erjavec. 2012. slownet 3.0: development, extension and cleaning. In Proceedings of 6th International Global Wordnet Conference (GWC 2012), pages 113–117. find
  24. Hamidreza Ghader and Christof Monz. 2017. What does attention in neural machine translation pay attention to? In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 30–39, Taipei, Taiwan. Asian Federation of Natural Language Processi… find
  25. Xavier Gómez Guinovart. 2011. Galnet: Wordnet 3.0 do galego. Linguamática, 3(1):61–67. find
  26. Birgit Hamp and Helmut Feldweg. 1997. Germanet-a lexical-semantic net for german. In Automatic information extraction and building of lexical semantic resources for NLP applications.
  27. Daniel Hershcovich, Zohar Aizenbud, Leshem Choshen, Elior Sulem, Ari Rappoport, and Omri Abend. 2019. SemEval-2019 task 1: Cross-lingual semantic parsing with UCCA. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1–10, Minneapolis, Minnesota, USA. Association for Comp… find
  28. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. Proceedings of the 37th International Conference on Machine Learning (ICML).
  29. Chu-Ren Huang, Shu-Kai Hsieh, Jia-Fei Hong, Yun-Zhu Chen, I-Li Su, Yong-Xiang Chen, and Sheng-Wei Huang. 2010. Chinese Wordnet: Design, Implementation and Application of an Infrastructure for Cross-Lingual Knowledge Processing. Journal of Chinese Information Processing, 24(2):14. find
  30. Eric Huang, Richard Socher, Christopher Manning, and Andrew Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 873–882, Jeju Island, Korea.…
  31. Hitoshi Isahara, Francis Bond, Kiyotaka Uchimoto, Masao Utiyama, and Kyoko Kanzaki. 2008. Development of the Japanese WordNet. In Sixth International conference on Language Resources and Evaluation. find
  32. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, Syste…
  33. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find
  34. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver. Association for Computational Linguistics. find
  35. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315– 7330, Online. Association for Computational Lingu…
  36. Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019. On the word alignment from neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1293–1303, Florence, Italy. Association for Computational Linguistics. find
  37. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoı̂t Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–721…
  38. Márton Miháltz, Csaba Hatvani, Judit Kuti, György Szarvas, János Csirik, Gábor Prószéky, and Tamás Váradi. 2008. Methods and Results of the Hungarian WordNet Project. In Proceedings of The Fourth Global WordNet Conference, pages 311–321. find
  39. George A Miller. 1995. WordNet: a lexical database for english. Communications of the ACM, 38(11):39–41.
  40. Andrea Moro and Roberto Navigli. 2015a. SemEval-2015 task 13: Multilingual all-words sense disambiguation and entity linking. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 288–297, Denver, Colorado. Association for Computational Linguistics. find
  41. Andrea Moro and Roberto Navigli. 2015b. Semeval-2015 task 13: Multilingual all-words sense disambiguation and entity linking. Proceedings of SemEval-2015. find
  42. Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity Linking meets Word Sense Disambiguation: a Unified Approach. Transactions of the Association for Computational Linguistics (TACL), 2:231–244.
  43. Roberto Navigli. 2009. Word Sense Disambiguation: A survey. ACM Computing Surveys, 41(2):1–69. find
  44. Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. SemEval-2013 task 12: Multilingual word sense disambiguation. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages… find
  45. Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217–250. find
  46. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  47. Bolette S. Pedersen, Sanni Nimb, Jørg Asmussen, Nicolai Hartvig Sørensen, Lars Trap-Jensen, and Henrik Lorentzen. 2009. DanNet: the challenge of compiling a wordnet for Danish by reusing a monolingual dictionary. Language Resources and Evaluation, 43:269–299. find
  48. Emanuele Pianta, Luisa Bentivogli, and Christian Girardi. 2002. MultiWordNet: Developing an Aligned Multilingual Database. In Proceedings of the 1st International Conference on Global WordNet, pages 293–302. find
  49. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the Word-in-Context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
  50. Eli Pociello, Antton Gurrutxaga, Eneko Agirre, Izaskun Aldezabal, and German Rigau. 2008. WN-TERM: Enriching the MCR with a terminological dictionary. In LREC 2008. find
  51. Marten Postma, Emiel van Miltenburg, Roxane Segers, Anneleen Schoen, and Piek Vossen. 2016. Open Dutch WordNet. In Proceedings of the Eight Global Wordnet Conference. find
  52. Quentin Pradet, Gaël de Chalendar, and Jeanne Des-ormeaux Baguenier. 2014. Wonef, an improved, expanded and evaluated automatic french translation of wordnet. In Proceedings of the Seventh Global Wordnet Conference (GWC2014), pages 32–39. find
  53. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In In Association for Computational Linguistics (ACL) System Demonstrations.
  54. Ida Raffaelli, Marko Tadić, Božo Bekavac, and Željko Agić. 2008. Building croatian wordnet. In Fourth global wordnet conference (gwc 2008). find
  55. Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017. Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison. In Proc. of EACL, pages 99–110.
  56. Ervin Ruci. 2008. On the current state of Albanet and related applications. Technical report, University of Vlora. (http://fjalnet.com/technicalreportalbanet. pdf). find
  57. Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2020. With More Contexts Comes Better Performance: Contextualized Sense Embeddings for All-Round Word Sense Disambiguation. In Proc. of EMNLP. find
  58. Karin Kipper Schuler, Anna Korhonen, and Susan Brown. 2009. VerbNet overview, extensions, mappings and applications. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Tutorial Ab… find
  59. Federico Scozzafava, Marco Maru, Fabrizio Brignone, Giovanni Torrisi, and Roberto Navigli. 2020. Personalized pagerank with syntagmatic information for multilingual word sense disambiguation. In Proc. of ACL, pages 37–46. find
  60. Mehrnoush Shamsfard, Akbar Hesabi, Hakimeh Fadaei, Niloofar Mansoory, Payam Noor, Ali Reza Gholi Famian, Somayeh Bagherbeigi, Elham Fekri, and Maliheh Monshizadeh. 2010. Semi automatic development of FarsNet; the Persian WordNet. In Proceedings of 5th global WordNet conference. find
  61. Kiril Simov and Petya Osenova. 2010. Constructing of an Ontology-based Lexicon for Bulgarian. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). find
  62. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA). find
  63. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT — Building open translation services for the World. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT), Lisbon, Portugal. find
  64. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  65. Kadri Vider and Heili Orav. 2002. Estonian WordNet and Lexicography. In Proceedings of the Eleventh International Symposium on Lexicography, pages 549–555. find
  66. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Proceedings of NeurIPS.
  67. Ae-Sun Yoon, Soon-Hee Hwang, Eun-Ryoung Lee, and Hyuk-Chul Kwon. 2009. Construction of Korean WordNet. Journal of KIISE: Software and Applications, 36(1):92–108. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/raganato-2020-xl-wic-multilingual-benchmark-evaluating/