📄 SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding

Summary

Idiomatic multiword expressions, such as noun compounds whose meaning cannot be derived from their parts, remain difficult for language models. This shared task asks two things: whether systems can tell idiomatic uses from literal ones, and whether their sentence embeddings reflect the intended meaning of such expressions. The organisers extended an earlier English–Portuguese resource with Galician. They chose potentially ambiguous compounds, collected sentences with surrounding context from news, Wikipedia or CC-100, wrote short paraphrases for every sense and had two annotators label each example.

Subtask A is binary idiomaticity classification in Zero Shot and One Shot settings. Subtask B is a semantic similarity task scored with Spearman correlation, in Pre Train and Fine Tune settings. About 100 people in 25 teams took part, and the best systems beat the transformer baselines.

The analysis shows that generalising to unseen idioms and exploiting a single example call for quite different strategies. The task supplies a multilingual benchmark and baselines for studying non-compositional meaning, including in a low-resource language.

Key points

• The dataset has 8,683 entries: English 5,352, Portuguese 2,555 and Galician 776. Galician had only 63 training items (One Shot data), so the test checks cross-lingual transfer to a low-resource language.
• English and Portuguese compounds came from the Noun Compound Senses dataset and were spread across compositionality levels. Galician noun-adjective compounds were mined from Wikipedia and CC-100. 'Proper Noun' and 'Meta Usage' were added as extra labels.
• Participation reached roughly 100 registrants in 25 teams. They made over 650 submissions in the practice phase and over 150 in the evaluation phase. The CodaLab competition remains open post-evaluation.
• In Subtask A, regularisation, data augmentation and dropout helped most in the Zero Shot setting. Larger, ensembled or adversarially trained models did best with One Shot data. A linguistically motivated span–context similarity method placed third in Zero Shot, about 11 points behind the leader.
• In Subtask B, the top Pre Train score was 0.6402, against a baseline of 0.4810. That system learned single-token MWE representations. YNU-HPCC led Fine Tune with 0.6648, against a baseline of 0.5951.