Elamur Research
☰
home / papers / 2014 / SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment

SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/S14-2001.pdf

Summary

This SemEval-2014 shared task targeted a gap in benchmarking compositional distributional semantic models (CDSMs). Earlier sentence-level test sets were either tiny and syntactically rigid, or, like STS and RTE, mixed in difficulties unrelated to composition, such as named entities and encyclopedic facts. The organizers built SICK (Sentences Involving Compositional Knowledge), about 10,000 English sentence pairs derived from image and video captions. They simplified the original sentences, then rewrote them into paraphrases, contradictions, and word-shuffled variants with changed meaning, and added random unrelated pairings. Crowdworkers rated each pair for relatedness and assigned an entailment label. Systems of any type could take part in two subtasks: predicting graded relatedness and three-way entailment classification. Twenty-one teams participated. The leading systems used compositional features alongside non-compositional cues and resources such as WordNet. Systems built purely on composition performed around the average of the field. Some teams also found shortcuts in the data, which points to concrete improvements for future benchmarks.

Key points

  • SICK was assembled from 750 image description pairs from 8K ImageFlickr and 750 pairs from the SemEval-2012 STS MSR-Video Descriptions data. It was divided evenly into training and test halves.
  • Ten annotators judged each pair, with sentence order counterbalanced. Gold relatedness is the mean rating on a 1–5 scale. The entailment gold label (ENTAILMENT / CONTRADICTION / NEUTRAL) comes from a majority vote.
  • The relatedness subtask received 17 submissions (66 runs) and the entailment subtask 18 (65 runs). Composition was used by 13 of the 21 teams.
  • The best results were roughly 0.82 Pearson r for relatedness and 84.5% accuracy for entailment. The non-compositional UNAL-NLP came close, placing 4th (0.80 r) and 3rd (83%). Purely compositional approaches exceeded 70% accuracy and 0.70 r.
  • According to Illinois-LH, the presence of negation words alone identifies 86.4% of contradiction pairs, an artifact the organizers say future releases of SICK should control for.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

142 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 133 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/s14-2001

@inproceedings{marelli2014semeval,
  title = {{SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment}},
  author = {Marelli, Marco and Bentivogli, Luisa and Baroni, Marco and Bernardi, Raffaella and Menini, Stefano and Zamparelli, Roberto},
  booktitle = {Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)},
  year = {2014},
  doi = {10.3115/v1/s14-2001},
  url = {https://doi.org/10.3115/v1/s14-2001}
}

References

23 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), volume 2. find our card
  2. Ana O. Alves, Adirana Ferrugento, Mariana Lorenço, and Filipe Rodrigues. 2014. ASAP: Automatica semantic alignment for phrases. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find
  3. Marco Baroni and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of EMNLP, pages 1183–1193, Boston, MA. find our card
  4. Islam Beltagy, Stephen Roller, Gemma Boleda, Katrin Erk, and Raymon J. Mooney. 2014. UTexas: Natural language semantics using distributional semantics and probablisitc logic. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find
  5. Luisa Bentivogli, Ido Dagan, Hoa T. Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge. In The Text Analysis Conference (TAC 2009). find
  6. Yves Bestgen. 2014. CECL: a new baseline and a non-compositional approach for the Sick benchmark. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find
  7. Ergun Biçici and Andy Way. 2014. RTM-DCU: Referential translation machines for semantic similarity. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find
  8. Johannes Bjerva, Johan Bos, Rob van der Goot, and Malvina Nissim. 2014. The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find
  9. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. Evaluating predictive uncertainty, visual object classification, and recognising textual entailment, pages 177– 190. Springer. find
  10. Lorenzo Ferrone and Fabio Massimo Zanzotto. 2014. haLF:comparing a pure CDSM approach and a standard ML system for RTE. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find
  11. Edward Grefenstette and Mehrnoosh Sadrzadeh. 2011. Experimental support for a categorical compositional distributional model of meaning. In Proceedings of EMNLP, pages 1394–1404, Edinburgh, UK. find
  12. Rohit Gupta, Ismail El Maarouf Hannah Bechara, and Costantin Orasǎn. 2014. UoW: NLP techniques developed at the University of Wolverhampton for Semantic Similarity and Textual Entailment. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find
  13. Sergio Jimenez, George Duenas, Julia Baquero, and Alexander Gelbukh. 2014. UNAL-NLP: Combining soft cardinality features for semantic textual similarity, relatedness and entailment. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find
  14. Alice Lai and Julia Hockenmaier. 2014. Illinois-lh: A denotational and distributional approach to semantics. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find
  15. Saúl León, Darnes Vilarino, David Pinto, Mireya Tovar, and Beatrice Beltrán. 2014. BUAP:evaluating compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find
  16. Elisabeth Lien and Milen Kouylekov. 2014. UIO-Lien: Entailment recognition using minimal recursion semantics. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find
  17. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of LREC, Reykjavik. find
  18. Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In Proceedings of ACL, pages 236–244, Columbus, OH. find
  19. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  20. Thomas Proisl and Stefan Evert. 2014. SemantiK-LUE: Robust semantic similarity at multiple levels using maximum weight matching. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find
  21. Richard Socher, Brody Huval, Christopher Manning, and Andrew Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of EMNLP, pages 1201–1211, Jeju Island, Korea. find
  22. An N. P. Vo, Octavian Popescu, and Tommaso Caselli. 2014. FBK-TR: SVM for Semantic Relatedness and Corpus Patterns for RTE. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find
  23. Jiang Zhao, Tian Tian Zhu, and Man Lan. 2014. ECNU: One Stone Two Birds: Ensemble of Heterogenous Measures for Semantic Relatedness and Textual Entailment. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. find