Elamur Research⌕
☰
home / papers / 2014 / A SICK cure for the evaluation of compositional distributional semantic models

A SICK cure for the evaluation of compositional distributional semantic models

Summary

Compositional distributional semantic models (CDSMs) build a sentence meaning vector from the vectors of individual words. The authors note that suitable data for evaluating such models has been lacking. Older datasets contained only a few hundred short sentences with the same structure. The STS and RTE corpora, by contrast, require recognizing named entities, handling dates, and having encyclopedic knowledge, which goes beyond the task of composition. To fill this gap, the authors created SICK, an English benchmark. As source material they took image and video descriptions from 8K ImageFlickr and SemEval 2012 STS MSR-Video. The sentences were first normalized by removing named entities, numbers, and multiword expressions. Each sentence was then transformed into new variants: one close in meaning, one with negation or contrast, and one with the same words in a different order. The resulting pairs were rated through crowdsourcing in two ways: by degree of semantic relatedness and by type of logical relation. The final resource became the basis of SemEval-2014 Task 1. It makes it possible to test specifically the compositional abilities of models without side NLP tasks.

Key points

  • The dataset contains about 10,000 pairs of English sentences. Each pair has a relatedness score on a scale from 1 to 5 and one of the labels ENTAILMENT, CONTRADICTION or NEUTRAL (the RTE-4 scheme).
  • The starting point was 750 + 750 pairs from two sources. Each sentence was normalized by two annotators, and their versions matched in only about half of the cases. The remaining disputed variants were resolved by a third expert.
  • The labels were distributed as follows: NEUTRAL about 57%, ENTAILMENT about 29%, CONTRADICTION about 14%. Mean relatedness is 4.57 for ENTAILMENT, 3.59 for CONTRADICTION and 2.98 for NEUTRAL, and these differences are statistically significant.
  • Pairs close in meaning from one of the datasets received the ENTAILMENT label in 94.2% of cases. Pairs with negation or contrast were labeled CONTRADICTION in only 58.2% of cases. The reason lies in indefinite subjects: participants assumed that the sentences might describe different people.
  • A simple word-overlap baseline yields a Pearson correlation of 0.63 for relatedness and an accuracy of 56.2% for entailment. The majority baseline for entailment reaches 56.7%. The data is distributed under the CC BY-NC-SA license.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Evaluation & benchmarks

Institutions

🔬Fondazione Bruno KesslerTrento · Italy18 works · 42 903 citations🎓University of TrentoTrento · Italy50 works · 13 649 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{marelli2014sick,
  title = {{A SICK cure for the evaluation of compositional distributional semantic models}},
  author = {Marelli, Marco and Menini, Stefano and Baroni, Marco and Bentivogli, Luisa and Bernardi, Raffaella and Zamparelli, Roberto},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/39qdhuevzbqa},
  url = {https://doi.org/10.63317/39qdhuevzbqa}
}

References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdi, H. and Williams, L. (2010). Newman-Keuls and Tukey test. In Salkind, N., Frey, B., and Dougherty, D., editors, Encyclopedia of Research Design, pages 897– 904. Sage, Thousand Oaks, CA. find
  2. Agirre, E., Cer, D., Diab, M., and Gonzalez-Agirre., A. (2012). Semeval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), volume 2. find our card
  3. Baroni, M. and Zamparelli, R. (2010). Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of EMNLP, pages 1183–1193, Boston, MA. http://alt.qcri.org/semeval2014/task1/http://clic.cimec.unitn.it/composes/sick/ ↗
  4. Cooper, R., Crouch, D., Eijck, J. V., Fox, C., Genabith, J. V., Jaspars, J., Kamp, H., Milward, D., Pinkal, M., Poesio, M., and Pulman, S. (1996). Using the framework. Technical Report Technical Report LRE 62-051 D-16, The FraCaS Consortium. find
  5. Giampiccolo, D., Dang, H. T., Bernardo, M., Dagan, I., and Cabrio, E. (2008). The fourth pascal recognising textual entailment challenge. In n Proceedings of the TAC 2008 Workshop on Textual Entailment. find
  6. Grefenstette, E. and Sadrzadeh, M. (2011). Experimental support for a categorical compositional distributional model of meaning. In Proceedings of EMNLP, pages 1394–1404, Edinburgh, UK. find
  7. MacCartney, B. and Manning, C. D. (2007). Natural logic for textual inference. In ACL Workshop on Textual Entailment and Paraphrasing. find
  8. Marelli, M., Bentivogli, L., Baroni, M., Bernardi, R., Menini, S., and Zamparelli, R. (2014). Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of SemEval 2014: International Workshop on Semantic Evaluation. In press. find our card
  9. Mitchell, J. and Lapata, M. (2008). Vector-based models of semantic composition. In Proceedings of ACL, pages 236–244, Columbus, OH. find
  10. Mitchell, J. and Lapata, M. (2010). Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  11. Negri, M., Marchetti, A., Mehdad, Y., Bentivogli, L., and Giampiccolo, D. (2012). Semeval-2012 Task 8: Cross-lingual Textual Entailment for Content Synchronization. In Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012). find
  12. Negri, M., Marchetti, A., Mehdad, Y., Bentivogli, L., and Giampiccolo, D. (2013). Semeval-2013 Task 8: Cross-lingual Textual Entailment for Content Synchronization. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013). find
  13. Snow, R., O’Connor, B., Jurafsky, D., and Ng, A. Y. (2008). Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 254– 263, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  14. Socher, R., Huval, B., Manning, C., and Ng, A. (2012). Semantic compositionality through recursive matrix-vector spaces. In Proceedings of EMNLP, pages 1201–1211, Jeju Island, Korea. find our card
  15. Toledo, A., Katrenko, S., Alexandropoulou, S., Klockmann, H., Stern, A., Dagan, I., and Winter, Y. (2012). Semantic annotation for textual entailment recognition. In Springer-Verlag, editor, Proceedings of the 11th Mexican International Conference on Artificial Intelligence, Lecture Notes in Artificial Inteligence, pages 12–25. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/