Elamur Research⌕
☰
home / papers / 2015 / SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability

SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability

Summary

This paper reports on the 2015 edition of the SemEval shared task on semantic textual similarity (STS). In STS, a system assigns a graded, symmetric score showing how close in meaning two short texts are. The organizers built new evaluation data in English and Spanish. The English data includes news headlines (HDL), image captions (Images), tutoring-system answers (Answers-student), Stack Exchange answers (Answers-forum) and DEFT discussion-forum sentences (Belief). The Spanish data adds news and Wikipedia pairs. Candidate pairs were sampled by string similarity, labelled by crowd workers and then filtered automatically.

A new pilot subtask on interpretable STS asked systems to explain their scores. Systems had to align chunks across the two sentences, label each alignment with a relation type and give each aligned pair its own score. Participation was strong:

  • 29 teams (74 runs) in English;
  • 7 teams (16 runs) in Spanish;
  • 7 teams (29 runs) in the pilot.

In the pilot, systems did clearly worse when they had to find chunks themselves than when gold chunks were provided. The task moves STS toward explainable similarity judgments that are useful for MT evaluation, QA and summarization.

Key points

  • English test sets: HDL, Images and Answers-student have 750 pairs each; Answers-forum and Belief have 375 each. Training data came from the 2012–2014 STS releases (MSRpar, MSRvid, OnWN, SMTnews, Deft-forum, Tweet-news, etc.).
  • Scoring scales: English uses 0–5. Spanish uses 0–4, with the English grades 3 and 4 merged into a single grade.
  • English annotation setup on Amazon Mechanical Turk:
    • only Master-qualified workers;
    • 5 pairs per HIT at $0.20;
    • 5 labels per pair;
    • 105 hidden gold pairs from STS 2013, used to detect and remove poor annotators.
  • Interpretable subtask design:
    • run on Headlines and Images;
    • two scenarios, gold chunks and system chunks;
    • F-measures for alignment, type, score and type+score.
  • Interpretable subtask results:
    • Images was generally harder than Headlines. With gold chunks, the average F Type+Score was 0.4748 on Images versus 0.5381 on Headlines.
    • In the system-chunks scenario, the UMDuluth BlueTeam runs ranked first on both Headlines and Images.

Interesting ideas

  • I1. Balanced sampling by string similarity: Candidate pairs were drawn evenly across string-similarity ranges, both from same-cluster and different-cluster sources. This gives a dataset that covers the whole similarity scale instead of piling up near zero. It is cheap to reuse when building any graded-similarity benchmark, for example a Ukrainian one.
  • I2. Automatic filtering against three criteria: After annotation, pairs were selected purely automatically to balance three goals: an even score spread, high annotator agreement, and difficulty for a string-matching baseline. The release of the scripts and raw labels makes the final gold standard reproducible.
  • I3. Gold items used only for after-the-fact cleanup: Hidden gold pairs were not used to train or auto-ban workers. They served only to identify and discard bad annotators' data later. This is a softer quality-control scheme that avoids discouraging skilled crowd workers.
  • I4. Agreement measured like system evaluation: Annotator quality was estimated by correlating each annotator with the average of the others. Because this matches how systems are scored, it doubles as a rough human upper bound.
  • I5. Explanations evaluated as a layered alignment: Interpretability was operationalized as chunk alignment plus relation type plus per-chunk score, each with its own F-measure. Comparing a gold-chunk scenario with a system-chunk scenario separates segmentation errors from alignment errors, which is a useful diagnostic for any explainable-NLP evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

292 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 275 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s15-2045

@inproceedings{agirre2015semeval,
  title = {{SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability}},
  author = {Agirre, Eneko and Banea, Carmen and Cardie, Claire and Cer, Daniel and Diab, Mona and Gonzalez-Agirre, Aitor and Guo, Weiwei and Lopez-Gazpio, Iñigo and Maritxalar, Montse and Mihalcea, Rada and Rigau, German and Uria, Larraitz and Wiebe, Janyce},
  booktitle = {Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)},
  year = {2015},
  doi = {10.18653/v1/s15-2045},
  url = {https://doi.org/10.18653/v1/s15-2045}
}

References

17 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Abney. 1991. Parsing by chunks. In Principle-based parsing: Computation and psycholinguistics. Robert Berwick and Steven Abney and Carol Tenny(eds.), pages 257–278. find
  2. Rodrigo Agerri, Josu Bermudez, and German Rigau. 2014. Ixa pipeline: Efficient and ready to use multilingual NLP tools. In Proceedings of the 9th Language Resources and Evaluation Conference (LREC 2014), pages 26–31. find
  3. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 Task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceed… find our card
  4. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 Task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2… find our card
  5. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL 2014). find our card
  6. Clive Best, Erik van der Goot, Ken Blackler, Teófilo Garcia, and David Horby. 2005. Europe Media Monitor - System description. In EUR Report 22173-En, Ispra, Italy. find
  7. Chris Brockett. 2007. Aligning the RTE 2006 corpus. Microsoft Research. find
  8. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2013), pages 758–764, Atlanta, Georgia, J… find our card
  9. Omer Levy, Torsten Zesch, Ido Dagan, and Iryna Gurevych. 2013. Recognizing partial textual entailment. In ACL (2), pages 451–455. find
  10. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of Workshop at the International Conference on Learning Representations (ICLR 2013). find
  11. George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41, November. find our card
  12. Rodney D. Nielsen, Wayne Ward, and James H. Martin. 2009. Recognizing entailment in intelligent tutoring systems. Natural Language Engineering, 15(04):479– 501. find
  13. William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. Numerical recipes: The art of scientific computing V 2.10 with Linux or single-screen license. find
  14. Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. 2010. Collecting image annotations using Amazon’s Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, CSLDAMT 2010, pages 139–147. find our card
  15. Vasile Rus, Mihai Lintean, Cristian Moldovan, William Baggett, Nobal Niraula, and Brent Morgan. 2012. The SIMILAR corpus: A resource to foster the qualitative understanding of semantic similarity of texts. In Semantic Relations II: Enhancing Resources and Applications, The 8th Language Resources and… find
  16. Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task: Chunking. In Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning-Volume 7 (CoNLL 2000), pages 127–132. find our card
  17. Frane Šarić, Goran Glavaš, Mladen Karan, Jan Šnajder, and Bojana Dalbelo Bašić. 2012. TakeLab: Systems for measuring semantic text similarity. In Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), Montréal, Canada, 7-8 June. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/agirre-2015-semeval-2015-task-2-semantic/