Elamur Research
☰
home / papers / 2012 / SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity

SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/S12-1051.pdf

Summary

The paper describes the pilot SemEval-2012 Task 6 shared task on Semantic Textual Similarity (STS). Participants had to judge how close in meaning two sentences are. Unlike textual entailment and paraphrase, similarity here is symmetric and graded rather than a binary "yes/no". No comparable dataset for this task existed before, so the organizers assembled sentence pairs from existing resources. For training, they took 2000 pairs from MSRpar, MSRvid and "reference translation – machine translation" pairs from WMT. The test set contains another 2000 pairs from the same sources plus two surprise out-of-domain datasets. Pairs were selected by bands of string similarity in order to cover different levels of similarity evenly. Each pair was rated by crowdsourced annotators on Amazon Mechanical Turk on a 0–5 scale, where each score has a verbal definition. Systems were compared with human judgments using Pearson correlation. The best systems exceeded 80%, whereas a simple lexical baseline scored only 31%. This paper laid the foundation for subsequent STS benchmarks. At the same time, the authors acknowledge that the evaluation metric still needs refinement.

Key points

  • Data: 2000 training and 2000 test pairs from MSRpar, MSRvid and SMTeuroparl (WMT 2007/2008, fr-en only). Surprise test sets: about 400 pairs from WMT 2007 news (399 in practice, due to duplicates) and 750 pairs of definitions (glosses) from OntoNotes 4.0 and WordNet 3.1.
  • Annotation: Amazon Mechanical Turk, 5 ratings for each HIT of 5 pairs, paid $0.20 per HIT. Annotators were required to have a 95% approval rating and to pass a qualification test. Ratings from annotators whose correlation with the gold standard was below 50% were discarded.
  • Agreement: in the authors' own pilot annotation, the Pearson correlation between them was 84–87%. AMT annotator agreement was about 90%.
  • Participation and results: 35 teams submitted 88 runs. The best systems achieved a Pearson correlation above 0.80, the lexical baseline 0.31.
  • Open questions: in the end, three different evaluation metrics were published, since none proved fully satisfactory. Among the resources most often used by participants were WordNet, Wikipedia, monolingual corpora, as well as machine learning and string similarity tools.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

460 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 439 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{agirre2012semeval,
  title = {{SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity}},
  author = {Agirre, Eneko and Cer, Daniel and Diab, Mona and Gonzalez-Agirre, Aitor},
  booktitle = {*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)},
  year = {2012},
  url = {https://aclanthology.org/S12-1051/}
}

References

10 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT ’07, pages 136–158. find
  2. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, StatMT ’08, pages 70–106. find
  3. David L. Chen and William B. Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meetings of the Association for Computational Linguistics (ACL). find
  4. B. Dolan, C. Quirk, and C. Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In COLING 04: Proceedings of the 20th international conference on Computational Linguistics, page 350. find our card
  5. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. MIT Press. find
  6. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL. find
  7. Michael D. Lee, Brandon Pincombe, and Matthew Welsh. 2005. An empirical evaluation of models of text document similarity. In Proceedings of the 27th Annual Conference of the Cognitive Science Society, pages 1254–1259, Mahwah, NJ. find
  8. Y. Li, D. McLean, Z. A. Bandar, J. D. O’Shea, and K. Crockett. 2006. Sentence similarity based on semantic nets and corpus statistics. IEEE Transactions on Knowledge and Data Engineering, 18(8):1138– 1150, August. find
  9. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Commun. ACM, 8(10):627–633, October. find
  10. E. Ukkonen. 1985. Algorithms for approximate string matching. Information and Contro, 64:110–118. find