Elamur Research
☰
home / papers / 2013 / *SEM 2013 shared task: Semantic Textual Similarity

*SEM 2013 shared task: Semantic Textual Similarity

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/S13-1004.pdf

Summary

This paper reports on the Semantic Textual Similarity (STS) shared task at *SEM 2013. Systems score how close two texts are in meaning on a graded 0–5 scale. The organizers ran two tracks. The CORE track followed the SemEval 2012 setup: the full 2012 data served as training material, but test pairs came from related yet different genres. These were news headlines clustered by the Europe Media Monitor, MT evaluation pairs (HTER and HYTER subsets), and gloss pairs linking OntoNotes or FrameNet to WordNet. The new TYPED pilot asked systems to explain why two items are similar. It used Europeana cultural heritage records, scored on general similarity plus seven specific dimensions such as author, time period and location. Gold labels were collected through CrowdFlower crowdsourcing, with inter-annotator correlations of 62–87%. CORE drew 34 participants submitting 89 runs, and TYPED drew 6 teams with 14 runs. The top CORE systems came close to human agreement, and TYPED results suggest that typed similarity is practically feasible. This makes STS a shared benchmark for combining and comparing semantic components.

Key points

  • CORE 2013 test sets:
    • HDL: 750 headline pairs, half from the same EMM cluster and half from different clusters, balanced by string similarity.
    • SMT: 750 pairs, made up of 150 HTER pairs from DARPA GALE and 600 HYTER pairs.
    • OnWN: 561 gloss pairs.
    • FNWN: 189 gloss pairs, drawn from FrameNet 1.5 mapped to WordNet 3.1.
  • TYPED contains 1500 Europeana item pairs, generated semi-automatically:
    • The seed set was 163 manually chosen pairs.
    • It was expanded via the Europeana API at "distance 1" and "distance 2".
    • Systems work from metadata fields such as dcTitle, dcSubject, dcDescription, dcCreator, dcDate and dcSource.
  • TYPED similarity types come from a PATHS project study:
    • author/creator, people involved, time period, location, event/action, subject and description;
    • a general similarity score is also included.
  • Resources and tools used by participants:
    • WordNet and monolingual corpora were the most common resources, followed by Wikipedia (CORE) and acronyms (TYPED).
    • Lemmatizers and POS taggers were widespread.
    • Top CORE systems include UMBC EBIQUITY-ParingWords and MayoClinicNLP-r3wtCD; deft-baseline relied solely on distributional similarity.
  • Planned future work includes data covering modality and sentiment, and typed similarity for unstructured text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

298 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 282 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{agirre2013sem,
  title = {{*SEM 2013 shared task: Semantic Textual Similarity}},
  author = {Agirre, Eneko and Cer, Daniel and Diab, Mona and Gonzalez-Agirre, Aitor and Guo, Weiwei},
  booktitle = {Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity},
  year = {2013},
  url = {https://aclanthology.org/S13-1004/}
}

References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre and Enrique Amigó. In prep. Exploring evaluation measures for semantic textual similarity. In Unpublished manuscript. find
  2. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceed… find our card
  3. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The berkeley framenet project. In COLING ’98 Proceedings of the 17th international conference on Computational linguistics - Volume 1. find our card
  4. Daniel Bar, Chris Biemann, Iryna Gurevych, and Torsten Zesch. 2012. Ukp: Computing semantic textual similarity by combining multiple content similarity measures. In Proceedings of the 6th International Workshop on Semantic Evaluation, in conjunction with the 1st Joint Conference on Lexical and Compu… find
  5. Clive Best, Erik van der Goot, Ken Blackler, Tefilo Garcia, and David Horby. 2005. Europe media monitor - system description. In EUR Report 22173-En, Ispra, Italy. find
  6. Markus Dreyer and Daniel Marcu. 2012. Hyter: Meaning-equivalent semantics for translation evaluation. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics. find
  7. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. MIT Press. find
  8. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL. find
  9. W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flannery. 2002. Numerical Recipes: The Art of Scientific Computing V 2.10 With Linux Or Single-Screen License. Cambridge University Press. find
  10. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of Association for Machine Translation in the Americas. find our card
  11. Frane Šarić, Goran Glavaš, Mladen Karan, Jan Šnajder, and Bojana Dalbelo Bašić. 2012. Takelab: Systems for measuring semantic text similarity. In Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 441–448, Montréal, Canada, 7-8 June. Association for C… find