Elamur Research
☰
home / papers / 2016 / SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation

SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/S16-1081.pdf

Summary

This paper reports on the 2016 edition of the Semantic Textual Similarity (STS) shared task. Systems receive pairs of sentence-length snippets and must predict how close their meanings are on a six-level scale. The levels run from 0 (unrelated) to 5 (fully equivalent), with defined intermediate grades for topical overlap or partial agreement. Systems are ranked by how well their scores correlate with human judgements, and they may output values on any scale.

The monolingual English subtask added new evaluation material from:

  • plagiarism detection
  • post-edited machine translation
  • question-question pairs from Q&A forums

It also revisited news headlines and answer-answer pairs. Earlier STS releases could be reused for training.

For the first time, the organizers ran a pilot cross-lingual Spanish-English subtask. Most entrants translated the Spanish side into English and then applied a monolingual system. Even so, the best cross-lingual results were close to what monolingual systems achieve on similar sources.

Participation grew substantially. The task offers a common benchmark for meaning-similarity methods relevant to MT evaluation, summarization and QA.

Key points

  • English test data: five sets (Headlines, Plagiarism, Postediting, Answer-Answer, Question-Question), each with 209–254 pairs.
  • English training data: 14,250 gold-labelled pairs from STS 2012–2015, plus about 5,500 additional pairs with noisy annotations from 2015.
  • Participation:
    • English: 43 teams submitted 119 runs, a 45% rise in teams over 2015.
    • Cross-lingual pilot: 10 teams submitted 26 runs.
  • Cross-lingual results (team UWB): an unsupervised system weighting aligned and unaligned words by IDF reached 0.912 on the news set. A supervised regression system combining lexical, syntactic and semantic features led the multi-source set with 0.819.
  • Genuinely cross-lingual approaches:
    • CNRC: XMEANT plus semantic-role features.
    • FBK HLT: cross-lingual embeddings plus QuEst features.
    • GWU NLP: a shared cross-lingual vector space.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

301 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 281 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s16-1081

@inproceedings{agirre2016semeval,
  title = {{SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation}},
  author = {Agirre, Eneko and Banea, Carmen and Cer, Daniel and Diab, Mona and Gonzalez-Agirre, Aitor and Mihalcea, Rada and Rigau, German and Wiebe, Janyce},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1081},
  url = {https://doi.org/10.18653/v1/s16-1081}
}

References

31 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Naveed Afzal, Yanshan Wang, and Hongfang Liu. 2016. MayoNLP at SemEval-2016 Task 1: Semantic textual similarity based on lexical semantic net and deep learning semantic model. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, CA, USA. find
  2. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 Task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceed… find our card
  3. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic Textual Similarity. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarit… find our card
  4. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 Task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2… find our card
  5. Hanan Aldarmaki and Mona Diab. 2016. GWU NLP at SemEval-2016 Shared Task 1: Matrix factorization for crosslingual STS. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, CA, USA. Duygu Ataman, Jose G. C. de Souza, and Marco Turchi1 Matteo Negri. 2016.… find
  6. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet Project. In COLING ’98 Proceedings of the 17th international conference on Computational linguistics - Volume 1. Clive Best, Erik van der Goot, Ken Blackler, Teófilo Garcia, and David Horby. 2005. Europe Media Monit… find our card
  7. Tomas Brychcin and Lukas Svoboda. 2016. UWB at SemEval-2016 Task 1: Semantic textual similarity using lexical, syntactic, and semantic information. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, CA, USA. find
  8. C. Christiane Fellbaum. 1998. find
  9. WordNet: An Electronic Lexical Database. MIT Press. Paul Clough and Mark Stevenson. 2011. Developing a corpus of plagiarised short answers. Language Resources and Evaluation, 45(1):5–24. find
  10. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010. Recognizing textual entailment: Rational, evaluation and approaches. Natural Language Engineering, 16:105–105. Günes Erkan and Dragomir R. Radev. 2004. LexRank: Graph-based lexical centrality as salience in text summarization. J. Artif. I… find
  11. Hua He, Kevin Gimpel, and Jimmy Lin. 2015. Multi-perspective sentence similarity modeling with convolutional neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1576–1586, Lisbon, Portugal. find
  12. Hua He, John Wieting, Kevin Gimpel, Jinfeng Rao, and Jimmy Lin. 2016. UMD-TTIC-UW at SemEval-2016 Task 1: Attention-based multi-perspective convolutional neural networks for textual similarity measurement. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San D… find
  13. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL. find
  14. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using click-through data. ACM International Conference on Information and Knowledge Management (CIKM). find
  15. Hideo Itoh. 2016. RICOH at SemEval-2016 Task 1: Ir-based semantic textual similarity estimation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, CA, USA. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Be… find
  16. Dekang Lin. 1998. An information-theoretic definition of similarity. In Proceedings of the Fifteenth International Conference on Machine Learning, ICML ’98, pages 296–304, San Francisco, CA, USA. find
  17. Chi-kiu Lo, Meriem Beloucif, Markus Saers, and Dekai Wu. 2014. XMEANT: Better semantic MT evaluation without reference translations. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). find
  18. Chi-kiu Lo, Cyril Goutte, and Michel Simard. 2016. CNRC at SemEval-2016 Task 1: Experiments in crosslingual semantic textual similarity. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, CA, USA. find
  19. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. find our card
  20. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of the International Conference on Learning Representations (ICLR), Scottsdale, AZ, USA. find
  21. George A. Miller. 1995. WordNet: A lexical database for english. Commun. ACM, 38(11):39–41. find our card
  22. Eugene W. Myers. 1986. Ano(nd) difference algorithm and its variations. Algorithmica, 1(1):251–266. find
  23. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  24. Piotr Przybyła, Nhung T. H. Nguyen, Matthew Shardlow, Georgios Kontonatsios, and Sophia Ananiadou. 2016. NaCTeM at SemEval-2016 Task 1: Inferring sentence-level semantic similarity from an ensemble of complementary lexical and sentence-level features. In Proceedings of the 10th International Worksho… find
  25. Philip Resnik. 1995. Using information content to evaluate semantic similarity in a taxonomy. In Proceedings of the 14th International Joint Conference on Artificial Intelligence - Volume 1, IJCAI’95, pages 448–453. find our card
  26. Barbara Rychalska, Katarzyna Pakulska, Krystyna Chodorowska, Wojciech Walczak, and Piotr Andruszkiewicz. 2016. Samsung Poland NLP Team at SemEval-2016 Task 1: Necessity for diversity; combining recursive autoencoders, wordnet and ensemble methods to measure semantic similarity. In Proceedings of the… find
  27. Lucia Specia, Kashif Shah, Jose G.C. de Souza, and Trevor Cohn. 2013. QuEst - a translation quality estimation framework. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations. find
  28. Lucia Specia. 2011. Exploiting objective annotations for measuring translation post-editing effort. In 15th Conference of the European Association for Machine Translation, EAMT, pages 73–80, Leuven, Belgium. find
  29. Stack Exchange, Inc. 2016. Stack exchange data dump. https://archive.org/details/ stackexchange. ↗
  30. Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2015. DLS@CU: Sentence similarity from word alignment and semantic vector composition. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), Denver, CO, USA. find
  31. Junfeng Tian and Man Lan. 2016. ECNU at SemEval-2016 Task 1: Leveraging word embedding from macro and micro views to boost performance for semantic textual similarity. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, CA, USA. find