SemEval-2014 Task 10: Multilingual Semantic Textual Similarity
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/S14-2010.pdf
Summary
This shared task benchmarks systems that score how close in meaning two short texts are, on a graded scale rather than as a yes/no judgment. The 2014 edition added Spanish alongside English for the first time. English systems were tested on six sets. Two continued earlier genres: news headlines (HDL) and OntoNotes–WordNet gloss pairs (OnWN). Four were new: Images, Deft-forum, Deft-news and Tweet-news. Earlier task data from 2012–2013 could be used for training.
The Spanish data came from Spanish Wikipedia and newswire, with only a small labeled trial set available. Annotation was crowdsourced, mostly through Amazon Mechanical Turk, and systems were ranked by Pearson correlation with the human scores.
English drew 15 teams and 38 runs, and Spanish drew 9 teams and 22 runs. The top weighted mean was 76.1% for English and 80.7% for Spanish. The authors attribute the higher Spanish score to its cleaner, more formal text.
Many Spanish entries simply translated the input into English. The task matters because it extends graded similarity evaluation beyond a single language and across noisier genres.
Key points
- English test sets (pairs):
- HDL: 750
- OnWN: 750
- Images (PASCAL VOC-2008 subset): 750
- Tweet-news: 750
- Deft-forum: 450
- Deft-news: 300
- Spanish setup: participants received just 65 labeled trial pairs. The Spanish scale tops out at 4, because the English levels 3 and 4 were merged; English uses 0–5.
- AMT annotation:
- Five labels per pair, restricted to Master-qualified workers.
- 105 hidden gold pairs used to filter out weak annotators.
- CrowdFlower was tried but gave far worse agreement (22.8% vs. 73.6% on AMT).
- Difficulty varied by genre:
- Inter-tagger correlation was 79.4% on HDL but only 58.6% on Deft-forum.
- The best system reached just 50.8% on Deft-forum versus 85.9% on OnWN.
- Spanish approaches:
- Most systems relied on machine translation.
- Others used transfer learning (Rios), referential translation machines (RTM-DCU), or overlap-based metrics with the Spanish WordNet.
Cited by
- SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation
- LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
- SentEval: An Evaluation Toolkit for Universal Sentence Representations
- SimCSE: Simple Contrastive Learning of Sentence Embeddings
- DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
- Generating Sentences by Editing Prototypes
- Learning Distributed Representations of Sentences from Unlabelled Data
- ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translation
- On the Sentence Embeddings from Pre-trained Language Models
- An efficient framework for learning sentence representations
- Learned in Translation: Contextualized Word Vectors
- MTEB: Massive Text Embedding Benchmark
- Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models
- Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features
- Generating Datasets with Pretrained Language Models
- Towards Universal Paraphrastic Sentence Embeddings
- ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations
- Beyond BLEU: Training Neural Machine Translation with Semantic Similarity
- C-Pack: Packed Resources For General Chinese Embeddings
- ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Cite
@inproceedings{agirre2014semeval,
title = {{SemEval-2014 Task 10: Multilingual Semantic Textual Similarity}},
author = {Agirre, Eneko and Banea, Carmen and Cardie, Claire and Cer, Daniel and Diab, Mona and Gonzalez-Agirre, Aitor and Guo, Weiwei and Mihalcea, Rada and Rigau, German and Wiebe, Janyce},
booktitle = {Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)},
year = {2014},
doi = {10.3115/v1/s14-2010},
url = {https://doi.org/10.3115/v1/s14-2010}
}References
- Eneko Agirre and Enrique Amigó. In prep. Exploring evaluation measures for semantic textual similarity. In Unpublished manuscript. find
- Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceed… find our card
- Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 Shared Task: Semantic textual similarity, including a pilot on typed-similarity. In The Second Joint Conference on Lexical and Computational Semantics (*SEM 2013), pages 32–43. find our card
- Clive Best, Erik van der Goot, Ken Blackler, Tefilo Garcia, and David Horby. 2005. Europe media monitor - system description. In EUR Report 22173-En, Ispra, Italy. find
- Ergun Biçici and Andy Way. 2014. RTM-DCU: Referential translation machines for semantic similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
- Davide Buscaldi, Jorge J. Garcia Flores, Joseph Le Roux, Nadi Tomeh, and Belem Priego Sanchez. 2014. LIPN: Introducing a new geographical context similarity measure and a statistical similarity measure based on the Bhattacharyya coefficient. In Proceedings of the 8th International Workshop on Semant… find
- Alexander Chavez, Hector Davila, Yoan Gutierrez, Antonio Fernandez-Orquin, Andrés Montoyo, and Rafael Munoz. 2014. UMCC DLSI SemSim: Multilingual system for measuring semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland… find
- Christiane Fellbaum. 1998. WordNet - An electronic lexical database. MIT Press. find
- Weiwei Guo, Hao Li, Heng Ji, and Mona Diab. 2013. Linking tweets to news: A framework to enrich online short text data in social media. In Proceedings of the 51th Annual Meeting of the Association for Computational Linguistics, pages 239–249. find
- Anubhav Gupta. 2014. TeamZ: Measuring semantic textual similarity for Spanish using an overlap-based approach. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
- Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 57–60. find
- Sergio Jimenez, George Dueñas, Julia Baquero, and Alexander Gelbukh. 2014. UNAL-NLP: Combining soft cardinality features for semantic textual similarity, relatedness and entailment. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
- Abhay Kashyap, Lushan Han, Roberto Yus, Jennifer Sleeman, Taneeya Satyapanich, Sunil Gandhi, and Tim Finin. 2014. Meerkat Mafia: Multilingual and cross-level semantic textual similarity systems. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
- Thomas K. Landauer, Darrell Laham, Bob Rehder, and M. E. Schreiner. 1997. How well can passage meaning be derived without using word order? A comparison of latent semantic analysis and humans. Cognitive Science. find
- Claudia Leacock and Martin Chodorow. 1998. Combining local context and WordNet similarity for word sense identification. In WordNet: An Electronic Lexical Database, pages 305–332. find
- Dekang Lin. 1998. An information-theoretic definition of similarity. In Proceedings of the Fifteenth International Conference on Machine Learning, pages 296–304, Madison, Wisconsin. find
- André Lynum, Partha Pakray, Björn Gambäck, and Sergio Jimenez. 2014. NTNU: Measuring semantic similarity with sublexical feature representations and soft cardinality. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
- Juan Martinez-Romo, Lourdes Araujo, Javier Borge-Holthoefer, Alex Arenas, José A. Capitán, and José A. Cuesta. 2011. Disentangling categorical relationships through a graph of co-occurrences. Phys. Rev. E, 84:046108, Oct. find
- John P. McCrae, Philipp Cimiano, and Roman Klinger. 2013. Orthonormal explicit topic analysis for cross-lingual document matching. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1732–1740, Seattle, Washington, USA. find
- Francesco Pozzi, Tiziana Di Matteo, and Tomaso Aste. 2012. Exponential smoothing weighted correlations. The European Physical Journal B, 85(6). find
- William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. 2002. Numerical recipes: The art of scientific computing V 2.10 with Linux or single-screen license. Cambridge University Press. find
- Thomas Proisi, Stefan Evert, Paul Greiner, and Besim Kabashi. 2014. SemantiKLUE: Robust semantic similarity at multiple levels using maximum weight matching. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
- Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. 2010. Collecting image annotations using Amazon’s Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, CSLDAMT ’10, pages 139–147, Stroudsburg, PA, USA. find
- Miguel Rios. 2014. UoW: Multi-task learning Gaussian process for semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
- Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, pages 207–218. find our card
- Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014a. Back to basics for monolingual alignment: Exploiting word similarity and contextual evidence. Transactions of the Association for Computational Linguistics, 2:219–230. find
- Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014b. DLS@CU: Sentence similarity from word aligment. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
- Darnes Vilariño, David Pinto, Saúl León, Mireya Tovar, and Beatriz Beltrán. 2014. BUAP: Evaluating features for multilingual and cross-level semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
- Ngoc Phuoc An Vo, Tommaso Caselli, and Octavian Popescu. 2014. FBK-TR: Applying SVM with multiple linguistic features for cross-level semantic similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
- Frane Šarić, Goran Glavaš, Mladen Karan, Jan Šnajder, and Bojana Dalbelo Bašić. 2012. Takelab: Systems for measuring semantic text similarity. In Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 441–448, Montréal, Canada, 7-8 June. find
- Zhibiao Wu and Martha Palmer. 1994. Verbs semantics and lexical selection. In Proceedings of the 32nd annual meeting on Association for Computational Linguistics, pages 133–138, Las Cruces, New Mexico. find