Elamur Research
☰
home / papers / 2014 / SemEval-2014 Task 10: Multilingual Semantic Textual Similarity

SemEval-2014 Task 10: Multilingual Semantic Textual Similarity

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/S14-2010.pdf

Summary

This shared task benchmarks systems that score how close in meaning two short texts are, on a graded scale rather than as a yes/no judgment. The 2014 edition added Spanish alongside English for the first time. English systems were tested on six sets. Two continued earlier genres: news headlines (HDL) and OntoNotes–WordNet gloss pairs (OnWN). Four were new: Images, Deft-forum, Deft-news and Tweet-news. Earlier task data from 2012–2013 could be used for training.

The Spanish data came from Spanish Wikipedia and newswire, with only a small labeled trial set available. Annotation was crowdsourced, mostly through Amazon Mechanical Turk, and systems were ranked by Pearson correlation with the human scores.

English drew 15 teams and 38 runs, and Spanish drew 9 teams and 22 runs. The top weighted mean was 76.1% for English and 80.7% for Spanish. The authors attribute the higher Spanish score to its cleaner, more formal text.

Many Spanish entries simply translated the input into English. The task matters because it extends graded similarity evaluation beyond a single language and across noisier genres.

Key points

  • English test sets (pairs):
    • HDL: 750
    • OnWN: 750
    • Images (PASCAL VOC-2008 subset): 750
    • Tweet-news: 750
    • Deft-forum: 450
    • Deft-news: 300
  • Spanish setup: participants received just 65 labeled trial pairs. The Spanish scale tops out at 4, because the English levels 3 and 4 were merged; English uses 0–5.
  • AMT annotation:
    • Five labels per pair, restricted to Master-qualified workers.
    • 105 hidden gold pairs used to filter out weak annotators.
    • CrowdFlower was tried but gave far worse agreement (22.8% vs. 73.6% on AMT).
  • Difficulty varied by genre:
    • Inter-tagger correlation was 79.4% on HDL but only 58.6% on Deft-forum.
    • The best system reached just 50.8% on Deft-forum versus 85.9% on OnWN.
  • Spanish approaches:
    • Most systems relied on machine translation.
    • Others used transfer learning (Rios), referential translation machines (RTM-DCU), or overlap-based metrics with the Spanish WordNet.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

293 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 272 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/s14-2010

@inproceedings{agirre2014semeval,
  title = {{SemEval-2014 Task 10: Multilingual Semantic Textual Similarity}},
  author = {Agirre, Eneko and Banea, Carmen and Cardie, Claire and Cer, Daniel and Diab, Mona and Gonzalez-Agirre, Aitor and Guo, Weiwei and Mihalcea, Rada and Rigau, German and Wiebe, Janyce},
  booktitle = {Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)},
  year = {2014},
  doi = {10.3115/v1/s14-2010},
  url = {https://doi.org/10.3115/v1/s14-2010}
}

References

31 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre and Enrique Amigó. In prep. Exploring evaluation measures for semantic textual similarity. In Unpublished manuscript. find
  2. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceed… find our card
  3. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 Shared Task: Semantic textual similarity, including a pilot on typed-similarity. In The Second Joint Conference on Lexical and Computational Semantics (*SEM 2013), pages 32–43. find our card
  4. Clive Best, Erik van der Goot, Ken Blackler, Tefilo Garcia, and David Horby. 2005. Europe media monitor - system description. In EUR Report 22173-En, Ispra, Italy. find
  5. Ergun Biçici and Andy Way. 2014. RTM-DCU: Referential translation machines for semantic similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
  6. Davide Buscaldi, Jorge J. Garcia Flores, Joseph Le Roux, Nadi Tomeh, and Belem Priego Sanchez. 2014. LIPN: Introducing a new geographical context similarity measure and a statistical similarity measure based on the Bhattacharyya coefficient. In Proceedings of the 8th International Workshop on Semant… find
  7. Alexander Chavez, Hector Davila, Yoan Gutierrez, Antonio Fernandez-Orquin, Andrés Montoyo, and Rafael Munoz. 2014. UMCC DLSI SemSim: Multilingual system for measuring semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland… find
  8. Christiane Fellbaum. 1998. WordNet - An electronic lexical database. MIT Press. find
  9. Weiwei Guo, Hao Li, Heng Ji, and Mona Diab. 2013. Linking tweets to news: A framework to enrich online short text data in social media. In Proceedings of the 51th Annual Meeting of the Association for Computational Linguistics, pages 239–249. find
  10. Anubhav Gupta. 2014. TeamZ: Measuring semantic textual similarity for Spanish using an overlap-based approach. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
  11. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 57–60. find
  12. Sergio Jimenez, George Dueñas, Julia Baquero, and Alexander Gelbukh. 2014. UNAL-NLP: Combining soft cardinality features for semantic textual similarity, relatedness and entailment. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
  13. Abhay Kashyap, Lushan Han, Roberto Yus, Jennifer Sleeman, Taneeya Satyapanich, Sunil Gandhi, and Tim Finin. 2014. Meerkat Mafia: Multilingual and cross-level semantic textual similarity systems. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
  14. Thomas K. Landauer, Darrell Laham, Bob Rehder, and M. E. Schreiner. 1997. How well can passage meaning be derived without using word order? A comparison of latent semantic analysis and humans. Cognitive Science. find
  15. Claudia Leacock and Martin Chodorow. 1998. Combining local context and WordNet similarity for word sense identification. In WordNet: An Electronic Lexical Database, pages 305–332. find
  16. Dekang Lin. 1998. An information-theoretic definition of similarity. In Proceedings of the Fifteenth International Conference on Machine Learning, pages 296–304, Madison, Wisconsin. find
  17. André Lynum, Partha Pakray, Björn Gambäck, and Sergio Jimenez. 2014. NTNU: Measuring semantic similarity with sublexical feature representations and soft cardinality. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
  18. Juan Martinez-Romo, Lourdes Araujo, Javier Borge-Holthoefer, Alex Arenas, José A. Capitán, and José A. Cuesta. 2011. Disentangling categorical relationships through a graph of co-occurrences. Phys. Rev. E, 84:046108, Oct. find
  19. John P. McCrae, Philipp Cimiano, and Roman Klinger. 2013. Orthonormal explicit topic analysis for cross-lingual document matching. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1732–1740, Seattle, Washington, USA. find
  20. Francesco Pozzi, Tiziana Di Matteo, and Tomaso Aste. 2012. Exponential smoothing weighted correlations. The European Physical Journal B, 85(6). find
  21. William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. 2002. Numerical recipes: The art of scientific computing V 2.10 with Linux or single-screen license. Cambridge University Press. find
  22. Thomas Proisi, Stefan Evert, Paul Greiner, and Besim Kabashi. 2014. SemantiKLUE: Robust semantic similarity at multiple levels using maximum weight matching. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
  23. Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. 2010. Collecting image annotations using Amazon’s Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, CSLDAMT ’10, pages 139–147, Stroudsburg, PA, USA. find
  24. Miguel Rios. 2014. UoW: Multi-task learning Gaussian process for semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
  25. Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, pages 207–218. find our card
  26. Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014a. Back to basics for monolingual alignment: Exploiting word similarity and contextual evidence. Transactions of the Association for Computational Linguistics, 2:219–230. find
  27. Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014b. DLS@CU: Sentence similarity from word aligment. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
  28. Darnes Vilariño, David Pinto, Saúl León, Mireya Tovar, and Beatriz Beltrán. 2014. BUAP: Evaluating features for multilingual and cross-level semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
  29. Ngoc Phuoc An Vo, Tommaso Caselli, and Octavian Popescu. 2014. FBK-TR: Applying SVM with multiple linguistic features for cross-level semantic similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland. find
  30. Frane Šarić, Goran Glavaš, Mladen Karan, Jan Šnajder, and Bojana Dalbelo Bašić. 2012. Takelab: Systems for measuring semantic text similarity. In Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 441–448, Montréal, Canada, 7-8 June. find
  31. Zhibiao Wu and Martha Palmer. 1994. Verbs semantics and lexical selection. In Proceedings of the 32nd annual meeting on Association for Computational Linguistics, pages 133–138, Las Cruces, New Mexico. find