Elamur Research⌕
☰
home / papers / 2015 / SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT)

SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT)

Summary

This shared task asks how well systems can judge whether two tweets mean the same thing, given that Twitter text is short, noisy and full of lexical variation. The organizers reused their Twitter Paraphrase Corpus (PIT-2015) of 18,762 sentence pairs. The pairs come from trending topics and were labeled by five Mechanical Turk workers each. The test pairs also received expert labels on a 0–5 similarity scale. Participants faced two subtasks on the same data. In Paraphrase Identification (PI) they gave a binary decision, scored by F1. In Semantic Textual Similarity (SS) they gave a score from 0 to 1, evaluated by Pearson correlation. To make annotation affordable, candidate sentences were pre-filtered with a SumBasic-style salience score, which doubled the share of paraphrases. Nineteen teams took part. The top systems reached 0.674 F1 and 0.619 Pearson, ahead of a logistic regression baseline but well below the more than 0.80 typical on edited text. The released corpus, baselines and scripts give a realistic benchmark for paraphrase work on informal language.

Key points

  • Corpus splits: train 13,063 pairs, dev 4,727 pairs, test 972 pairs. The test set holds 18.0% paraphrases and 13.8% debatable cases. About 70% of pairs are non-paraphrases, compared with 34% in the Microsoft Paraphrase Corpus.
  • Participation and results: 19 teams submitted 36 PI runs and 26 SS runs. The best scores were 0.674 F1 (PI) and 0.619 Pearson (SS), against a logistic regression baseline of 0.589 F1 and 0.511 Pearson.
  • Annotation format: each item showed workers one original sentence and 10 candidates from the same topic. Five workers answered each item, at $0.02 per worker per item and about 30–45 seconds per answer.
  • Annotation reliability: crowd and expert labels correlated at Pearson 0.735. Against expert labels, the combined crowd judgement reached F1 0.823 (precision 0.752, recall 0.908).
  • Filtering effect: without filtering, random same-topic pairs were paraphrases under 8% of the time. In a 20-topic trial of 2,000 pairs, SumBasic filtering raised the paraphrase count from 152 to 329.

Interesting ideas

  • I1. Salience filtering to find paraphrases: the authors rank sentences by the average probability of their words within a topic. They draw originals from the top 10% and candidates from the top 50%. This borrows the summarization insight that salient sentences tend to repeat each other, and it is a cheap way to raise yield when collecting rare positives for any new corpus.
  • I2. One-against-ten annotation: instead of rating isolated pairs, workers choose matches for one sentence from a list of ten. This makes binary crowd labeling fast and cheap. Aggregating five such votes came close to expert graded judgements.
  • I3. Dropping medium-score cases from binary scoring: pairs that annotators found borderline are excluded from F1 in the binary task but kept for the graded task. This keeps the classification metric from being dominated by inherently ambiguous items.
  • I4. One dataset, two evaluation views: graded similarity systems are also scored by the best F1 along their precision-recall curve. This allows direct comparison with binary classifiers and helps show how the PI and SS problems relate.
  • I5. A test set built without topic selection: the test data keeps a natural label distribution, while training data is enriched with positives. This guards against benchmarks that overstate performance, a bias the authors note for corpora lacking natural non-paraphrases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

90 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 84 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s15-2001

@inproceedings{xu2015semeval,
  title = {{SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT)}},
  author = {Xu, Wei and Callison-Burch, Chris and Dolan, Bill},
  booktitle = {Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)},
  year = {2015},
  doi = {10.18653/v1/s15-2001},
  url = {https://doi.org/10.18653/v1/s15-2001}
}

References

50 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agirre, E., Diab, M., Cer, D., and Gonzalez-Agirre, A. (2012). Semeval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of SemEval. find our card
  2. Artstein, R. and Poesio, M. (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4). ↗
  3. Bertero, D. and Fung, P. (2015). HLTC-HKUST: A neural network paraphrase classifier using translation metrics, semantic roles and lexical similarity features. In Proceedings of SemEval. find
  4. Biçici, E. and Way, A. (2014). RTM-DCU: Referential translation machines for semantic similarity. In Proceedings of SemEval. find
  5. Biçici, E. and Yuret, D. (2014). Optimizing instance selection for statistical machine translation with https://github.com/cocoxu/SemEval-PIT2015 feature decay algorithms. IEEE/ACM Transactions On Audio, Speech, and Language Processing (TASLP). ↗
  6. Bicici, E. (2015). RTM-DCU: Predicting semantic similarity with referential translation machines. In Proceedings of SemEval. find
  7. Bjerva, J., Bos, J., van der Goot, R., and Nissim, M. (2014). The meaning factory: Formal semantics for recognizing textual entailment and determining semantic similarity. In Proceedings of SemEval. ↗
  8. Blacoe, W. and Lapata, M. (2012). A comparison of vector-based representations for semantic composition. In Proceedings of EMNLP-CoNLL. find
  9. Chen, D. L. and Dolan, W. B. (2011). Collecting highly parallel data for paraphrase evaluation. In Proceedings of ACL. find our card
  10. Das, D. and Smith, N. A. (2009). Paraphrase identification as probabilistic quasi-synchronous recognition. In Proceedings of ACL-IJCNLP. find our card
  11. Derczynski, L., Ritter, A., Clark, S., and Bontcheva, K. (2013). Twitter part-of-speech tagging for all: Overcoming sparse and noisy data. In Proceedings of RANLP. find
  12. Dolan, B., Quirk, C., and Brockett, C. (2004). Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of COLING. find our card
  13. Eyecioglu, A. and Keller, B. (2015). ASOBEK: Twitter paraphrase identification with simple overlap features and SVMs. In Proceedings of SemEval. ↗
  14. Fellbaum, C. (2010). WordNet. In Theory and Applications of Ontology: Computer Applications. Springer. find
  15. Fernando, S. and Stevenson, M. (2008). A semantic similarity approach to paraphrase detection. Computational Linguistics UK (CLUK) 11th Annual Research Colloquium. find
  16. Francis, W. N. and Kucera, H. (1979). Brown corpus manual. Technical report, Brown University. Department of Linguistics. find
  17. Glavaš, G., Karan, M., Šnajder, J., Bašić, B. D., Vulić, I., and Moens, M.-F. (2015). TKLBLIIR: Detecting Twitter paraphrases with TweetingJay. In Proceedings of SemEval. find
  18. Guo, W. and Diab, M. (2012). Modeling sentences in the latent space. In Proceedings of ACL. ↗
  19. Guo, W., Liu, W., and Diab, M. (2014). Fast tweet retrieval with compact binary codes. In Proceedings of COLING. find
  20. Han, L., Kashyap, A., Finin, T., Mayfield, J., and Weese, J. (2013). UMBC EBIQUITY-CORE: Semantic textual similarity systems. In Proceedings of *SEM. ↗ our card
  21. Hassan, S. and Mihalcea, R. (2011). Semantic relatedness using salient semantic analysis. In Proceedings of AAAI. find
  22. Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., and Weischedel, R. (2006). OntoNotes: the 90% solution. In Proceedings of HLT-NAACL. find our card
  23. Islam, A. and Inkpen, D. (2007). Semantic similarity of short texts. In Proceedings of RANLP. find
  24. Ji, Y. and Eisenstein, J. (2013). Discriminative improvements to distributional sentence similarity. In Proceedings of EMNLP. find
  25. Karampatsis, R.-M. (2015). CDTDS: Predicting paraphrases in Twitter via support vector regression. In Proceedings of SemEval. find
  26. Kornai, A., Makrai, M., Nemeskey, D., and Recski, G. (2015). Extending 4lang using monolingual dictionaries. Unpublished manuscript. find
  27. Madnani, N. and Dorr, B. J. (2010). Generating phrasal and sentential paraphrases: A survey of data-driven methods. Computational Linguistics. ↗ our card
  28. Madnani, N., Tetreault, J., and Chodorow, M. (2012). Re-examining machine translation metrics for paraphrase identification. In Proceedings of NAACL-HLT. find our card
  29. Mihalcea, R., Corley, C., and Strapparava, C. (2006). Corpus-based and knowledge-based measures of text semantic similarity. In Proceedings of AAAI. find
  30. Nenkova, A. and Vanderwende, L. (2005). The impact of frequency on summarization. Technical report, Microsoft Research. MSR-TR-2005-101. find
  31. Ngoc Phuoc An Vo, S. M. and Popescu, O. (2015). FBK-HLT: An application of semantic textual similarity for paraphrase and semantic similarity in Twitter. In Proceedings of SemEval. find
  32. O’Connor, B., Krieger, M., and Ahn, D. (2010). Tweetmotif: Exploratory search and topic summarization for Twitter. In Proceedings of ICWSM. find
  33. Petrović, S., Osborne, M., and Lavrenko, V. (2012). Using paraphrases for improving first story detection in news and twitter. In Proceedings of NAACL-HLT. find
  34. Qiu, L., Kan, M.-Y., and Chua, T.-S. (2006). Paraphrase recognition via dissimilarity significance classification. In Proceedings of EMNLP. find
  35. Ritter, A., Clark, S., and Etzioni, O. (2011). Named entity recognition in tweets: an experimental study. In Proceedings of EMNLP. find
  36. Robbins, H. (1985). Some aspects of the sequential design of experiments. In Herbert Robbins Selected Papers. Springer. ↗
  37. Rus, V., McCarthy, P. M., Lintean, M. C., McNamara, D. S., and Graesser, A. C. (2008). Paraphrase identification with lexico-syntactic graph subsumption. In Proceedings of FLAIRS. find
  38. Satyapanich, T., Gao, H., and Finin, T. (2015). Ebiquity: Paraphrase and semantic similarity in Twitter using skipgrams. In Proceedings of SemEval. find
  39. Socher, R., Huang, E. H., Pennin, J., Manning, C. D., and Ng, A. (2011). Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Proceedings of NIPS. find
  40. Tran-Thanh, L., Stein, S., Rogers, A., and Jennings, N. R. (2012). Efficient crowdsourcing of unknown experts using multi-armed bandits. In Proceedings of ECAI. find
  41. van der Goot, R. and van Noord, G. (2015). ROB: Using semantic meaning to recognize paraphrases. In Proceedings of SemEval. find
  42. Vanderwende, L., Suzuki, H., Brockett, C., and Nenkova, A. (2007). Beyond SumBasic: Task-focused summarization with sentence simplification and lexical expansion. Information Processing & Management, 43. ↗
  43. Wan, S., Dras, M., Dale, R., and Paris, C. (2006). Using dependency-based features to take the para-farce out of paraphrase. In Proceedings of the Australasian Language Technology Workshop. find
  44. Wang, L., Dyer, C., Black, A. W., and Trancoso, I. (2013). Paraphrasing 4 microblog normalization. In Proceedings of EMNLP. find
  45. Xu, W. (2014). Data-Drive Approaches for Paraphrasing Across Language Variations. PhD thesis, Department of Computer Science, New York University. find
  46. Xu, W., Ritter, A., Callison-Burch, C., Dolan, W. B., and Ji, Y. (2014). Extracting lexically divergent paraphrases from Twitter. Transactions of the Association for Computational Linguistics (TACL), 2(1). ↗
  47. Xu, W., Ritter, A., and Grishman, R. (2013). Gathering and generating paraphrases from twitter with application to normalization. In Proceedings of the Sixth Workshop on Building and Using Comparable Corpora. find
  48. Zanzotto, F. M., Pennacchiotti, M., and Tsioutsiouliklis, K. (2011). Linguistic redundancy in twitter. In Proceedings of EMNLP. find
  49. Zarrella, G., Henderson, J., Merkhofer, E. M., and Strickhart, L. (2015). MITRE: Seven systems for semantic similarity in tweets. In Proceedings of SemEval. find
  50. Zhao, J. and Lan, M. (2015). ECNU: Boosting performance for paraphrase and semantic similarity in Twitter by leveraging word embeddings. In Proceedings of SemEval. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2015-semeval-2015-task-1-paraphrase/