Elamur Research⌕
☰
home / papers / 2017 / A Continuously Growing Dataset of Sentential Paraphrases

A Continuously Growing Dataset of Sentential Paraphrases

Summary

Paraphrase research suffers from a shortage of parallel corpora. Earlier resources such as MSRP and PIT-2015 also depended on a classifier or on manual topic selection to pick candidate pairs before annotation. Lan et al. propose a simpler candidate source: tweets that link to the same URL, mostly news articles posted by 22 English news accounts. After rule-based removal of manual retweets, sentences sharing a link were labeled on Amazon Mechanical Turk. The result is the Twitter News URL Corpus of 51,524 human-labeled sentence pairs, the largest such resource at the time. It includes realistic non-paraphrases and multiple references per source sentence, and the authors use it for the first cross-domain benchmark of paraphrase identification. Because the URL signal never stops producing candidates, the pipeline can automatically collect over 30,000 new sentential paraphrases each month at about 70% precision. Word alignment over this data yields current phrasal paraphrases, such as slang and name variants, that rarely appear in PPDB. This matters because it turns paraphrase collection into an ongoing, low-cost process that covers both formal and informal language.

Key points

  • Corpus size and splits: The gold set has 42,200 training tweets from 4,272 URLs (collected Oct–Nov 2016) and 9,324 test tweets from 915 URLs (collected Jan 2017).
  • Annotation setup: Each question showed one original sentence and 10 candidates. Six crowd workers answered it for $0.03 each, and labels were aggregated by majority vote.
  • Annotation quality: Agreement between an expert and the crowd majority on 300 pairs reached Cohen's kappa 0.739. Answers from workers with kappa below 0.4 were republished to the top 528 of 876 workers.
  • Retweet filtering: Removing manual retweets cut the average from about 205 to about 75 tweets per URL. It also raised PINC from 0.615 to 0.760 and lowered Jaccard from 0.364 to 0.252, meaning the remaining pairs are more genuinely reworded.
  • Phrasal paraphrases: Pairs extracted with GIZA++, Jacana and Sultan aligners clearly outperformed the earlier BUCC-2013 Twitter table. With the Sultan aligner, 77.2% of all extracted pairs were rated top quality, versus 36.3% for BUCC-2013. The extracted pairs overlapped very little with PPDB.

Interesting ideas

  • I1. Shared links as a pivot: Grouping texts that point to the same external object narrows the search space without any trained selector. The same trick could gather paraphrase or comparable-text candidates in other languages, for example from Ukrainian news accounts.
  • I2. Choosing a curated source: News accounts almost always attach article URLs, while the general stream is noisy and hard to read. Picking a curated source over the raw firehose can matter more than any clever filtering applied later.
  • I3. Simple filters plus diversity metrics: Near-copies were removed with substring, punctuation-only and page-metadata (title/description) checks. PINC and Jaccard then confirmed that real rewording remained, which is a cheap and verifiable way to clean any crowd-sourced or scraped parallel data.
  • I4. One-versus-many annotation format: Showing one sentence against 10 candidates is cheaper than judging pairs one by one. Scoring each worker's kappa against the majority of the others, then reassigning their items to reliable workers, is a reusable crowd quality loop.
  • I5. Learned ranking from a small labeled set: An SVM regression trained on just 500 human-rated phrase pairs combined language-model, translation, GloVe and Moses scores. It correlated with human judgments better than any single score (ρ 0.572 vs 0.472 for GloVe alone), and the approach could transfer to ranking any automatically mined lexicon.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

109 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 101 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1126

@inproceedings{lan2017continuously,
  title = {{A Continuously Growing Dataset of Sentential Paraphrases}},
  author = {Lan, Wuwei and Qiu, Siyu and He, Hua and Xu, Wei},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1126},
  url = {https://doi.org/10.18653/v1/d17-1126}
}

References

53 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, et al. 2015. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In Proceedings of the 9th international workshop on semantic evaluation (SemEval). find
  2. Maria Antoniak, Eric Bell, and Fei Xia. 2015. Leveraging paraphrase labels to extract synonyms from twitter. In Proceedings of the 28th Florida Artificial Intelligence Research Society Conference (FLAIRS). find
  3. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics 34(4):555–596. ↗
  4. Eytan Bakshy, Jake M Hofman, Winter A Mason, and Duncan J Watts. 2011. Everyone’s an influencer: quantifying influence on twitter. In Proceedings of the fourth ACM international conference on Web search and data mining (WSDM). find
  5. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (ACL). find our card
  6. Regina Barzilay and Noemie Elhadad. 2003. Sentence alignment for monolingual comparable corpora. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  7. Regina Barzilay and Lillian Lee. 2003. Learning to paraphrase: an unsupervised approach using multiple-sequence alignment. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (NAACL-HLT). ↗ our card
  8. Rahul Bhagat and Eduard Hovy. 2013. What is a paraphrase? Computational Linguistics 39(3). ↗ our card
  9. Johannes Bjerva, Johan Bos, Rob Van der Goot, and Malvina Nissim. 2014. The meaning factory: Formal semantics for recognizing textual entailment and determining semantic similarity. In Proceedings of the 8th international workshop on semantic evaluation (SemEval). ↗
  10. Steven Burrows, Martin Potthast, and Benno Stein. 2013. Paraphrase acquisition via crowdsourcing and machine learning. ACM Transactions on Intelligent Systems and Technology (TIST) 4(3):43. ↗
  11. Chris Callison-Burch. 2008. Syntactic constraints on paraphrases extracted from parallel corpora. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  12. David L. Chen and William B. Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  13. Trevor Cohn, Chris Callison-Burch, and Mirella Lapata. 2008. Constructing corpora for the development and evaluation of paraphrase systems. Computational Linguistics 34(4):597–614. ↗ our card
  14. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Proceedings of the First international conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment (MLCW). find
  15. Dipanjan Das and Noah A Smith. 2009. Paraphrase identification as probabilistic quasi-synchronous recognition. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing (ACL-IJCNLP). find our card
  16. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP). find our card
  17. William B. Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of the 20th International Conference on Computational Linguistics (COL-ING). find our card
  18. Anthony Fader, Luke S Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question answering. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  19. Manaal Faruqui, Jesse Dodge, Sujay K Jauhar, Chris Dyer, Eduard Hovy, and Noah A Smith. 2015. Retrofitting word vectors to semantic lexicons. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). ↗
  20. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (NAACL-HLT). find our card
  21. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). ↗
  22. Adam Grycner, Saarland Informatics Campus, and Gerhard Weikum. 2016. POLY: Mining relational paraphrases from multilingual sentences. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  23. Weiwei Guo and Mona Diab. 2012. Modeling sentences in the latent space. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL). ↗
  24. Weiwei Guo, Wei Liu, and Mona Diab. 2014. Fast tweet retrieval with compact binary codes. In Proceedings of the 30th International Conference on Computational Linguistics (COLING). find
  25. Chikara Hashimoto, Kentaro Torisawa, Stijn De Saeger, Jun’ichi Kazama, and Sadao Kurohashi. 2011. Extracting paraphrases from definition sentences on the web. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). find
  26. Hua He and Jimmy Lin. 2016. Pairwise word interaction modeling with deep neural networks for semantic similarity measurement. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). find
  27. Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation (WMT). find our card
  28. Hamid Izadinia, Fereshteh Sadeghi, Santosh K Divvala, Hannaneh Hajishirzi, Yejin Choi, and Ali Farhadi. 2015. Segment-phrase table for semantic segmentation, visual entailment and paraphrasing. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). find
  29. P. Jaccard. 1912. The distribution of the flora in the alpine zone. New Phytologist 11(2):37–50. find
  30. Yangfeng Ji and Jacob Eisenstein. 2013. Discriminative improvements to distributional sentence similarity. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  31. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions. find
  32. Tao Li and Vivek Srikumar. 2016. Exploiting sentence similarities for better alignments. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  33. Dekang Lin and Patrick Pantel. 2001. DIRT – Discovery of inference rules from text. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). find
  34. Nitin Madnani and Bonnie J. Dorr. 2010. Generating phrasal and sentential paraphrases: A survey of data-driven methods. Computational Linguistics 36(3). ↗ our card
  35. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval). find our card
  36. Ramtin Mehdizadeh Seraj, Maryam Siahbani, and Anoop Sarkar. 2015. Improving statistical machine translation with a multilingual paraphrase database. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  37. Jasmina Milićević. 2006. A short guide to the meaning-text linguistic theory. Journal of Koralex 8:187–233. find
  38. Ndapandula Nakashole, Gerhard Weikum, and Fabian Suchanek. 2012. PATTY: a taxonomy of relational patterns with semantic types. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  39. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  40. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevich, and Chris Callison-Burch Ben Van Durme. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  41. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  42. Satoshi Sekine. 2006. On-demand information extraction. In Proceedings of the 22nd International Conference on Computational Linguistics (COLING). find
  43. Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014. Back to basics for monolingual alignment: Exploiting word similarity and contextual evidence. Transactions of the Association for Computational Linguistics (TACL) 2:219–230. ↗
  44. Wei Xu, Alan Ritter, Chris Callison-Burch, William B. Dolan, and Yangfeng Ji. 2014. Extracting lexically divergent paraphrases from Twitter. Transactions of the Association for Computational Linguistics (TACL) 2:435–448. ↗
  45. Wei Xu, Alan Ritter, and Ralph Grishman. 2013. Gathering and generating paraphrases from twitter with application to normalization. In Proceedings of the Sixth Workshop on Building and Using Comparable Corpora (BUCC). find
  46. Xuchen Yao, Benjamin Van Durme, Chris Callison-Burch, and Peter Clark. 2013. A lightweight and high performance monolingual word aligner. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL). find
  47. Fabio Massimo Zanzotto, Marco Pennacchiotti, and Kostas Tsioutsiouliklis. 2011. Linguistic redundancy in Twitter. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  48. Congle Zhang, Stephen Soderland, and Daniel S Weld. 2015. Exploiting parallel news streams for unsupervised event extraction. Transactions of the Association for Computational Linguistics (TACL) 3:117– 129. ↗
  49. Kristina Toutanova, Chris Brockett, Ke M. Tran, and Saleema Amershi. 2016. A dataset and evaluation metrics for abstractive compression of sentences and short paragraphs. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  50. Soroush Vosoughi and Deb Roy. 2016. A semi-automatic method for efficient detection of stories on social media. In Tenth International AAAI Conference on Web and Social Media (ICWSM). ↗
  51. Ling Wang, Chris Dyer, Alan W. Black, and Isabel Trancoso. 2013. Paraphrasing 4 microblog normalization. In Proceedings of the Conference on Empirical Methods on Natural Language Processing (EMNLP). find
  52. J. Wieting, M. Bansal, K. Gimpel, K. Livescu, and D. Roth. 2015. From paraphrase database to compositional paraphrase model and back. Transactions of the Association for Computational Linguistics (TACL) 3:345–358. ↗ our card
  53. Wei Xu, Chris Callison-Burch, and William B. Dolan. 2015. SemEval-2015 Task 1: Paraphrase and semantic similarity in Twitter (PIT). In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval). find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/lan-2017-continuously-growing-dataset-sentential-paraphrases/