Automatically Constructing a Corpus of Sentential Paraphrases
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/I05-5002.pdf
Summary
Research on recognizing and generating paraphrases has lacked a large, public, human-labeled set of sentence-level paraphrase pairs. Existing resources are small, cover narrow domains, or rely on multiple translations of uneven quality. Dolan and Brockett describe how they built the Microsoft Research Paraphrase Corpus (MSRP) from web news stories grouped into topic clusters. They first filtered candidate sentence pairs using heuristics on length, shared words and lexical distance, and kept only articles whose author could be identified. They then applied an SVM classifier (SMO) using string-similarity, morphological, WordNet and Encarta thesaurus features. This classifier was deliberately biased toward over-predicting paraphrases. From its output, 5801 pairs were randomly sampled for human labeling, and 67% were judged semantically equivalent. The authors explain why they loosened the equivalence criterion and acknowledge the corpus's biases. They also propose that research groups pool their corpora into a shared "virtual" resource. MSRP later became a standard benchmark for paraphrase identification.
Key points
- The source data was 13,127,938 sentence pairs from 32,408 news clusters collected over two years. Heuristics, including at least three shared words, 5–40 words per sentence and a length ratio of at least 66.6%, reduced this to 49,375 pairs from author-identified articles.
- The SVM (SMO, Platt 1999) was trained on 10,000 separately annotated pairs (2968 positive, 7032 negative). It selected 20,574 likely paraphrases, and 5801 of these were sampled for annotation.
- Two judges labeled each pair as a paraphrase or not, and a third judge settled disagreements. The final count was 3900 positive pairs (67%).
- The idea of "semantic equivalence" was relaxed to allow pairs that mostly entail each other, with minor differences in content. Pairs that were near-identical at the string level were filtered out.
- Known limitations include high lexical overlap (mean 0.7), no paraphrases with little word overlap, and a small size. Because of these, the authors see MSRP mainly as an evaluation set rather than training data. Planned extensions include better positive/negative balance, word-aligned data evaluated with AER, and game-based collection from web volunteers.
Cited by
- Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
- Muppet: Massive Multi-task Representations with Pre-Finetuning
- UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- Reducing Transformer Depth on Demand with Structured Dropout
- Making Pre-trained Language Models Better Few-shot Learners
- SimCSE: Simple Contrastive Learning of Sentence Embeddings
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
- Editing Models with Task Arithmetic
- T-STAR: Truthful Style Transfer using AMR Graph as Intermediate Representation
- SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
- SpanBERT: Improving Pre-training by Representing and Predicting Spans
- Revealing the Dark Secrets of BERT
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
- Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges
- Generating Phrasal and Sentential Paraphrases: A Survey of Data-Driven Methods
- Merging Models with Fisher-Weighted Averaging
- Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
- Are Sixteen Heads Really Better than One?
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
- Weisfeiler-Leman in the Bamboo: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity
- AdapterFusion: Non-Destructive Task Composition for Transfer Learning
- Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
- Combating Adversarial Misspellings with Robust Word Recognition
- Multitask Prompted Training Enables Zero-Shot Task Generalization
- Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
- MPNet: Masked and Permuted Pre-training for Language Understanding
- Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
- Patient Knowledge Distillation for BERT Model Compression
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
- Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
- SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- Lost in Translationese? Reducing Translation Effect Using Abstract Meaning Representation
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
- ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
- LinkBERT: Pretraining Language Models with Document Links
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
- Revisiting Few-sample BERT Fine-tuning
- mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval
- FreeLB: Enhanced Adversarial Training for Natural Language Understanding
- PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
- VALUE: Understanding Dialect Disparity in NLU
Cite
@inproceedings{dolan2005automatically,
title = {{Automatically Constructing a Corpus of Sentential Paraphrases}},
author = {Dolan, William B. and Brockett, Chris},
booktitle = {Proceedings of the Third International Workshop on Paraphrasing (IWP2005)},
year = {2005},
url = {https://aclanthology.org/I05-5002/}
}References
- Regina Barzilay and Katherine. R. McKeown. 2001. Extracting Paraphrases from a parallel corpus. In Proceedings of the ACL/EACL. find our card
- Regina Barzilay and Lillian Lee. 2003. Learning to Paraphrase; an unsupervised approach using multiple-sequence alignment. In Proceedings of HLT/NAACL 2003. find our card
- Chris Brockett and William B. Dolan. 2005. Support Vector Machines for Paraphrase Identification and Corpus Construction. In Proceedings of The Third International Workshop on Paraphrasing (IWP2005), Jeju, Republic of Korea. find
- Chris Brockett and William B. Dolan. 2005. Echo Chamber: A Game for Eliciting a Colloquial Paraphrase Corpus. AAAI 2005 Spring Symposium, Knowledge Collection from Volunteer Contributors (KCVC05). Stanford, CA. March 21-23, 2005. find
- P. Brown, S. A. Della Pietra, V.J. Della Pietra and R. L. Mercer. 1993. The Mathematics of Statistical Machine Translation. Computational Linguistics, Vol. 19(2): 263-311. find
- John Burger and Lisa Ferro. 2005. Generating an Entailment Corpus from News Headlines. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment. pp 49-54. find
- Timothy Chklovski. 2005 1001 Paraphrases: Incent-ing Responsible Contributions in Collecting Paraphrases from Volunteers. AAAI 2005 Spring Symposium, Knowledge Collection from Volunteer Contributors (KCVC05). Stanford, CA. March 21-23, 2005. find
- Courtney Courley and Rada Mihalcea. 2005. Measuring the Semantic Similarity of Texts. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment. Pp 13-18. Hal Daumé III and Daniel Marcu. (forthcoming) Induction of Word and Phrase Alignments for Automatic Document Summarization. To appear in Computational Linguistics. William. find
- B. Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources. Proceedings of COLING 2004, Geneva, Switzerland. find our card
- Susan Dumais. 1998. Using SVMs for Text Categorization. find
- IEEE Intelligent Systems, Jul.-Aug. 1998: 21-23 find
- Susan Dumais, John Platt, David Heckerman, Mehran Sahami. 1998. Inductive learning algorithms and representations for text categorization. In Proceedings of the Seventh International Conference on Information and Knowledge Management. find
- Christiane Fellbaum (ed.). 1998. WordNet: An Electronic Lexical Database. The MIT Press. find
- Andrew Finch, Taro Watanabe, Yasuhiro Akiba and Eiichiro Sumita. 2004. Paraphrasing as Machine Translation. Journal of Natural Language Processing, 11(5), pp 87-111. find
- Pascale Fung and Percy Cheung. 2004. Multi-level Bootstrapping for Extracting Parallel Sentences from a Quasi-Comparable Corpus. In Proceedings of Coling 2004, 1051-1057. find
- Shudong Huang, David Graff, and George Doddington (eds.) 2002. Multiple-Translation Chinese Corpus. Linguistic Data Consortium. find
- Thorsten Joachims. 2002. Learning to Classify Text Using Support Vector Machines: Methods, Theory, and Algorithms. Kluwer Academic Publishers, Boston/Dordrecht/London. find
- V. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physice-Doklady 10: 707-710. Microsoft Research Paraphrase Corpus. http://research.microsoft.com/research/downloads/default.aspx ↗
- Franz Joseph Och and H. Ney. 2000. Improved Statistical Alignment Models. In Proceedings of the 38th Annual Meeting of the ACL, Hong Kong, China, pp 440-447. find
- Franz Joseph Och and Hermann Ney. 2003. A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics, 29 (1): 19-52. find
- Bo Pang, Kevin Knight, and Daniel Marcu. 2003. Syntax-based Alignment of Multiple Translations: Extracting Paraphrases and Generating New Sentences. In Proceedings of NAACL-HLT. find our card
- John C. Platt. 1999. Fast Training of Support Vector Machines Using Sequential Minimal Optimization. In Bernhard Schölkopf, Christopher J. C. Burges and Alexander J. Smola (eds.). 1999. Advances in Kernel Methods: Support Vector Learning. The MIT Press, Cambridge, MA. 185-208. find
- Chris Quirk, Chris Brockett, and William B. Dolan. 2004. Monolingual Machine Translation for Paraphrase Generation, In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, 25-26 July 2004, Barcelona Spain, pp. 142-149. find our card
- Kathy Rooney (ed.) 2001. Encarta Thesaurus. Bloomsbury Publishing. find
- Satoshi Shirai, Kazuhide Yamamoto, Francis Bond & Hozumi Tanaka. 2002. Towards a thesaurus of predicates. In Proceedings of LREC 2002 (Third International Conference on Language Resources and Evaluation), (May 29-31, 2002). Vol.6, pp. 1965-1972. find
- Vladimir N. Vapnik. 1995. The Nature of Statistical Learning Theory. Springer-Verlag, New York. find
- Julie Weeds, David Weir and Bill Keller. 2005. The Distributional Similarity of Subparses. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment. pp 7-12. find
- Dekai Wu. 2005. Recognizing Paraphrases and Textual Entailment using Inversion Transduction Grammars. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment. Pp 25-30. find
- Yujie Zhang and Kazuhide Yamamoto. 2002 Paraphrasing of Chinese Utterances. Proceedings of Coling 2002, pp.1163-1169. find