PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification

What it is about

PPDB 1.0 ranked its pivot-extracted paraphrases with a fixed heuristic, and this paper replaces the ranking with a discriminative model. The model is trained on 26,000 human judgments of phrase pairs, sampled evenly across the range of the pivot probability. Its features include the pivot probability, a rarity penalty for how often a pair was seen, and similarity from word embeddings. Its rank correlates better with the human judgments than the pivot probability (Spearman 0.41), the 1.0 heuristic (0.41) or the cosine similarity of word2vec vectors (0.46), with the supervised ranking highest of the four in the paper's figure. The release also adds fine-grained entailment labels and style annotations to each pair, and it has over 100 million paraphrases in total. The lesson is that a cheap pivot probability is a weak ranking, and that a small number of human ratings can fix much of that.

Ideas

I1. A small count-based ranker for the translator: take a few hundred merges that the residual has already judged (kept or undone) and fit a logistic model on features the translator already has, such as alignment counts, the number of verses where the two words meet, and rarity. Store the weights in a BTreeMap and check that two runs agree.
I2. Keep a fixed list of human-judged merges as a held-out check. A new lexicon rule must raise the correlation on that list, measured twice, before it is accepted. The check should be kept apart from the rules that were fitted to it.
I3. Use an entailment direction as a feature of a merge. The paper's example of a noise merge is be with have; a one-way label would keep a merge only where meaning goes one way, which addresses the second open problem directly.

For Mova

The problems are open problems 2 and 5: which merges are noise, and number and person features that a merge should not lose. The ranker is a count-based stand-in for the paper's model; the word embeddings are not for Mova, only the idea that similarity can be counted from context. Summary only, as the licence is non-commercial.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P15-2070.pdf

Summary

The Paraphrase Database (PPDB) holds more than 100 million automatically extracted paraphrases. However, PPDB 1.0 ordered them with a hand-tuned weighting of seven of its 33 features, and it treated every pair as a true equivalent. PPDB 2.0 replaces that heuristic with a supervised ranking. The authors gathered crowd ratings for 26,455 phrase pairs, each scored by five annotators on a 5-point Likert scale. They then trained a ridge regression on the original 33 features plus 176 new ones, including embedding cosine similarity, lexical overlap, WordNet-based and distributional signals.

The learned scores match human judgments far better than the previous rankings. Ranked lists for random Wikipedia phrases also improved, with more good paraphrases near the top. The release adds three annotations to every rule, drawn from companion work:

natural-logic entailment labels
complexity and formality style scores
Multiview LSA embeddings

The new ranking makes the large but noisy database more reliable for downstream tasks. The finer relations and style signals support tasks such as entailment and text simplification.

Key points

Spearman ρ with human ratings was 0.41 for the pivot probability p(e2|e1) and for the PPDB 1.0 heuristic, 0.46 for word2vec, and 0.71 for PPDB 2.0. The PPDB 2.0 figure was cross-validated, holding out 200 phrases per fold.
On full paraphrase lists for 100 Wikipedia phrases, PPDB 2.0 raised P@1 by about 17 points and P@5 by about 9. MRR rose by 9–12 points and AP by 7–9, depending on the goodness threshold (≥3 to ≥4.5).
Entailment labels follow MacCartney's natural logic. They cover forward entailment, reverse entailment, topical relatedness, unrelatedness and exclusion (Pavlick et al., 2015).
Style scores rate complexity and formality (Pavlick and Nenkova, 2015). Embeddings come from MVLSA (Rastogi et al., 2015).
The 26k human-rated pairs are released together with PPDB 2.0 at paraphrase.org.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-2070

@inproceedings{pavlick2015ppdb,
  title = {{PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification}},
  author = {Pavlick, Ellie and Rastogi, Pushpendre and Ganitkevitch, Juri and Durme, Benjamin Van and Callison-Burch, Chris},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-2070},
  url = {https://doi.org/10.3115/v1/p15-2070}
}

References

28 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In ACL, pages 597–604. find our card
Islam Beltagy, Stephen Roller, Gemma Boleda, Katrin Erk, and Raymond J Mooney. 2014. Utexas: Natural language semantics using distributional semantics and probabilistic logic. In SemEval. find
Jonathan Berant, Ido Dagan, and Jacob Goldberger. 2011. Global learning of typed entailment rules. In ACL. find
Rahul Bhagat, Patrick Pantel, Eduard H Hovy, and Marina Rey. 2007. LEDIR: An unsupervised algorithm for learning directionality of inference rules. In EMNLP-CoNLL, pages 161–170. find
Johannes Bjerva, Johan Bos, Rob van der Goot, and Malvina Nissim. 2014. The meaning factory: Formal semantics for recognizing textual entailment and determining semantic similarity. In SemEval. find
Chris Callison-Burch. 2008. Syntactic constraints on paraphrases extracted from parallel corpora. In EMNLP, pages 196–205. find
Tsz Ping Chan, Chris Callison-Burch, and Benjamin Van Durme. 2011. Reranking bilingually extracted paraphrases using monolingual distributional similarity. In GEMS, pages 33–42. find
Paramveer S. Dhillon, Dean Foster, and Lyle Ungar. 2011. Multi-view learning of word embeddings via CCA. In NIPS. find
Manaal Faruqui, Jesse Dodge, Sujay K Jauhar, Chris Dyer, Eduard Hovy, and Noah A Smith. 2015. Retrofitting word vectors to semantic lexicons. In NAACL. find
Juri Ganitkevitch, Chris Callison-Burch, Courtney Napoles, and Benjamin Van Durme. 2011. Learning sentential paraphrases from bilingual parallel corpora for text-to-text generation. In EMNLP, pages 1168–1179. find
Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In NAACL-HLT, pages 758–764, Atlanta, Georgia, June. find our card
Lushan Han, Abhay Kashyap, Tim Finin, James Mayfield, and Jonathan Weese. 2013. Umbc ebiquity-core: Semantic textual similarity systems. In Proceedings of the Second Joint Conference on Lexical and Computational Semantics, volume 1, pages 44– 52. find
Yangfeng Ji and Jacob Eisenstein. 2013. Discriminative improvements to distributional sentence similarity. In EMNLP, pages 891–896. find
Bill MacCartney. 2009. Natural language inference. Ph.D. thesis, Citeseer. find
Prodromos Malakasiotis and Ion Androutsopoulos. 2011. A generate and rank approach to sentence paraphrasing. In EMNLP, pages 96–106. find
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Workshop at ICLR. find
Ellie Pavlick and Ani Nenkova. 2015. Inducing lexical style properties for paraphrase and genre differentiation. In NAACL. find
Ellie Pavlick, Johan Bos, Malvina Nissim, Charley Beller, Benjamin Van Durme, and Chris Callison-Burch. 2015. Adding semantics to data-driven paraphrasing. In ACL. find
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), 12. find
Pushpendre Rastogi, Benjamin Van Durme, and Raman Arora. 2015. Multiview LSA: Representation learning via generalized CCA. In NAACL. find
Rion Snow, Daniel Jurafsky, and Andrew Y Ng. 2004. Learning syntactic patterns for automatic hypernym discovery. In NIPS. Md. find
Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014a. Back to basics for monolingual alignment: Exploiting word similarity and contextual evidence. Transactions of the Association for Computational Linguistics, 2:219–230. find
Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014b. Dls@cu: Sentence similarity from word alignment. In SemEval. find
Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In COLING. find our card
Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. TACL. find our card
Xuchen Yao, Benjamin Van Durme, Chris Callison-Burch, and Peter Clark. 2013. Semi-markov phrase-based monolingual alignment. In EMNLP, pages 590–600. find
Mo Yu and Mark Dredze. 2014. Improving lexical embeddings with semantic knowledge. In ACL, volume 2, pages 545–550. find
Shiqi Zhao, Cheng Niu, Ming Zhou, Ting Liu, and Sheng Li. 2008. Combining multiple resources to improve SMT-based paraphrasing model. In ACL. find