Elamur Research
☰
home / papers / 2015 / From Paraphrase Database to Compositional Paraphrase Model and Back

From Paraphrase Database to Compositional Paraphrase Model and Back

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/Q15-1025.pdf

Summary

PPDB is a large, automatically built collection of paraphrase pairs, but it has three weaknesses. Two phrases can only be compared if both appear in it. Its size grows with the source data. Its confidence scores come from hand-weighted features of uncertain quality. Wieting et al. use PPDB as training data for compact parametric models that can score any pair of phrases. First, they fine-tune skip-gram embeddings on word pairs from PPDB, producing PARAGRAM vectors. They then build phrase representations either by adding these vectors or by training a supervised recursive neural network (RNN) on PPDB phrase pairs. For evaluation, the authors created two crowd-annotated datasets: Annotated-PPDB and ML-Paraphrase, a paraphrase-oriented re-annotation of the Mitchell and Lapata (2010) bigram set. The learned models reach state-of-the-art results on word and bigram similarity and beat strong baselines on the new phrase tasks. They also rank PPDB pairs closer to human judgments than PPDB's own scores do. The work turns a static paraphrase table into reusable models with much wider coverage.

Key points

  • PPDB contains about 220 million paraphrase pairs. PARAGRAM word vectors trained on its lexical pairs achieve state-of-the-art results on SimLex-999 and also improve sentiment analysis.
  • Annotated-PPDB was built from the first 10M pairs of PPDB XXL. The authors kept only pairs with word overlap below 0.5 and balanced them by confidence level and phrase length, going from 15,000 candidates down to 3,000 pairs. Each pair was rated 1–5 by five Mechanical Turk workers, with a mean deviation of 0.80.
  • ML-Paraphrase re-labels the Mitchell and Lapata (2010) bigram similarity data so that it measures paraphrase strength rather than general relatedness.
  • Additive composition and the RNN behave differently. Addition works well on true paraphrases that are similar in length and share many words. The RNN is stronger on harder pairs and on non-paraphrases, cutting absolute error by about 13% when the length ratio is ≤0.6.
  • The vocabulary is the 100K most frequent English Wikipedia words. The datasets, annotation guidelines, code and trained models are publicly released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Cited by

107 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 106 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00143 · arXiv 1506.03487

@article{wieting2015paraphrase,
  title = {{From Paraphrase Database to Compositional Paraphrase Model and Back}},
  author = {Wieting, John and Bansal, Mohit and Gimpel, Kevin and Livescu, Karen},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.1162/tacl\_a\_00143},
  eprint = {1506.03487},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00143}
}

References

42 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and wordnet-based approaches. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 19–27. Association for Computational Linguistics. find
  2. Ion Androutsopoulos and Prodromos Malakasiotis. 2010. A survey of paraphrasing and textual entailment methods. Journal of Artificial Intelligence Research, pages 135–187. find
  3. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 597–604. Association for Computational Linguistics. find our card
  4. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find our card
  5. Marco Baroni and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1183–1193. Association for Computational Linguistics. find our card
  6. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. The Journal of Machine Learning Research, 3:1137–1155. find
  7. Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of ACL. find
  8. Johannes Bjerva, Johan Bos, Rob van der Goot, and Malvina Nissim. 2014. The meaning factory: Formal semantics for recognizing textual entailment and determining semantic similarity. SemEval 2014, page 642. find
  9. William Blacoe and Mirella Lapata. 2012. A comparison of vector-based representations for semantic composition. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL ’12, pages 546–556, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  10. Wauter Bosma and Chris Callison-Burch. 2007. Paraphrase substitution for recognizing textual entailment. In Proceedings of the 7th International Conference on Cross-Language Evaluation Forum: Evaluation of Multilingual and Multi-modal Information Retrieval, CLEF’06, pages 502–509, Berlin, Heidelberg. Springer-Verlag. find
  11. Chih-Chung Chang and Chih-Jen Lin. 2011. Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):27. find
  12. Scott C. Deerwester, Susan T Dumais, Thomas K. Landauer, George W. Furnas, and Richard A. Harshman. 1990. Indexing by latent semantic analysis. JAsIs, 41(6):391–407. find
  13. Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of Coling 2004, pages 350–356, Geneva, Switzerland, Aug 23–Aug 27. COLING. find our card
  14. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12:2121–2159, July. find
  15. Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question answering. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1608–1618, Sofia, Bulgaria, August. Association for Computational Linguistics. find
  16. Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1606– 1615. find
  17. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th international conference on World Wide Web, pages 406–414. ACM. find
  18. J.R. Firth. 1957. A Synopsis of Linguistic Theory, 1930-1955. find
  19. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. Ppdb: The paraphrase database. In HLT-NAACL, pages 758–764. The Association for Computational Linguistics. find our card
  20. Kazuma Hashimoto, Pontus Stenetorp, Makoto Miwa, and Yoshimasa Tsuruoka. 2014. Jointly learning word representations and composition functions using predicate-argument structures. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, October. Association for Computational Linguistics. find
  21. Felix Hill, KyungHyun Cho, Sebastien Jean, Coline Devin, and Yoshua Bengio. 2014a. Not all neural embeddings are born equal. arXiv preprint arXiv:1410.0718. ↗
  22. Felix Hill, Roi Reichart, and Anna Korhonen. 2014b. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. CoRR, abs/1408.3456. ↗ our card
  23. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar, October. Association for Computational Linguistics. find our card
  24. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. find our card
  25. Yuval Marton, Chris Callison-Burch, and Philip Resnik. 2009. Improved statistical machine translation using monolingually-derived paraphrases. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 381–390, Singapore, August. Association for Computational Linguistics. find
  26. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  27. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  28. Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In ACL, pages 236– 244. Citeseer. find
  29. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1439. find
  30. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), 12. find
  31. Chris Quirk, Chris Brockett, and William Dolan. 2004. Monolingual machine translation for paraphrase generation. In Dekang Lin and Dekai Wu, editors, Proceedings of EMNLP 2004, pages 142–149, Barcelona, Spain, July. Association for Computational Linguistics. find our card
  32. Pushpendre Rastogi and Benjamin Van Durme. 2014. Augmenting FrameNet via PPDB. In Proceedings of the Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pages 1–5, Baltimore, Maryland, USA, June. Association for Computational Linguistics. find
  33. Fabio Rinaldi, James Dowdall, Kaarel Kaljurand, Michael Hess, and Diego Mollá. 2003. Exploiting paraphrases in a question answering system. In Proceedings of the Second International Workshop on Paraphrasing, pages 25–32, Sapporo, Japan, July. Association for Computational Linguistics. find
  34. Richard Socher, Christopher D Manning, and Andrew Y Ng. 2010. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Feature Learning Workshop, pages 1–9. find
  35. Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y Ng. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems, pages 801–809. find
  36. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631– 1642, Seattle, Washington, USA, October. Association for Computational Linguistics. find
  37. Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. TACL, 2:207–218. find our card
  38. James H Steiger. 1980. Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2):245. find
  39. Xuchen Yao, Benjamin Van Durme, Chris Callison-Burch, and Peter Clark. 2013. Semi-markov phrase-based monolingual alignment. In EMNLP, pages 590– 600. find
  40. Mo Yu and Mark Dredze. 2014. Improving lexical embeddings with semantic knowledge. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 545–550, Baltimore, Maryland, June. Association for Computational Linguistics. find
  41. Mo Yu and Mark Dredze. 2015. Learning composition models for phrase embeddings. Transactions of the Association for Computational Linguistics, 3:227– 242. find
  42. Fabio Massimo Zanzotto, Ioannis Korkontzelos, Francesca Fallucchi, and Suresh Manandhar. 2010. Estimating linear models for compositional distributional semantics. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 1263–1271. Association for Computational Linguistics. find