Towards Universal Paraphrastic Sentence Embeddings

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1511.08198

Summary

Word embeddings transfer easily across tasks, but comparably reusable sentence vectors were still lacking. This work learns sentence encoders whose cosine similarity reflects how strongly two texts paraphrase each other. The training signal comes from noisy phrase pairs in the Paraphrase Database (PPDB). The authors compare six composition functions of rising complexity: plain word averaging, averaging plus a linear projection, deep averaging networks (DAN), a vanilla RNN, an identity-initialized RNN, and an LSTM. They test on PPDB-like in-domain data and on 22 SemEval textual similarity sets covering news, tweets, forums and captions.

The LSTM wins in-domain, but it degrades sharply on other domains. The averaging model, whose only parameters are the word vectors, transfers far better and rivals systems built for the individual tasks. In supervised settings, averaging is again stronger on SICK similarity and entailment, while the LSTM is best on sentiment. The released PARAGRAM-PHRASE XXL embeddings give a cheap, strong baseline for universal sentence representations.

Key points

• Out of domain, word averaging beats the best LSTM by 16.5 Pearson's r on average across the 22 SemEval STS datasets.
• Word vectors learned for averaging outperform averaged pretrained vectors:
  • +17.1 points over GloVe (840B-token Common Crawl);
  • +12.8 points over PARAGRAM-SL999.
• For each SemEval STS competition held between 2012 and 2015, PARAGRAM-PHRASE XXL places within the top 25% of participating entries. It is best or tied for best on 4 datasets.
• On the Stanford Sentiment Treebank, the LSTM reaches 89.2% on coarse-grained (binary) classification, which the authors report as a new state of the art.
• Using the pretrained embeddings as a prior gives near state-of-the-art results on SICK similarity and entailment. As a black-box feature extractor, they compare favorably with skip-thought vectors (Kiros et al., 2015). Models and code are publicly released.