ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1042.pdf

Summary

Large freely available collections of sentence-level paraphrases did not exist. The closest resource, PPDB, works well for word embeddings but is weak for learning sentence representations. Wieting and Gimpel address this by scaling up an earlier back-translation idea. They took the CzEng 1.6 Czech-English bitext and ran its Czech side through a pretrained NMT system. Each English output was then paired with the original English reference, producing ParaNMT-50M, a corpus of over 50 million English-English paraphrase pairs.

The authors compared candidate source corpora and manually checked the quality of the resulting pairs. They then used the data to train paraphrastic sentence embeddings, adding a "mega-batching" technique that widens the pool from which negative examples are drawn. The models score sentence pairs only by cosine similarity between independently encoded sentences, with no task-specific supervision. Even so, they beat every system submitted to the SemEval STS competitions from 2012 to 2016. The corpus also supports paraphrase generation, including syntactically controlled paraphrasing. All data, code and embeddings are publicly released.

Key points

• Pairs were built with the pretrained Czech-English model of Sennrich et al. (2017), using beam size 12. Back-translating CzEng (51.4M pairs) took more than 10,000 GPU hours. Among Common Crawl, Europarl, News Commentary and CzEng, CzEng gave the best embeddings at equal data size.
• In a manual evaluation of 100-pair samples across paraphrase-score ranges, about 90% of back-translations were judged fluent. Paraphrase strength rose with the automatic score, and the low-scoring pairs often traced back to misaligned sentences in the original bitext.
• The best model, a jointly trained concatenation of WORD and TRIGRAM encoders, outperformed the top supervised STS systems of every year from 2012 to 2016. Training on ParaNMT instead of SimpWiki improved correlation by 3–6 points.
• Similarity scores taken directly from NMT encoders (Hill et al., 2016; Mallinson et al., 2017) trail these embeddings by roughly 10–35 absolute Pearson points.
• An encoder-decoder trained on the corpus learned to canonicalize text and fix grammatical errors. The corpus was also used to train the SCPN of Iyyer et al. (2018), which rewrites sentences to match parse templates.