Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N18-1049.pdf

Summary

The paper asks whether the simple, scalable recipe behind word embeddings can also produce strong general-purpose vectors for whole sentences without labeled data. The authors propose Sent2Vec, which extends the C-BOW objective from a fixed local window to the full sentence. A sentence vector is the mean of learned source embeddings for its words and its n-grams. Each word is predicted from the rest of the sentence using negative sampling and subsampling of frequent targets. Unlike C-BOW, the model drops dynamic windows and input-side subsampling, and it stores n-grams through FastText-style hashing. Models were trained on the Toronto book corpus, Wikipedia sentences and tweets. Both training and inference cost only a constant number of vector operations per token. Across most benchmarks, Sent2Vec beats earlier unsupervised methods and often semi-supervised ones too, while running as cheaply as plain averaging. This matters because strong sentence representations can then be trained on very large corpora using CPUs.

Key points

• Tokenization used the Stanford NLP library for Wikipedia and Toronto books and the NLTK tweet tokenizer for Twitter. Training used SGD and parallelizes easily.
• Against Arora et al. (2017), the Sent2Vec tweet model (19.7B words) with unigrams and bigrams scored 0.701 on STS 2014, 0.715 on SICK 2014 and 0.835 on the supervised average. GloVe + WR scored 0.685, 0.722 and 0.815, despite using a Common Crawl corpus 42 times larger.
• On semantic similarity, supervised PSL + WR still leads (0.735 on STS 2014, 0.729 on SICK). Sent2Vec, however, outperforms it on downstream supervised tasks.
• Bigrams improve supervised classification, likely by capturing phrases such as "not good", but slightly hurt unsupervised similarity. Models trained on Toronto books set the best result on the STS 2014 images subset, ahead of the supervised Caption-Rep.
• The learned word-vector norms down-weight both very frequent and very rare tokens, which roughly matches Luhn's hypothesis. Code and models were released at github.com/epfml/sent2vec.