Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D17-1070.pdf
Summary
Pretrained word embeddings are standard in NLP, but there was still no widely adopted way to get general-purpose vectors for whole sentences. Unsupervised encoders such as SkipThought and FastSent were the main options. The authors ask whether supervised training could do for sentences what ImageNet pretraining does for computer vision, and they propose natural language inference as the source task. They train on the SNLI corpus, which has 570k labeled English sentence pairs. A shared encoder turns the premise and the hypothesis into separate vectors. These are combined by concatenation, element-wise product and absolute difference, and an MLP classifies the pair. They compare seven encoder designs: LSTM/GRU variants, BiLSTM with mean or max pooling, a self-attentive network and a hierarchical ConvNet. The BiLSTM with max pooling transfers best. They freeze its embeddings and fit only a simple classifier on each downstream task. On this broad benchmark suite it outperforms SkipThought, even though it uses far less training data and trains in under a day on one GPU. This suggests that NLI can act as a general pretraining task for sentence encoders. The encoder is released as InferSent.
Key points
- Training setup: SNLI has three labels (entailment, contradiction, neutral). Word inputs are frozen 300-d GloVe vectors trained on Common Crawl 840B. Training uses SGD with mini-batches of 64, and the classifier is an MLP with one hidden layer of 512 units.
- Comparison with SkipThought: InferSent trains on 570k sentence pairs, while SkipThought trains on 64M sentences. Training takes under a day, compared with about a month for SkipThought-LN. On STS14, InferSent's cosine similarity reaches a Pearson score of 0.68, against 0.29 for SkipThought and 0.44 for SkipThought-LN.
- SICK results (new state of the art): SICK-R reaches a Pearson score of 0.885, above the previous 0.868 (Tai et al.). SICK-E reaches 86.3% accuracy, above the previous hand-engineered best of 84.5% and the 80.8% of earlier SNLI→SICK fine-tuning.
- COCO image-caption retrieval (with ResNet-101 features): Caption retrieval R@1 is 42.4, compared with 37.9 for SkipThought. Image retrieval R@1 is 33.2, compared with 30.6. These scores are close to m-CNN, which is trained directly on the retrieval task.
- Source-task comparison: NLI-trained encoders beat encoders trained on other supervised data, namely COCO captions, dictionary definitions, NMT, PPDB and SST. The code is at github.com/facebookresearch/InferSent.
Cited by
- Publicly Available Clinical BERT Embeddings
- Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
- LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
- Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference
- Abductive Commonsense Reasoning
- e-SNLI: Natural Language Inference with Natural Language Explanations
- Universal Sentence Encoder for English
- Universal Sentence Encoder
- QuAC: Question Answering in Context
- Semi-Supervised Sequence Modeling with Cross-View Training
- SPECTER: Document-level Representation Learning using Citation-informed Transformers
- SentEval: An Evaluation Toolkit for Universal Sentence Representations
- What you can cram into a single \$&!\#* vector: Probing sentence embeddings for linguistic properties
- Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference
- Language-agnostic BERT Sentence Embedding
- SimCSE: Simple Contrastive Learning of Sentence Embeddings
- DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
- Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition
- Neural CRF Model for Sentence Alignment in Text Simplification
- Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
- On the Sentence Embeddings from Pre-trained Language Models
- Towards General Text Embeddings with Multi-stage Contrastive Learning
- Linguistic Knowledge and Transferability of Contextual Representations
- An efficient framework for learning sentence representations
- On Measuring Social Biases in Sentence Encoders
- Learned in Translation: Contextualized Word Vectors
- Advances in Pre-Training Distributed Word Representations
- TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP
- MTEB: Massive Text Embedding Benchmark
- Generative Representational Instruction Tuning
- Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models
- NOPE: A Corpus of Naturally-Occurring Presuppositions in English
- Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
- Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation
- Explain Yourself! Leveraging Language Models for Commonsense Reasoning
- EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference
- Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
- Probing Natural Language Inference Models through Semantic Fragments
- Lessons from Natural Language Inference in the Clinical Domain
- Generating Datasets with Pretrained Language Models
- RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation
- How to Fine-Tune BERT for Text Classification?
- Patient Knowledge Distillation for BERT Model Compression
- What do you learn from context? Probing for sentence structure in contextualized word representations
- Temporal Reasoning in Natural Language Inference
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
- Improving Text Embeddings with Large Language Models
- Dialogue Natural Language Inference
- Beyond BLEU: Training Neural Machine Translation with Semantic Similarity
- ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
- Multilingual Universal Sentence Encoder for Semantic Retrieval
- Word-level Textual Adversarial Attacking as Combinatorial Optimization
Cite
@inproceedings{conneau2017supervised,
title = {{Supervised Learning of Universal Sentence Representations from Natural Language Inference Data}},
author = {Conneau, Alexis and Kiela, Douwe and Schwenk, Holger and Barrault, Loïc and Bordes, Antoine},
booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
year = {2017},
doi = {10.18653/v1/d17-1070},
eprint = {1705.02364},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/d17-1070}
}References
- Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2… find our card