Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2011.01403

Summary

Fine-tuning a pre-trained language model with cross-entropy alone tends to generalize poorly when labels are scarce, reacts badly to noisy labels, and produces results that vary a lot between runs. Gunel et al. address this by adding a supervised contrastive learning (SCL) term to the fine-tuning loss. Within each batch, the l2-normalized [CLS] embeddings of examples that share a label are pulled together, and those with different labels are pushed apart. A temperature τ controls how much hard negatives count, and a weight λ balances SCL against cross-entropy. Unlike self-supervised contrastive approaches, the method needs no augmented views, memory banks, extra unlabeled text or architectural changes. The authors fine-tune RoBERTa-Large on GLUE tasks (SST-2, CoLA, MRPC, RTE, QNLI, MNLI) with 20, 100 or 1000 labeled examples. They also test on noisy training sets built with back-translation and on transfer from SST-2 to related sentiment data. Improvements are largest in the smallest-data settings and shrink with full training sets. This makes SCL a cheap, drop-in option for low-resource text classification.

Key points

• Few-shot gains: SCL improves on the CE-only RoBERTa-Large baseline by up to 10.7 points with 20 labeled examples on SST-2, QNLI and MNLI. For example, QNLI rises from 65.0 to 75.7 and MNLI from 39.3 to 42.7 at N=20.
• Robustness to noise: The authors build noisy training sets with a back-translation model, using its sampling temperature to set the noise level. On these sets, CE+SCL beats CE by up to 7 points.
• Transfer: A model fine-tuned on SST-2 and transferred to Amazon-2 gains 2.9 points over the CE-only model. It also shows much lower variance across few-shot training samples.
• Full-data results and ablations: With full training data, gains are modest and mainly significant on MRPC (86.4 → 89.5) and QNLI (90.4 → 93.9). A "CE+CE" ablation keeps the normalization and temperature but drops SCL, and it recovers much of the improvement. The paper also includes a comparison with Khosla et al.'s (2020) approach, which trains in two separate stages.
• Cost and batch size (RoBERTa-Base): SCL slows fine-tuning somewhat, for example 1.54 vs 2.46 updates/sec at batch size 256. Larger batches help it slightly: MNLI improves from 85.3 at batch size 16 to 86.6 at 256.