📄 Generate, Annotate, and Learn: NLP with Synthetic Text

Summary

Many NLP tasks, especially GLUE tasks with sentence-pair inputs, lack in-domain unlabeled text, which limits self-training and knowledge distillation (KD). The authors propose GAL ("generate, annotate, and learn"), a framework that uses language models as a source of synthetic task-specific inputs. They fine-tune GPT-2 on a dataset's inputs with the labels removed, or prompt a large LM with a few examples. They then sample new text without any prompt. The strongest available classifier assigns soft pseudo labels to this text. For few-shot learning, the LM itself supplies hard labels. Finally, a new model is trained on the real labeled data plus the pseudo-labeled data, with no filtering step. The paper motivates the approach through empirical and vicinal risk minimization. It also argues, in theory and in experiments, that unconditional generation followed by teacher annotation beats class-conditional generation of labeled text. GAL sets a new state of the art for 6-layer transformer distillation on the GLUE leaderboard and also improves self-training and few-shot learning. This makes it a simple, widely applicable way to augment data.

Key points

• Generation setup: GPT-2 is fine-tuned without labels. Sentence pairs are joined with a [SEP] token, and generation starts from a [BOS] token using top-k sampling. Self-training runs use RoBERTa-base with 40× synthetic data.

• Gains over RoBERTa-base on GLUE dev sets (with GPT-2-large):

  • SST-2: 94.8 → 95.3
  • RTE: 78.8 → 81.4
  • MRPC: 90.1 → 91.7
  • CoLA: 63.6 → 65.1

  Larger GPT-2 models mainly help on MRPC and CoLA.

• Soft vs. hard pseudo labels: Soft labels outperform hard labels on all four tasks, for example 65.1 vs. 63.0 on CoLA.

• Class-conditional generation: When class-conditional GPT-2 output is used with its predefined labels, results fall below the baseline (CoLA 58.4, RTE 74.4). Re-annotating the same text with GAL brings it back on par with unconditional generation.

• Few-shot learning: On four 4-shot tasks, GAL gives an average gain of 1.3% and outperforms GPT-3-6B.

• Limitations: Only sentence-level tasks were studied, and no data filtering was used. Document-level and structured prediction tasks (e.g., MT) are left for future work.