Elamur Research⌕
☰
home / papers / 2021 / Improving and Simplifying Pattern Exploiting Training

Improving and Simplifying Pattern Exploiting Training

Paper · 2021 · Derek Tam, Rakesh R. Menon, Mohit Bansal et al.

Summary

Large language models such as GPT-3 can handle tasks from only a handful of labeled examples, but their size makes them hard to study or deploy. PET (Pattern Exploiting Training) closes much of this gap with a smaller ALBERT model by turning tasks into cloze questions. However, it also depends on thousands of task-specific unlabeled examples, which can be costly to collect for benchmarks like SuperGLUE.

The authors present ADAPET, which drops the unlabeled data and instead extracts more training signal from the labeled set. It combines two changes to PET's objective:

  • Decoupled label loss: probabilities are normalized over the entire vocabulary, and a binary cross-entropy term promotes the correct label token while suppressing the incorrect ones. The same formulation covers single-token, WSC-style, and multi-token labels.
  • Label-conditioned MLM: the model reconstructs masked input tokens only when the input is paired with the correct label.

With just 32 labeled examples per SuperGLUE task, ADAPET surpasses PET, iPET, and GPT-3 on the hidden test set. This shows that a denser objective can replace extra unlabeled data in few-shot learning.

Key points

  • Setup: SuperGLUE with 32 labeled examples per task, using the same split, pre-trained ALBERT model, and hyperparameters as PET, but trained for 1k batches. Code is available at github.com/rrmenon10/ADAPET.
  • Test-set average: ADAPET reaches 76.0, compared with iPET 75.4, PET 74.0, and GPT-3 71.8. iPET and PET also use roughly 9K unlabeled examples per task.
  • Dev-set average: ADAPET reaches 77.3, about 0.5 points above iPET (76.8). It also beats the single-pattern baseline sPET (74.2) and GPT-3 (73.2).
  • Ablation: removing label conditioning sharply lowers CB from 91.1/88.1 to 75.0/62.8 (Acc./F1) and RTE from 75.1 to 64.3.
  • The decoupled loss replaces PET's hinge loss for multi-token labels (COPA, ReCoRD) by splitting the loss into per-token terms, which avoids normalization problems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of North Carolina at Chapel HillChapel Hill · United States37 works · 14,332 citations🎓Toyota Technological Institute at ChicagoChicago · United States29 works · 10,273 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava and Colin Raffel. 2021. Improving and Simplifying Pattern Exploiting Training. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.407

@inproceedings{tam2021improving,
  title = {{Improving and Simplifying Pattern Exploiting Training}},
  author = {Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava and Colin Raffel},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.407},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.407}
}

References

3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… our card
  2. Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix-Text: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147– 2157, Online. Association for Computational L… find
  3. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:… our card

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tam-2021-improving-simplifying-pattern-exploiting-training/