Elamur Research⌕
☰
home / papers / 2022 / Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models

Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models

Summary

Few-shot learning with masked language models usually depends on carefully crafted prompts. Prompt-based finetuning is accurate but needs a full model copy for every task, while in-context learning reuses one frozen model but is very sensitive to prompt wording. The authors test whether simple prompt design and memory efficiency can be achieved together. They use RoBERTa and ALBERT on eight GLUE/SuperGLUE classification tasks, with 16 examples per label and cross-validated model selection. They introduce "null prompts," which simply concatenate the input fields with a [MASK] token and use no template text or demonstrations. Under finetuning, these prompts perform about as well as manually written ones. Automated prompt tuning for frozen models generally fell short of manual prompts. Among lightweight finetuning options, updating only the bias terms (BitFit) gave the best accuracy-to-cost balance, often matching or beating full finetuning. The recommended recipe is null prompts plus BitFit. It reduces the effort of prompt engineering and per-task storage, and it suggests that few-shot gains come largely from the [MASK]-prediction setup itself rather than from clever patterns.

Key points

  • The eight evaluation tasks were SST-2, RTE, QQP, QNLI, MRPC, MNLI, CB and BoolQ, each trained with K = 16 examples per label. Hyperparameters were chosen by 4-fold cross-validation, and results were averaged over 10 random seeds. WiC and WNLI were dropped because every model performed near chance.
  • With RoBERTa, null prompts plus BitFit reached 90.6 F1 on CB and 65.1 F1 on QQP. In-context learning scored 51.2 and 55.6, and full prompt-based finetuning scored 90.6 and 62.9.
  • BitFit trains roughly 10^5 parameters per task, about 0.1% of the model, which the authors describe as around 1000x more memory efficient than full finetuning. The other lightweight methods compared were Adapters (~10^7 parameters), LM head tuning (~10^3) and calibration (~10^1).
  • Accuracy showed diminishing returns as more parameters were trained. Going beyond BitFit to Adapters or all parameters added little or even hurt results on 16-example training sets.
  • Code is released at github.com/ucinlp/null-prompts. Extending the findings to very large and left-to-right LMs is left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

87 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 72 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-acl.222

@inproceedings{iv2022cutting,
  title = {{Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models}},
  author = {IV, Robert Logan and Balazevic, Ivana and Wallace, Eric and Petroni, Fabio and Singh, Sameer and Riedel, Sebastian},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.222},
  url = {https://doi.org/10.18653/v1/2022.findings-acl.222}
}

References

27 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. find our card
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  3. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. ArXiv preprint, abs/2002.06305. find
  4. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa… find our card
  5. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Lon… find
  6. Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. In Science. find
  7. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevi… find
  8. Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2627–2636, Online. Association for Computational Linguistics. find our card
  9. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguist… find
  10. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… find
  11. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. ArXiv preprint, abs/2103.10385. find our card
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv preprint, abs/1907.11692. find
  13. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. ArXiv preprint, abs/2104.08786. find our card
  14. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021. On the stability of fine-tuning BERT: misconceptions, explanations, and strong baselines. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. find
  15. Juri Opitz. 2019. Argumentative relation classification as plausibility ranking. In Proceedings of the 15th Conference on Natural Language Processing (KON-VENS 2019): Long Papers, pages 193–202, Erlangen, Germany. German Society for Computational Linguistics & Language Technology. find
  16. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR. find
  17. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. ArXiv preprint, abs/2105.11447. find our card
  18. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203–5212, Online. Association for Computatio… find our card
  19. Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Comp… find our card
  20. Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017–5033, Online. Association for Co… find our card
  21. Timo Schick and Hinrich Schütze. 2021b. It’s not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. Assoc… find our card
  22. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin… find our card
  23. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural… find our card
  24. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRevi… find our card
  25. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2020. HuggingFace’s Transformers: State-of-the-art natural language processing. In EMNLP Demo Track. find our card
  26. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. find our card
  27. Dataset Pattern Verbalizer BoolQ {passage}. Question: {question}? Answer: [MASK]. True: "Yes" False: "No" CB {premise}? [SEP] [MASK], {hypothesis} entailment: "Yes" contradiction: "No" neutral: "Maybe" MNLI {sentence1}? [SEP] [MASK], {sentence2} entailment: "Yes" contradiction: "No" neutral: "Maybe"… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/iv-2022-cutting-down-prompts-parameters-simple/