Elamur Research
☰
home / papers / 2022 / P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks

P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.acl-short.8.pdf

Summary

Fine-tuning a pretrained language model updates every weight. This takes a lot of training memory and requires storing a full model copy for each task. Earlier prompt tuning methods (Lester et al., 2021; P-tuning) train only continuous input embeddings on a frozen backbone. They come close to fine-tuning only at around 10B parameters and do poorly on sequence labeling. The authors present P-Tuning v2, an NLU-oriented implementation of Deep Prompt Tuning. It adds trainable prefix prompts at every transformer layer, not just the input. They also describe several practical choices:

  • optional MLP reparameterization
  • task-dependent prompt length
  • optional multi-task pretraining of the prompts
  • a randomly initialized linear classification head instead of a verbalizer

Experiments use fully supervised SuperGLUE plus NER, extractive QA (SQuAD) and SRL (CoNLL05/12), on frozen models from roughly 330M to 10B parameters. P-Tuning v2 performs on par with fine-tuning while training only 0.1–3% of the parameters per task. This makes it a cheap, general alternative and baseline.

Key points

  • Backbones: BERT-large, RoBERTa-large, DeBERTa-xlarge and GLM-xlarge/xxlarge (about 300M–10B parameters). Code is at github.com/THUDM/P-tuning-v2.
  • Prompt length: Classification tasks work best with short prompts (under 20 tokens). Sequence labeling tasks prefer about 100 tokens.
  • Prompt depth (BERT-large): Placing prompts in deeper layers beats placing them in early layers, given the same budget. On RTE, prompting only layers 17–24 nearly matches prompting all layers.
  • Classification head (RoBERTa-large): A [CLS] linear head performs about as well as, or slightly better than, a verbalizer with LM head. Scores were SST-2 96.3 vs 95.8, RTE 88.4 vs 86.6, BoolQ 84.8 vs 84.6 and CB 96.4 vs 94.6.
  • Task-specific effects: Single-layer prompt tuning behaves poorly on SQuAD 2.0, likely because of its unanswerable questions. Multi-task prompts help most tasks except QA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

238 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 234 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-short.8

@inproceedings{liu2022p,
  title = {{P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks}},
  author = {Liu, Xiao and Ji, Kaixuan and Fu, Yicheng and Tam, Weng and Du, Zhengxiao and Yang, Zhilin and Tang, Jie},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-short.8},
  url = {https://doi.org/10.18653/v1/2022.acl-short.8}
}

References

26 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. find
  2. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. find
  3. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too. arXiv:2103.10385. find our card
  4. Xavier Carreras and Lluís Màrquez. 2004. Introduction to the CoNLL-2004 shared task: Semantic role labeling. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004, pages 89–97, Boston, Massachusetts, USA. Association for Computational Lingu… find
  5. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv e-prints. find
  6. Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2021. Noisy channel language model prompting for few-shot text classification. arXiv preprint arXiv:2108.04106. find our card
  7. Xavier Carreras and Lluís Màrquez. 2005. Introduction to the CoNLL-2005 shared task: Semantic role labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 152–164, Ann Arbor, Michigan. Association for Computational Linguistics. find our card
  8. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguisti… find our card
  9. Xiang Chen, Xin Xie, Ningyu Zhang, Jiahuan Yan, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2021. Adaprompt: Adaptive prompt-based finetuning for relation extraction. arXiv preprint arXiv:2104.07650. — [arXiv:2104.07650](https://arxiv.org/abs/2104.07650) find
  10. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying lms with mixtures of soft prompts. arXiv preprint arXiv:2104.06599. find our card
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  12. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv e-prints. find
  14. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  15. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2021. All nlp tasks are generation tasks: A general pretraining framework. arXiv preprint arXiv:2103.10360. find our card
  16. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. find our card
  17. Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723. find our card
  18. Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050. find
  19. Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2021. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint arXiv:2109.04332. find our card
  20. Timo Schick and Hinrich Schütze. 2020. It’s not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118. find our card
  21. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. find our card
  22. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980. find our card
  23. Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, et al. 2021. What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers. arXiv… find
  24. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In NeurIPS 2019, pages 3261– 3275. find our card
  25. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691. find
  26. A Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv e-prints. find our card