Elamur Research⌕
☰
home / papers / 2022 / OpenPrompt: An Open-source Framework for Prompt-learning

OpenPrompt: An Open-source Framework for Prompt-learning

Summary

Prompt-learning adapts pre-trained language models (PLMs) to downstream tasks by recasting those tasks as the kind of word prediction the models learned during pre-training. Existing code for it has been scattered, tied to narrow scenarios, and hard to reproduce. Details such as templates, verbalizers and tokenization also strongly affect results. The authors present OpenPrompt, an open-source PyTorch toolkit that splits the prompt-learning pipeline into separate but cooperating modules:

  • templates, either textual or soft, written in a dedicated template language;
  • verbalizers that map labels to vocabulary words;
  • a prompt-aware tokenizer;
  • dataset classes;
  • a PromptModel;
  • a trainer.

The central feature is combinability. Any supported PLM family can be paired with any task format and prompting method, for example running prefix-tuning on classification. Several published methods are reimplemented, and baselines cover a wide range of benchmarks. The latest scores live in the GitHub repository rather than in the paper itself. The toolkit matters because it supports reproducibility, fair comparison of methods, and an easier start for newcomers.

Key points

  • OpenPrompt is released at github.com/thunlp/OpenPrompt. It is built on PyTorch and loads PLMs directly from huggingface transformers.

  • It distinguishes three PLM families by pre-training objective:

    • MLM, e.g. BERT and RoBERTa;
    • autoregressive LM, e.g. GPT-3;
    • Seq2Seq, e.g. T5, MASS and BART.

    A unified interface computes the loss, so users do not write objective heads.

  • The core classes are Template, Verbalizer, PromptTokenizer, PromptDataset, PromptModel and PromptTrainer. Verbalizers are skipped in generation tasks.

  • Reimplemented approaches include PTR, P-tuning, Prefix-tuning and LM-BFF, plus simple baselines for text classification, knowledge probing and entity typing.

  • Evaluation spans several task types:

    • generation: WebNLG;
    • natural language understanding: GLUE and SuperGLUE;
    • information extraction: SemEval and Few-NERD;
    • text classification: MNLI, AG's News, DBPedia and IMDB;
    • knowledge probing: LAMA.

    All dataset processors inherit from one DataProcessor base class.

Interesting ideas

  • I1. Combinability as a design goal. The toolkit treats model, task format and prompting method as independent axes to mix freely. This makes it possible to test whether a method generalises beyond the setting it was proposed for, which is useful in any benchmarking framework.
  • I2. A small template language. Prompts are written in a human-readable markup with token-level attributes, such as marking soft or trainable tokens. This separates prompt design from code and would fit Ukrainian-language prompting experiments well.
  • I3. Prompt-aware tokenization. Tokenization handles mask and entity positions and never truncates the template, only the input. This guards against silent bugs, such as misaligned mask indices, that are easy to introduce when wrapping text for any model.
  • I4. One base class for all dataset processors. Inheriting every processor from a common class makes adding a new corpus cheap and keeps evaluation uniform. This is a good pattern for a shared library of Ukrainian datasets.
  • I5. Results kept in the repository. Publishing living results in the repo instead of freezing them in the paper keeps comparisons current as implementations improve.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

93 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 87 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-demo.10

@inproceedings{ding2022openprompt,
  title = {{OpenPrompt: An Open-source Framework for Prompt-learning}},
  author = {Ding, Ning and Hu, Shengding and Zhao, Weilin and Chen, Yulin and Liu, Zhiyuan and Zheng, Haitao and Sun, Maosong},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations},
  year = {2022},
  doi = {10.18653/v1/2022.acl-demo.10},
  url = {https://doi.org/10.18653/v1/2022.acl-demo.10}
}

References

44 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. unified view of parameter-efficient transfer learning. In Proceedings of ICLR. find
  2. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of ACL, pages 4171–4186, Minneapolis, Minnesota. find
  4. Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Pengjun Xie, Hai-Tao Zheng, Zhiyuan Liu, Juanzi Li, and Hong-Gee Kim. 2021a. Prompt-learning for fine-grained entity typing. Arxiv preprint, 2108.10604. find
  5. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2022. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.06904. — [arXiv:2203.06904](https://arxiv.org/abs/2203.06904) find
  6. Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Hai-Tao Zheng, and Zhiyuan Liu. 2021b. Few-nerd: A few-shot named entity recognition dataset. In Proceedings of ACL. find our card
  7. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2021. All nlp tasks are generation tasks: A general pretraining framework. arXiv preprint arXiv:2103.10360. find our card
  8. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of ACL, pages 3816–3830, Online. find our card
  9. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from rdf data. In Proceedings of INLG, pages 124–133. find our card
  10. Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2021. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint arXiv:2109.04332. find our card
  11. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021a. Pre-trained models: Past, pres… — [arXiv:2106.07139](https://arxiv.org/abs/2106.07139) find
  12. Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021b. Ptr: Prompt tuning with rules for text classification. ArXiv preprint, 2105.11259. find
  13. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. Towards a Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. SemEval-2010 task 8: Multi-way class… find
  14. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of ICML, pages 2790–2799. PMLR. find
  15. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In Proceedings of ICLR. find
  16. Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Juanzi Li, and Maosong Sun. 2021. Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification. ArXiv preprint, 2108.02035. find our card
  17. Teven Le Scao and Alexander M Rush. 2021. How many data points is a prompt worth? In Proceedings of NAACL, pages 2627–2636. find our card
  18. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167–195. find
  19. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. ArXiv preprint, abs/2104.08691. find
  20. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of ACL, pages 7871–7880, Online. find our card
  21. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings ACL, pages 4582–4597, Online. Association for Computational Linguistics. find
  22. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021a. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ArXiv preprint, abs/2107.13586. find our card
  23. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021b. Gpt understands, too. arXiv preprint arXiv:2103.10385. find our card
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv preprint, abs/1907.11692. find
  25. Robert L Logan IV, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting down on prompts and parameters: Simple few-shot learning with language models. arXiv preprint arXiv:2106.13353. find our card
  26. Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of ACL. find our card
  27. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL, pages 311–318. find our card
  28. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Proceedings of NeurIPS, 32:8026–8037. find our card
  29. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066. find our card
  30. Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, et al. 2021. Exploring low-dimensional intrinsic task subspace via prompt tuning. arXiv preprint arXiv:2110.07867. — [arXiv:2110.07867](https://arxiv.org/abs/2110.07867) find
  31. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, pages 1–26. find our card
  32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv preprint, abs/1910.10683. find
  33. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. find
  34. Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of EACL, pages 255–269, Online. Association for Computational Linguistics. find our card
  35. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450. find our card
  36. Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. 2021. Spot: Better frozen model adaptation through soft prompt transfer. arXiv preprint arXiv:2110.07904. find our card
  37. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537. find our card
  38. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  39. Colin Wei, Sang Michael Xie, and Tengyu Ma. 2021. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. In Proceedings of NeurIPS. find
  40. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426. find
  41. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… find our card
  42. Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. CrossFit: A few-shot learning challenge for cross-task generalization in NLP. In Proceedings of EMNLP, pages 7163–7189. Association for Computational Linguistics. find
  43. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of NIPS. find our card
  44. Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.09690. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ding-2022-openprompt-open-source-framework-prompt/