Elamur Research
☰
home / papers / 2022 / PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts

PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2202.01279

Summary

Prompting means recasting a task as natural-language text for a language model to complete. It has become central to zero- and few-shot NLP, but results depend heavily on prompt wording, and few tools existed for writing, checking and pooling prompts. The authors argue that prompts differ from ordinary annotations. Each prompt is a function applied to an entire dataset, and deliberate variety among prompts rules out agreement-based quality metrics.

PromptSource has three components:

  • a Jinja-based templating language that mixes dataset fields, fixed text and simple logic;
  • a web GUI connected to HuggingFace Datasets, with a Browse view for exploring datasets and existing prompts and a Sourcing view for drafting templates and checking their output on many examples;
  • community contribution guidelines enforced through code review.

The result is P3, an open, reviewed collection of over 2,000 prompts for roughly 170 datasets. It was used to train the T0 models and supported later work on multilingual and in-context prompting. This makes prompt research more standardized and reproducible.

Key points

  • P3 (Public Pool of Prompts): more than 2,000 reviewed prompts covering about 170 datasets. It is available at github.com/bigscience-workshop/promptsource.
  • Workflow: authors follow five stages: dataset exploration, prompt writing, documentation, iteration with variation, and global review through code review plus automatic tests.
  • Guidelines:
    • Prompts must be in natural language that a non-expert adult can understand.
    • Targets should contain only the answer.
    • Listing the valid answer options is encouraged, since prompts without them tended to perform worse.
    • Authors should write about 5–10 varied prompts per dataset.
  • Metadata per prompt: name, source reference, whether it reflects the original task, valid outputs, whether the template lists those outputs, and suggested metrics.
  • Case studies:
    • T0 massively multitask training (Sanh et al., 2021), using a subset of P3;
    • multilingual prompting with a model trained on 30 languages (Lin et al., 2021);
    • priming / in-context learning (Min et al., 2021), where adding P3 templates further improved results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

143 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 125 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2202.01279 · arXiv 2202.01279

@misc{bach2022promptsource,
  title = {{PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts}},
  author = {Bach, Stephen H. and Sanh, Victor and Yong, Zheng-Xin and Webson, Albert and Raffel, Colin and Nayak, Nihal V. and Sharma, Abheesht and Kim, Taewoon and Bari, M Saiful and Fevry, Thibault and Alyafeai, Zaid and Dey, Manan and Santilli, Andrea and Sun, Zhiqing and Ben-David, Srulik and Xu, Canwen and Chhablani, Gunjan and Wang, Han and Fries, Jason Alan and Al-shaibani, Maged S. and Sharma, Shanya and Thakker, Urmish and Almubarak, Khalid and Tang, Xiangru and Radev, Dragomir and Jiang, Mike Tian-Jian and Rush, Alexander M.},
  year = {2022},
  doi = {10.48550/arxiv.2202.01279},
  eprint = {2202.01279},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2202.01279}
}

References

39 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… find
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  3. Wei-Te Chen and Will Styler. 2013. Anafora: A web-based general purpose annotation tool. In Proceedings of the 2013 NAACL HLT Demonstration Session, pages 14–19, Atlanta, Georgia. Association for Computational Linguistics. find
  4. Hamish Cunningham, Diana Maynard, Kalina Bontcheva, and Valentin Tablan. 2002. GATE: an architecture for development of robust HLT applications. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 168–175, Philadelphia, Pennsylvania, USA. Association for… find
  5. Richard Eckart de Castilho, Éva Mújdricza-Maydt, Seid Muhie Yimam, Silvana Hartmann, Iryna Gurevych, Anette Frank, and Chris Biemann. 2016. A web-based tool for the integrated annotation of semantic and syntactic structures. In Proceedings of the Workshop on Language Technology Resources and Tools f… find
  6. The COLING 2016 Organizing Committee. find
  7. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa… find our card
  8. Philipp Helfrich, Elias Rieb, Giuseppe Abrami, Andy Lücking, and Alexander Mehler. 2018. TreeAnnotator: Versatile visual annotation of hierarchical text relations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Lang… find
  9. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dyna… find
  10. Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2627–2636, Online. Association for Computational Linguistics. find our card
  11. Dong-Ho Lee, Rahul Khanna, Bill Yuchen Lin, Seyeon Lee, Qinyuan Ye, Elizabeth Boschee, Leonardo Neves, and Xiang Ren. 2020. LEAN-LIFE: A label-efficient annotation framework towards learning from explanation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics:… find
  12. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguist… find
  13. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342, Vancouver, Canada. Association for Computational Linguistics. find
  14. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry,… find our card
  15. Yanzeng Li, Bowen Yu, Li Quangang, and Tingwen Liu. 2021. FITAnnotator: A flexible and intelligent text annotation system. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, pages 35– 41,… find
  16. Bill Yuchen Lin, Dong-Ho Lee, Frank F. Xu, Ouyu Lan, and Xiang Ren. 2019. AlpacaTag: An active learning-based crowd annotation framework for sequence tagging. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 58–63, Florence, Ita… find
  17. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona T. Diab, Veselin Stoy… find our card
  18. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. CoRR, abs/2107.13586. find our card
  19. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021. Metaicl: Learning to learn in context. CoRR, abs/2110.15943. find
  20. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773. find
  21. Mariana Neves and Jurica Ševa. 2021. An extensive review of tools for manual annotation of documents. Briefings in bioinformatics, 22(1):146–163. find
  22. Frédéric Papazian, Robert Bossy, and Claire Nédellec. 2012. AlvisAE: a collaborative web text annotation editor for knowledge acquisition. In Proceedings of the Sixth Linguistic Annotation Workshop, pages 149–152, Jeju, Republic of Korea. Association for Computational Linguistics. find
  23. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. NeurIPS. find our card
  24. Jan Wira Gotama Putra, Simone Teufel, Kana Matsumura, and Takenobu Tokunaga. 2020. TIARA: A tool for annotating discourse relations and sentence reordering. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6912–6920, Marseille, France. European Language Resources Associ… find
  25. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203–5212, Online. Association for Computatio… find our card
  26. David Salgado, Martin Krallinger, Marc Depaule, Elodie Drula, Ashish V. Tendulkar, Florian Leitner, Alfonso Valencia, and Christophe Marcelle. 2012. MyMiner: a web application for computer-assisted biocuration and text annotation. Bioinformatics, 28(17):2285–2287. find
  27. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Da… find our card
  28. Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Comp… find our card
  29. Timo Schick and Hinrich Schütze. 2021b. It’s not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. Assoc… find our card
  30. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin… find our card
  31. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. brat: a web-based tool for NLP-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, page… find
  32. Michael Stewart, Wei Liu, and Rachel Cardell-Oliver. 2019. Redcoat: A collaborative annotation tool for hierarchical entity typing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMN… find
  33. Albert Webson and Ellie Pavlick. 2021. Do prompt-based models really understand the meaning of their prompts? ArXiv, abs/2109.01247. find our card
  34. Chih-Hsuan Wei, Hung-Yu Kao, and Zhiyong Lu. 2013. PubTator: a web-based text mining tool for assisting biocuration. Nucleic Acids Research, 41(W1):W518–W522. find
  35. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners. CoRR, abs/2109.01652. find
  36. Jie Yang, Yue Zhang, Linwei Li, and Xingxuan Li. 2018. YEDDA: A lightweight collaborative text span annotation tool. In Proceedings of ACL 2018, System Demonstrations, pages 31–36, Melbourne, Australia. Association for Computational Linguistics. find
  37. Seid Muhie Yimam, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013. WebAnno: A flexible, web-based and visually supported system for distributed annotations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 1–… find
  38. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. CoRR, abs/2102.09690. find our card
  39. Figure 6: Most of the datasets have between 5 and 10 prompts. find