Elamur Research
☰
home / papers / 2021 / Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm

Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2102.07350

Summary

This exploratory paper challenges the common view that GPT-3 learns new tasks at inference time from a handful of in-context examples. The authors argue that few-shot demonstrations mostly help the model identify which task it already knows, a process they call task location, rather than serving as meta-learning. They test this with a single case study, WMT'14 French-to-English translation, scored with BLEU via SacreBLEU on the Babbage and Curie models from the OpenAI API. Carefully worded 0-shot prompts matched or beat few-shot prompts, while adding a single example sometimes lowered scores. From this the authors develop a natural-language, author-modelling approach to writing prompts. It includes narrative and cultural framing, prompts that make the model reason step by step before giving an answer, and metaprompts that have the model write its own task prompts. The work matters because it suggests that standard benchmarks underestimated the 0-shot abilities of large language models. It also proposes better ways to evaluate them.

Key points

  • On Curie, the "Simple Colon" 0-shot prompt (language names plus a colon) scored 33.3 BLEU, compared with 27.9 for the reproduced 10-shot OpenAI-style prompt and 18.7 for the reproduced OpenAI 0-shot prompt.
  • On Babbage, the "Master Translator" 0-shot prompt reached 26.5 BLEU, above the reproduced 10-shot result of 25.1. The 64-shot setting could not be replicated because of API limits.
  • With the Simple Colon format, moving from 0-shot to 1-shot reduced BLEU (Babbage 23.5 to 18.0; Curie 33.3 to 27.6). The authors attribute this to semantic contamination, where the model treats the example as part of the content rather than as a format guide.
  • Many failures of the original 0-shot prompt were catastrophic: the model did not attempt translation at all and instead continued in French or produced blanks.
  • For benchmarking, the authors propose:
    • reporting scores both with and without catastrophic failures,
    • evaluating with serializing or fill-in-the-blank metaprompts (illustrated on SAT analogies),
    • using text-based games to probe world-modelling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

229 papers of our library cite it (of the 140673 whose bibliographies we read); 25 of them have a page here, 204 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2102.07350 · arXiv 2102.07350

@misc{reynolds2021prompt,
  title = {{Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm}},
  author = {Reynolds, Laria and McDonell, Kyle},
  year = {2021},
  doi = {10.48550/arxiv.2102.07350},
  eprint = {2102.07350},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2102.07350}
}

References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ondrej Bojar et al. “Findings of the 2014 Workshop on Statistical Machine Translation”. In: Proceedings of the Ninth Workshop on Statistical Machine Translation. Baltimore, Maryland, USA: Association for Computational Linguistics, June 2014, pp. 12–58. Language models for evaluations find our card
  2. Gwern Branwen. “GPT-3 Creative Fiction”. In: (2020). As language models become more powerful, it becomes conceivable to use other language models to evaluate the quality of responses to open-ended find
  3. Tom B Brown et al. “Language models are few-shot learners”. In: arXiv preprint arXiv:2005.14165 (2020). 5.2.3 find
  4. Marc-Alexandre Côté et al. “TextWorld: A Learning Environment for Text-based Games”. In: (2019). arXiv: 1806.11532 [cs.LG]. [16] Ben Krause et al. “GeDi: Generative Discriminator Guided Sequence Generation”. In: arXiv preprint arXiv:2009.06367 (2020). find our card
  5. Jacob Devlin et al. “Bert: Pre-training of deep bidirectional transformers for language understanding”. In: arXiv preprint arXiv:1810.04805 (2018). [17] Xiang Lisa Li and Percy Liang. “Prefix-Tuning: Optimizing Continuous Prompts for Generation”. In: arXiv preprint arXiv:2101.00190 (2021). find
  6. Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical Neural Story Generation. 2018. arXiv: 1805.04833 [cs.CL]. [18] Jiangming Liu and Matt Gardner. “Multi-Step Inference for Reasoning Over Paragraphs”. In: arXiv preprint arXiv:2004.02995 (2020). find
  7. Zhe Gan et al. Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog. 2019. arXiv: 1902.00579 [cs.CV]. url: https://arxiv.org/abs/1902.00579. [19] Matt Post. “A Call for Clarity in Reporting BLEU Scores”. In: Proceedings of the Third Conference on Machine Translation: Research Papers.… ↗ our card
  8. Leo Gao. “Building AGI Using Language Models”. In: leogao.dev (2020). url: https://bit.ly/3rViLGk. ↗
  9. Tianyu Gao, Adam Fisch, and Danqi Chen. Making Pre-trained Language Models Better Few-shot Learners. 2020. arXiv: 2012.15723 [cs.CL]. [20] Zachary Robertson. You Can Probably Amplify GPT3 Directly. 2020. url: https://bit.ly/3tXT7Cw. [21] Arram Sabeti. GPT-3: Using Fiction to Demonstrate How Prompts… ↗ our card
  10. Dan Hendrycks et al. “Measuring massive multitask language understanding”. In: arXiv preprint arXiv:2009.03300 (2020). url: https://arxiv.org/abs/2009.03300. [22] Taylor Shin et al. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. 2020. arXiv: 2010.15980 [cs… ↗ our card
  11. Ari Holtzman et al. The Curious Case of Neural Text Degeneration. 2020. arXiv: 1904.09751 [cs.CL]. [23] Latitude Team. World Creation by Analogy. 2020. url: https://bit.ly/2N4vXK0. ↗
  12. Jeremy Howard and Sebastian Ruder. “Universal language model fine-tuning for text classification”. In: arXiv preprint arXiv:1801.06146 (2018). url: https://arxiv.org/abs/1801.06146. [24] Lilian Wang. “Controllable Neural Text Generation”. In: (2021). url: https://bit.ly/3pl2eKa. [25] Qinyuan Ye and… ↗
  13. KaryoKleptid. Seems to work. 2020. url: https://bit.ly/37dA1hY. ↗
  14. KaryoKleptid. Teaching GPT-3 to do a brute force ’for loop’ checking answers. 2020. url: https://bit.ly/2N7khX1. [26] Jianxing Yu et al. “Low-resource generation of multi-hop reasoning questions”. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020, pp.… ↗
  15. Nitish Shirish Keskar et al. “CTRL: A Conditional Transformer Language Model for Controllable Generation”. In: CoRR abs/1909.05858 (2019). arXiv: 1909.05858. url: http://arxiv.org/abs/1909.05858. [27] Eliezer Yudkowsky. “Rationalization”. In: lesswrong.com (2007). url: https://bit.ly/3pmYt6I. ↗ our card