Elamur Research⌕
☰
home / papers / 2022 / Instruction Induction: From Few Examples to Natural Language Task Descriptions

Instruction Induction: From Few Examples to Natural Language Task Descriptions

Paper · 2022 · Or Honovich, Uri Shaham, Samuel R. Bowman et al.

Summary

Large language models can solve new tasks from a few in-context examples, but this inference stays hidden inside the model. Honovich et al. ask whether a model can state the inferred task explicitly. They call this the instruction induction challenge: the model sees five input–output pairs and must write a natural language instruction that explains how inputs map to outputs. No fine-tuning is used. A single meta-prompt frames the problem as a puzzle about instructions given to a friend.

The authors built a benchmark of 24 tasks, ranging from spelling and morphosyntax to translation and GLUE-style sentiment and similarity tasks. They gathered human-written reference instructions and evaluated outputs in two ways: BERTScore against those references, and a new execution accuracy metric. Execution accuracy checks whether a model can perform the task zero-shot when given only the induced instruction.

InstructGPT reaches about two-thirds of human performance, while the original GPT-3 shows almost no such ability. The authors suggest that searching over natural language descriptions could become an interpretable learning paradigm in its own right.

Key points

  • Benchmark design: 24 tasks, each with separate induce and execute splits. For every task, 100 prompts are created, each containing 5 randomly sampled demonstrations. Ten verified gold instructions per task were written by college-educated English-speaking annotators. A separate control group of annotators provided a human baseline.
  • Main results: InstructGPT (175B parameters) scored a BERTScore of 44.4 against 60.0 for humans, and an execution accuracy of 43.6 against 66.4. This equals roughly 65.7% of human execution performance, compared with 9.8% for the original GPT-3. On about half the tasks, InstructGPT approaches or matches human performance.
  • Scale and instruction tuning both matter: Smaller InstructGPT variants and the base GPT-3 performed far worse. The ability appears to require both large scale and instruction tuning.
  • Typical failures:
    • On Passivization, 98% of InstructGPT's outputs only described swapping subject and object.
    • On Antonyms, most outputs said to "reverse" the input, which the execution model applied literally by reversing the letters.
  • Why execution accuracy helps: GPT-3 often produced generic but well-formed instructions. These inflated its BERTScore, whereas execution accuracy correctly penalized them. Code and data are released at github.com/orhonovich/instruction-induction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States219 works · 103,487 citations🎓Tel Aviv UniversityTel Aviv · Israel89 works · 11,326 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Or Honovich, Uri Shaham, Samuel R. Bowman and Omer Levy. 2022. Instruction Induction: From Few Examples to Natural Language Task Descriptions. arXiv:2205.10782.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2205.10782 · arXiv 2205.10782

@misc{honovich2022instruction,
  title = {{Instruction Induction: From Few Examples to Natural Language Task Descriptions}},
  author = {Or Honovich, Uri Shaham, Samuel R. Bowman and Omer Levy},
  year = {2022},
  doi = {10.48550/arxiv.2205.10782},
  eprint = {2205.10782},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.10782}
}

References

43 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. BIG-bench collaboration. Beyond the imitation game: Measuring and extrapolating the capabilities of language models., 2021. URL https://github.com/google/BIG-bench/. ↗
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  3. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. ↗
  4. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada, August…
  5. Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via language model in-context tuning, 2021. URL https://arxiv.org/abs/2110.07814. ↗
  6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — PaLM: Scaling Language Modeling with Pathways
  7. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Di… — GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
  8. Avia Efrat and Omer Levy. The turking test: Can language models understand instructions?, 2020. URL https://arxiv.org/abs/2010.11982. ↗
  9. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LR…
  10. C. Fellbaum. WordNet: An Electronic Lexical Database. MIT Press, 1998. find
  11. Deep Ganguli, Danny Hernandez, Liane Lovitt, Nova DasSarma, Tom Henighan, Andy Jones, Nicholas Joseph, Jackson Kernion, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Scott Johnston, Shauna Kravec, Neel Nand… — Predictability and Surprise in Large Generative Models
  12. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=XPZIaotutsD. ↗
  13. Nora Kassner and Hinrich Schütze. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811–7818, Online, July 2020. Association for Computational Linguistics. do…
  14. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, 2021. URL https://arxiv.org/abs/2107.13586. ↗
  15. Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy, July 2019. Association for Computational…
  16. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Byj72udxe. ↗
  17. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetaICL: Learning to learn in context. In NAACL-HLT, 2022a. find
  18. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint, 2022b.
  19. Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. Reframing instructional prompts to gptk’s language, 2021. URL https://arxiv.org/abs/2109.07830. ↗
  20. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470–3487, Dublin, Ireland, May… find
  21. Nikita Nangia, Adina Williams, Angeliki Lazaridou, and Samuel Bowman. The RepEval 2017 shared task: Multi-genre natural language inference with sentence representations. In Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP, pages 1–10, Copenhagen, Denmark, September…
  22. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. In Deep Learning for Code Wor…
  23. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — Training language models to follow instructions with human feedback
  24. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua…
  25. Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (L…
  26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, November 2016. Association for Computational Ling…
  27. Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA ’21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380959. doi: 10.…
  28. Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning, 2021. URL https://arxiv.org/abs/2112.08633. ↗
  29. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chan… — Multitask Prompted Training Enables Zero-Shot Task Generalization
  30. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642,…
  31. Robyn Speer and Catherine Havasi. Representing general relational knowledge in ConceptNet 5. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 3679–3686, Istanbul, Turkey, May 2012. European Language Resources Association (ELRA). URL http://w… find
  32. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. oLMpics-on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743–758, 2020. doi: 10.1162/tacl_a_00342. URL https://aclanthology.org/2020.tacl-1.48. ↗
  33. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Bruss…
  34. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors,…
  35. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2022a. URL https://arxiv.org/abs/2203.11171. ↗
  36. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv, 2022b. find
  37. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018. find
  38. Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts?, 2021.
  39. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a. URL https://openreview.net/forum?id= gEZrGCozdqR. ↗
  40. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models, 2022b. — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  41. Eric Zelikman, Yuhuai Wu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022. URL https://arxiv.org/abs/2203.14465. ↗
  42. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In ICLR 2020, 2020. find
  43. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML, pages 12697–12706, 2021. URL http://proceedings.mlr.press/v139/zhao21c.html. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/honovich-2022-instruction-induction-few-examples-natural/