Elamur Research
☰
home / papers / 2021 / True Few-Shot Learning with Language Models

True Few-Shot Learning with Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2105.11447

Summary

Reported few-shot results for pretrained language models often depend on large held-out sets for choosing prompts, hyperparameters and other design decisions. This weakens the claim that only a handful of labeled examples were needed. Perez, Kiela and Cho distinguish three settings: multi-distribution, tuned, and true few-shot learning. In true few-shot learning, every decision must be made using only the small training set. They then test how existing methods perform when model selection is restricted to those few examples. They use two standard criteria: cross-validation (including leave-one-out) and minimum description length, estimated through online coding. They also try a more conservative CV variant that penalizes variance. For both prompt selection across many LMs and hyperparameter selection for ADAPET, the criteria beat random choice only slightly and fall far behind selection with held-out data. They also often prefer models that do worse than a random pick. The authors conclude that earlier work probably overstated LM few-shot ability, that model selection is a central obstacle, and that stricter reporting protocols are needed.

Key points

  • Prompt selection was evaluated on 9 LMs spanning about three orders of magnitude in size, using 3 classification tasks and 41 tasks from the LAMA benchmark.
  • For ADAPET, a state-of-the-art few-shot method, true few-shot hyperparameter selection lowered performance by 2–10% across 8 tasks.
  • The findings held when the authors changed the number of examples, the compute budget, or how conservative the selection criterion was, so the problem is not limited to one configuration.
  • The authors recommend that papers report every candidate prompt and hyperparameter, count validation examples as part of the few-shot budget, run test evaluation ideally only once, and avoid reusing hyperparameters tuned on benchmark validation sets such as SuperGLUE.
  • The paper also notes that CLIP's zero-shot ImageNet prompts were tuned on the training set (1.28M examples), which contributed about 5% accuracy. Code is available at github.com/ethanjperez/true_few_shot.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

218 papers of our library cite it (of the 140673 whose bibliographies we read); 37 of them have a page here, 181 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2105.11447 · arXiv 2105.11447

@misc{perez2021true,
  title = {{True Few-Shot Learning with Language Models}},
  author = {Perez, Ethan and Kiela, Douwe and Cho, Kyunghyun},
  year = {2021},
  doi = {10.48550/arxiv.2105.11447},
  eprint = {2105.11447},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2105.11447}
}

References

91 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. find
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. https://github.com/openai/CLIP/blob/m… ↗
  3. Timo Schick and Hinrich Schütze. Exploiting cloze questions for few-shot text classification and natural language inference. Computing Research Repository, arXiv:2001.07676, 2020. find our card
  4. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? TACL, 8:423–438, 2020. find our card
  5. Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020. find our card
  6. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models, 2021. find our card
  7. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity, 2021. find our card
  8. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, L. Carin, and W. Chen. What makes good in-context examples for gpt-3? arXiv, abs/2101.06804, 2021. find our card
  9. Timo Schick and Hinrich Schütze. It’s not just size that matters: Small language models are also few-shot learners. Computing Research Repository, arXiv:2009.07118, 2020. find our card
  10. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. Rissanen data analysis: Examining dataset characteristics via description length. In ICML, 2021. find
  11. Timo Schick and H. Schutze. Few-shot text generation with pattern-exploiting training. arXiv, abs/2012.11926, 2020. — [arXiv:2012.11926](https://arxiv.org/abs/2012.11926) find
  12. Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. arxiv preprint arXiv:2103.11955, 2021. find
  13. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. find our card
  14. Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. Entailment as few-shot learner, 2021. find
  15. Teven Le Scao and Alexander M. Rush. How many data points is a prompt worth?, 2021. find our card
  16. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv, abs/1910.01108, 2019. find
  17. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In EMNLP, pages 2463–2473, Hong Kong, China, November 2019. ACL. find our card
  18. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, NeuRIPS, volume 29. Curran Associates, Inc., 2016. find
  19. Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, NeuRIPS, volume 30. Curran Associates, Inc., 2017. find
  20. S. Ravi and H. Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017. find
  21. Ke Li and Jitendra Malik. Learning to optimize. arXiv, abs/1606.01885, 2017. — [arXiv:1606.01885](https://arxiv.org/abs/1606.01885) find
  22. David M. Allen. The relationship between variable selection and data agumentation and a method for prediction. Technometrics, 16(1):125–127, 1974. find
  23. M. Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society. Series A (Methodological), 36:111–133, 1974. find
  24. Seymour Geisser. The predictive sample reuse method with applications. Journal of the American Statistical Association, 70(350):320–328, 1975. find
  25. Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001. find
  26. Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, volume 70 of ICML’17, pages 1126–1135. JMLR.org, 2017. find
  27. Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, NeurIPS, volume 32. Curran Associates, Inc., 2019. find
  28. J. Rissanen. Modeling by shortest data description. Automatica, 14(5):465 – 471, 1978. find
  29. J. Rissanen. Universal coding, information, prediction, and estimation. IEEE Transactions on Information Theory, 30(4):629–636, 1984. find
  30. A. P. Dawid. Present position and potential developments: Some personal views: Statistical theory: The prequential approach. Journal of the Royal Statistical Society. Series A (General), 147(2):278–292, 1984. find
  31. Peter Grünwald. A tutorial introduction to the minimum description length principle. CoRR, math.ST/0406077, 06 2004. find
  32. Léonard Blier and Yann Ollivier. The description length of deep learning models. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, NeuRIPS, volume 31, pages 2216–2226. Curran Associates, Inc., 2018. find
  33. Alselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K. Warmuth. Occam’s razor. Inf. Process. Lett., 24(6):377–380, April 1987. find
  34. Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. CoRR, abs/2104.06644, 2021. find our card
  35. P. J. Phillips, Hyeonjoon Moon, S. A. Rizvi, and P. J. Rauss. The feret evaluation methodology for face-recognition algorithms. TPAMI, 22(10):1090–1104, 2000. find
  36. Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson, editors, Fairness, Accountability and Transparency, volume 81 of PMLR, pages 77–91, New York, NY, USA, 23–24 Feb 2018. PMLR. find
  37. Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in data-driven dialogue systems. In AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, page 123–129, New York, NY, USA, 2018. Association for Computing… find
  38. Chandra Khatri, Behnam Hedayatnia, Anu Venkatesh, Jeff Nunn, Yi Pan, Qing Liu, Han Song, Anna Gottardi, Sanjeev Kwatra, Sanju Pancholi, Ming Cheng, Qinglang Chen, Lauren Stubel, Karthik Gopalakrishnan, Kate Bland, Raefer Gabriel, Arindam Mandal, Dilek Hakkani-Tür, Gene Hwang, Nate Michel, Eric King,… — [arXiv:1812.10757](https://arxiv.org/abs/1812.10757) find
  39. Javier García, Fern, and o Fernández. A comprehensive survey on safe reinforcement learning. JMLR, 16(42):1437–1480, 2015. find
  40. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. CoRR, abs/1606.06565, 2016. find our card
  41. Sumio Watanabe. Asymptotic equivalence of bayes cross validation and widely applicable information criterion in singular learning theory. JMLR, 11(116):3571–3594, 2010. find
  42. H. Akaike. A new look at the statistical model identification. TACON, 19(6):716–723, 1974. find
  43. C. L. Mallows. Some comments on cp. Technometrics, 15(4):661–675, 1973. find
  44. Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Carlos A. Coello Coello, editor, Learning and Intelligent Optimization, pages 507–523, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. find
  45. James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyperparameter optimization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger, editors, NeuRIPS, volume 24. Curran Associates, Inc., 2011. find
  46. Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, NeuRIPS, volume 25. Curran Associates, Inc., 2012. find
  47. Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. Chapter 15 - evolving deep neural networks. In Robert Kozma, Cesare Alippi, Yoonsuck Choe, and Francesco Carlo Morabito, editors,… find
  48. Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. AAAI, 33(01):4780–4789, Jul. 2019. find
  49. Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR. OpenReview.net, 2017. find our card
  50. J. Larsen, L.K. Hansen, C. Svarer, and M. Ohlsson. Design and regularization of neural networks: the optimal use of a validation set. In Neural Networks for Signal Processing VI. IEEE Signal Processing Society Workshop, pages 62–71, 1996. find
  51. Yoshua Bengio. Gradient-Based Optimization of Hyperparameters. Neural Computation, 12(8):1889–1900, 08 2000. find
  52. O. Chapelle, V. Vapnik, O. Bousquet, and S. Mukherjee. Choosing multiple parameters for support vector machines. Machine Learning, 46:131–159, 2004. find
  53. Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019. find
  54. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In EMNLP, pages 4222–4235, Online, November 2020. ACL. find our card
  55. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too, 2021. find our card
  56. Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [MASK]: learning vs. learning to recall. CoRR, abs/2104.05240, 2021. find our card
  57. Nina Poerner, Ulli Waltinger, and Hinrich Schütze. E-BERT: Efficient-yet-effective entity embeddings for BERT. In Findings of EMNLP, pages 803–818, Online, November 2020. ACL. find
  58. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… find
  59. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and S… find our card
  60. Elena Voita and Ivan Titov. Information-theoretic probing with minimum description length. In EMNLP, pages 183–196, Online, November 2020. ACL. find our card
  61. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors,… find our card
  62. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In ICLR, 2020. find
  63. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186, Minneapolis, Minnesota, June 2019. ACL. find
  64. Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. In ICLR, 2020. find
  65. Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in NLP. CoRR, abs/2104.08835, 2021. find
  66. Rich Caruana. Learning many related tasks at the same time with backpropagation. In G. Tesauro, D. Touretzky, and T. Leen, editors, NeuRIPS, volume 7. MIT Press, 1995. find
  67. Rich Caruana. Multitask learning. Machine Learning, 28(1):41–75, July 1997. find
  68. Jason Phang, Thibault Févry, and Samuel R. Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR, abs/1811.01088, 2018. find our card
  69. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In ACL, pages 4487–4496, Florence, Italy, July 2019. ACL. find our card
  70. Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. A surprisingly robust trick for the Winograd schema challenge. In ACL, pages 4837–4842, Florence, Italy, July 2019. ACL. find
  71. Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, NeuRIPS, volume 33, pages 6256–6268. Curran Associates, Inc., 2020. find our card
  72. Jiaao Chen, Zichao Yang, and Diyi Yang. MixText: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In ACL, pages 2147–2157, Online, July 2020. ACL. find
  73. Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. Generative data augmentation for commonsense reasoning. In Findings of EMNLP, pages 1008–1025, Online, November 2020. ACL. find
  74. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In ICLR, 2018. find our card
  75. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In ICLR, 2018. find our card
  76. Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. Unsupervised question decomposition for question answering. In EMNLP, pages 8864–8880, Online, November 2020. ACL. find
  77. Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. CoRR, abs/1804.09170, 2018. — [arXiv:1804.09170](https://arxiv.org/abs/1804.09170) find
  78. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In EMNLP, pages 2185–2194, Hong Kong, China, November 2019. ACL. find
  79. Allyson Ettinger. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. TACL, 8:34–48, 2020. find our card
  80. Aki Vehtari, Andrew Gelman, and Jonah Gabry. Practical bayesian model evaluation using leave-one-out cross-validation and waic. Statistics and Computing, 27(5):1413–1432, September 2017. find
  81. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. CoRR, abs/2002.06305, 2020. find
  82. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In NAACL, pages 2924–2936, Minneapolis, Minnesota, June 2019. ACL. find our card
  83. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. Sinn und Bedeutung, 23(2):107–124, July 2019. find our card
  84. Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Joaquin Quiñonero-Candela, Ido Dagan, Bernardo Magnini, and Florence d’Alché Buc, editors, Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recogni… find
  85. Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. In Second PASCAL Challenges Workshop on Recognising Textual Entailment, 2006. find
  86. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recognizing textual entailment challenge. In ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, RTE ’07, page 1–9, USA, 2007. ACL. find
  87. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In TAC, 2009. find
  88. Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: The word-in-context dataset for evaluating context-sensitive meaning representations. In NAACL. ACL, 2019. find our card
  89. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In KR, KR’12, page 552–561. AAAI Press, 2012. find
  90. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In NAACL. ACL, 2018. find our card
  91. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810.12885, 2018. find our card