🪄 Few-shot learning

About 18 minutes to read · every paper named here has its own page in this library

What it is

A child who has seen one zebra can usually recognise the next one. A conventional neural network cannot. It has to be trained on thousands or millions of labelled pictures before it tells zebras from horses reliably.

FSL asks how to close that gap: how a system can pick up a new category, word, language pair or task from a few examples instead of a large annotated dataset. The question matters whenever labels are expensive. That includes:

• rare diseases;
• clinical notes that only experts can annotate (Agrawal et al. 2022);
• languages with little digital text;
• tasks that change too quickly for anyone to collect data.

The field has two main traditions, and they meet in modern deep learning.

• The first is meta-learning, "learning to learn". A model is trained on many small tasks so that it becomes good at solving one more small task quickly. This tradition began mostly in computer vision.
• The second grew out of large language models. Here a single pretrained network is given a few worked examples inside its input text and continues the pattern without any change to its weights. This was named in-context learning when GPT-3 was introduced (Brown et al. 2020).

Between the two sits a family of methods that fine-tune a pretrained language model on a few examples. They first rephrase the task so that it resembles the model's pretraining objective (Schick & Schütze 2021).

The standard way to state a few-shot problem is the N-way K-shot setting. A task has N classes, and the learner receives K labelled examples of each. These N × K examples form the support set. The learner is then scored on fresh query examples from the same N classes.

Take a 5-way 1-shot task built from photographs. The learner sees one photo each of five bird species it has never encountered and must label new photos of those five species. In a 5-way 5-shot task it would see five photos per species.

Two limiting cases sit at the low end of the setting:

• One-shot learning is the case K = 1.
• Zero-shot learning removes the examples entirely. Only a description of the task or the labels is given. This can be a natural-language instruction (Brown et al. 2020) or a label rephrased as an entailment hypothesis (Yin et al. 2019).

How it works

Episodes and transfer

Every FSL method does most of its learning before it sees the few examples.

• Meta-learning methods train on episodes: many simulated N-way K-shot tasks drawn from a large pool of base classes. The skill being trained is the ability to solve a fresh episode (Vinyals et al. 2016).
• Pretrained language models acquire broad competence from raw text. The few examples then serve mainly to identify the task, and some researchers argue they do little actual teaching (Reynolds & McDonell 2021). Evidence for this view is discussed under in-context learning in the Methods section.

Learning a metric: prototypes

Metric-based methods learn an embedding space in which a simple comparison suffices. Prototypical networks are the clearest example (Snell et al. 2017). Each class is summarised by a single point, its prototype, computed as the mean of its embedded support examples:

c_k = (1 / |S_k|) · Σ_{(x_i, y_i) ∈ S_k} f_φ(x_i)

p(y = k | x) = exp(−d(f_φ(x), c_k)) / Σ_{k'} exp(−d(f_φ(x), c_{k'}))

The symbols are as follows:

• S_k is the set of support examples labelled with class k, and |S_k| is their number (K in a balanced task).
• x_i is an input and y_i its label.
• f_φ is the embedding network with parameters φ.
• c_k is the resulting prototype of class k.
• d is a distance, commonly squared Euclidean distance.

The second line turns negative distances into a probability over classes with a softmax. A query x is therefore assigned to the class whose prototype lies nearest. Training adjusts φ across many episodes so that this nearest-prototype rule works on classes the network has never seen. Matching networks, which preceded this approach, compare a query with every support example through an attention mechanism rather than through class means (Vinyals et al. 2016).

Learning to adapt: MAML

Optimisation-based methods learn a starting point from which ordinary gradient descent adapts quickly. Model-Agnostic Meta-Learning (MAML) does this for any model trained by gradients (Finn et al. 2017):

θ'_i = θ − α · ∇_θ L_{T_i}(θ)
θ ← θ − β · ∇_θ Σ_i L_{T_i}(θ'_i)

The symbols are as follows:

• θ are the shared initial parameters.
• T_i is one sampled training task.
• L_{T_i} is that task's loss, computed on its support data in the first line and on its query data in the second.
• α is the inner-loop step size and β the outer-loop (meta) step size.

The first line simulates fine-tuning on a single task. The second updates the initialisation so that this short fine-tuning yields low loss on held-out examples of the same task. Because the second gradient passes through the first, exact MAML involves second-order derivatives. These can be dropped as an approximation, as was done when MAML was carried over to translation (Gu et al. 2018).

Learning from the prompt

In-context learning needs no gradients at all. A prompt for GPT-3 contains a task description, a few input–output demonstrations and a new input. The model's continuation is read off as the answer (Brown et al. 2020).

The GPT-3 paper separated three settings by the number of demonstrations:

• zero-shot, with an instruction only;
• one-shot;
• few-shot, in which the model received as many examples as its context window allowed, typically between 10 and 100 (Brown et al. 2020).

A survey of the area formalises the process as scoring candidate answers conditioned on the demonstrations and an optional instruction. It also separates this from classical few-shot learning, in which parameters are updated (Dong et al. 2023).

The same idea carries beyond text. Flamingo interleaves images and text in its prompt and adapts to new visual tasks from a few such demonstrations (Alayrac et al. 2022).

Pattern-based fine-tuning

A middle route rewrites each labelled example as a fill-in-the-blank sentence that a masked language model can complete. In Pattern-Exploiting Training (PET), two components do the rewriting (Schick & Schütze 2021):

• A pattern turns an input into a cloze question with one mask.
• A verbalizer maps each label to a vocabulary word.

For a review, the pattern might append "It was ___." The verbalizer might map positive to "great" and negative to "terrible". The model is then fine-tuned to predict the right word, so even a few examples steer an objective it already knows well. This is the core of prompt engineering applied to fine-tuning rather than to frozen models.

History

The ambition is older than deep learning:

• Meta-learning was formulated in work by Schmidhuber (1987) and in the "learning to learn" programme of Thrun and Pratt (1998).
• Fei-Fei Li and colleagues (2006) gave one-shot object recognition its modern framing.
• Lake and colleagues built the Omniglot handwritten-character collection, first described in 2011. In 2015 they used it to compare human and machine concept learning.
• Koch and colleagues (2015) used siamese networks for one-shot image matching.

Matching networks were among the first deep meta-learning methods trained on simulated episodes (Vinyals et al. 2016). Prototypical networks (Snell et al. 2017) and MAML (Finn et al. 2017) followed in 2017. They became the reference points for the metric-based and optimisation-based families.

Natural language processing (NLP) took up the problem through transfer between tasks and languages:

• A multilingual translation system marked each input with a target-language token. It could then translate between language pairs it had never been trained on, a form of zero-shot transfer (Johnson et al. 2017).
• MetaNMT treated each language pair as one task in a MAML setup. It reached 22.04 BLEU, an n-gram overlap score for translation, on Romanian–English after fine-tuning on roughly 600 sentence pairs (Gu et al. 2018).
• A shared sentence encoder covering 93 languages let a classifier trained on English work in other languages without further training (Artetxe & Schwenk 2018).
• Zero-shot text classification was recast as textual entailment, with each label rewritten as a hypothesis (Yin et al. 2019).
• Relation extraction was recast as reading comprehension (Levy et al. 2017).

The turning point came in 2020. GPT-3, a 175-billion-parameter autoregressive Transformer, was evaluated purely through prompts. On some benchmarks its few-shot results approached the best fine-tuned systems. Across an aggregate of 42 accuracy-based benchmarks, its few-shot performance grew with model size faster than its zero-shot performance (Brown et al. 2020). That result shifted attention from training specialised models to prompting general ones, and it ties few-shot ability to scaling laws.

Several lines of work followed in quick succession:

• 2021, smaller models. Groups showed that models far smaller than GPT-3 could compete when fine-tuned on cloze-style reformulations (Schick & Schütze 2021; Gao et al. 2021).
• 2021, fragility. Other groups documented how unstable prompting was and how easily its evaluation could be inflated (Zhao et al. 2021; Perez et al. 2021).
• 2021, instruction tuning. Multitask training on tasks phrased as instructions improved zero-shot transfer (Wei et al. 2021; Sanh et al. 2021).
• 2022, reasoning. Chain-of-thought prompting (Wei et al. 2022) and its zero-shot variant (Kojima et al. 2022) made multi-step reasoning a central few-shot application.
• 2023, consolidation and critique. Surveys organised the area (Dong et al. 2023), and studies of faithfulness questioned what the written reasoning actually reflects (Turpin et al. 2023).

Methods

Metric-based meta-learning

Metric methods learn embeddings so that classification reduces to a nearest-neighbour or nearest-prototype decision (Snell et al. 2017; Vinyals et al. 2016). They have two practical strengths:

• They are cheap at test time, since a new class costs one forward pass per support example.
• They need no fine-tuning.

Their main limitation is that the embedding must already separate the new classes. When the new domain differs strongly from the training classes, a fixed metric has little to work with.

The same idea reappears in language work whenever examples are compared in an embedding space. Two examples are:

• zero-shot entity linking, which embeds mentions and entity descriptions in one space (Wu et al. 2020);
• the selection of prompt demonstrations by semantic similarity (Gao et al. 2021).

Optimisation-based meta-learning

MAML and its relatives learn an initialisation rather than a metric. They adapt every parameter with a few gradient steps. In principle they can therefore handle new tasks that need more than a re-weighting of familiar features (Finn et al. 2017).

The cost is a nested training loop that is expensive and sensitive to hyperparameters. In MetaNMT, the choice of validation task used for early stopping noticeably changed final quality (Gu et al. 2018).

In-context learning and the craft of demonstrations

Once a model is large enough, the few-shot learner is the prompt. Much research therefore concerns which demonstrations to show and in what form.

• Retrieval of demonstrations. EPR trains a small dense retriever to find training examples that raise the probability of the correct output when placed in the prompt. It was tested with GPT-Neo, an openly released GPT-style model, as both scorer and inference model. On MTOP, a task-oriented semantic-parsing benchmark, it reached 64.2% exact match, against 57% for the strongest baseline (Rubin et al. 2021).
• Ordering. The order of the same examples can move accuracy from near state-of-the-art to chance. One fix ranks candidate orders by the entropy of the labels the model predicts on a self-generated probing set. Over eleven text classification tasks, this gave an average relative improvement in classification performance of about 13% (Lu et al. 2022).
• Calibration. Language models favour labels that appear often, appear last, or are common tokens. Contextual calibration estimates this bias from a content-free input such as "N/A" and corrects it. It raised average accuracy by up to 30.0 points absolute (Zhao et al. 2021).
• What the demonstrations carry. Replacing gold labels in demonstrations with random ones cost only about 0–5% absolute across 26 datasets. What mattered was exposure to the label space, the input distribution and the format. This suggests demonstrations mostly activate abilities acquired in pretraining (Min et al. 2022).
• Zero-shot as a competitor. Carefully worded zero-shot prompts can match or beat few-shot prompts. In translation experiments, adding a single example sometimes lowered BLEU (Reynolds & McDonell 2021).

These methods connect few-shot learning to retrieval-augmented generation:

• Atlas pairs a language model with a document retriever to learn knowledge-intensive tasks from few examples (Izacard et al. 2022).
• HyDE uses a prompted model to write a hypothetical passage and retrieves real documents near it, without any relevance labels (Gao et al. 2022).

Chain of thought and decomposition

Chain-of-thought prompting extends each demonstration from an input–output pair to an input, a written line of reasoning, and an output (Wei et al. 2022). With PaLM 540B and eight such exemplars, accuracy on the GSM8K math word problems rose to about 57%, against about 18% with standard prompting. The benefit appeared only in models of roughly 100 billion parameters and above (Wei et al. 2022).

Several variants change how the reasoning is elicited or which examples are shown:

• Zero-shot chain of thought drops the examples and appends a single trigger phrase asking the model to think step by step. It was tested with InstructGPT, a GPT-3 variant tuned to follow instructions. This raised MultiArith accuracy from 17.7% to 78.7% and GSM8K accuracy from 10.4% to 40.7% (Kojima et al. 2022).
• Least-to-most prompting first decomposes a problem into simpler subquestions and then answers them in sequence. It reached at least 99% on every split of the SCAN compositional benchmark with 14 exemplars (al. 2022).
• Auto-CoT clusters questions and has the model write rationales for one representative per cluster. It matched or beat hand-written chains on ten reasoning benchmarks (Zhang et al. 2022).
• Complexity-based prompting prefers exemplars with more reasoning steps (Fu et al. 2022).

The same few-shot machinery drives planners for embodied agents (Song et al. 2022; Huang et al. 2022) and text-to-SQL pipelines (Pourreza & Rafiei 2023). These link the topic to reasoning and semantic parsing.

Pattern-based and parameter-efficient fine-tuning

PET fine-tunes one masked language model per pattern–verbalizer pair on the few labelled examples. The resulting ensemble labels a pool of unlabelled text, and a final classifier is distilled from those soft labels. The iterative variant, iPET, repeats this over generations (Schick & Schütze 2021).

A follow-up allowed labels that span several tokens. It was tested with ALBERT and 32 labelled examples per task on SuperGLUE, a suite of difficult English language-understanding tasks. PET and iPET outperformed GPT-3 on average while using about 0.1% of its parameters (Schick & Schütze 2021).

LM-BFF automates the recipe (Gao et al. 2021):

• It generates templates with T5.
• It searches for label words.
• It appends one similar demonstration per class to each input.

With 16 examples per class, this improved over standard fine-tuning by 11 points on average across 15 English tasks (Gao et al. 2021).

Prompt tuning learns only continuous "soft prompt" vectors. It lags behind full fine-tuning when data are scarce, unless the prompts are first pre-trained on unlabelled text as in PPT (Gu et al. 2022).

T-Few combines a multitask-prompted backbone with a light rescaling method, (IA)3. It scored 75.8% on the RAFT benchmark, above the 73.5% human baseline. It also used over 1,000 times fewer inference FLOPs (floating-point operations) than in-context learning with GPT-3 (Liu et al. 2022). The trade-off against prompting is therefore partly a matter of efficient models.

Instruction tuning

A related idea trains a model on many tasks phrased as natural-language instructions, so that unseen tasks become easier zero-shot. This was shown with FLAN (Wei et al. 2021).

T0 was trained on prompted English datasets with whole task groups held out. Although it is about 16 times smaller than GPT-3, it matched or surpassed GPT-3 on 9 of 11 held-out datasets (Sanh et al. 2021).

Datasets and benchmarks

Vision-based meta-learning grew up on two collections:

• Omniglot, first described by Lake and colleagues in 2011.
• miniImageNet, a reduced ImageNet split organised into few-shot episodes, introduced with matching networks (Vinyals et al. 2016).

Later suites such as Meta-Dataset (Triantafillou and colleagues, 2020) mixed several image domains to test transfer across them.

In language, few-shot evaluation mostly reuses existing evaluation benchmarks with the training data cut down. A few resources were designed specifically for the setting.

FLEURS uses "few-shot" loosely. About 12 hours of transcribed speech per language is scarce supervision for low-resource and unseen languages. It is not, however, an N-way K-shot task in the sense defined above (Conneau et al. 2022).

Reasoning studies lean on math word problems such as GSM8K and MultiArith and on BIG-Bench Hard tasks (Wei et al. 2022; Kojima et al. 2022; Turpin et al. 2023). Flamingo was assessed on 16 image and video benchmarks. It outperformed fine-tuned state-of-the-art systems on 6 of them while using only 32 task examples (Alayrac et al. 2022).

For multilingual NLP and low-resource languages, cross-lingual transfer studies show two things. Zero-shot transfer from English degrades for distant languages, and a few target-language examples help a great deal.

One study used Universal Dependencies treebanks for dependency parsing, with target languages drawn from eight families and models first fine-tuned on English. Continuing training on just 10 target-language sentences raised the unlabelled attachment score (UAS) by as much as 40 points for some targets. UAS is the share of words given their correct syntactic head (Lauscher et al. 2020).

Applied studies extend the setting to clinical information extraction, including coreference resolution and acronym sense disambiguation (Agrawal et al. 2022).

How results are evaluated, and where they mislead

Few-shot numbers are unusually fragile, and several studies show how they can overstate what a method does.

• Hidden validation data. Many reported results chose prompts and hyperparameters on large held-out sets, which contradicts the premise of having few examples. When selection had to rely on the few training examples alone, the criteria barely beat random choice. These criteria were cross-validation and minimum description length. For ADAPET, a PET-derived few-shot method, hyperparameters chosen this way scored 2–10% lower across 8 tasks than those chosen with held-out data (Perez et al. 2021). The authors recommend counting any validation data within the few-shot budget and reporting every candidate prompt (Perez et al. 2021).
• Variance from example choice and order. On 4-shot SST-2, a binary movie-review sentiment task, reordering the same examples moved GPT-3 2.7B accuracy from 54.3% to 93.4% (Zhao et al. 2021). Good orders also failed to carry over between model sizes (Lu et al. 2022). Different random sets of 32 training examples shifted F1, the harmonic mean of precision and recall, between 57.4 and 84.0. This was on CB, the CommitmentBank inference task (Schick & Schütze 2021). Comparisons are therefore meaningful only on identical examples and with several seeds or splits (Gao et al. 2021).
• Weak baselines. Given the random-label result described under Methods, zero-shot baselines built from unlabelled inputs are stronger than often assumed (Min et al. 2022). Poorly worded zero-shot prompts can make few-shot gains look larger than they are (Reynolds & McDonell 2021).
• Prompts that are not understood. Irrelevant and even misleading templates often let models learn natural language inference about as fast as sensible instructions. The choice of target words mattered more than the instruction itself (Webson & Pavlick 2022).
• Contamination. Benchmark items may already sit in the pretraining corpus, a risk the GPT-3 authors discussed openly (Brown et al. 2020). Flamingo's authors kept a held-out set of benchmarks out of all design decisions to obtain unbiased estimates (Alayrac et al. 2022).
• Plausible but unfaithful reasoning. Biasing cues shifted model answers and cut BIG-Bench Hard accuracy by up to 36%. One such cue was always placing the correct option at "(A)". In a manual review, only 1 of 426 explanations supporting biased answers acknowledged the cue (Turpin et al. 2023).

People and institutions

Several researchers recur across the prompting literature.

Luke Zettlemoyer, of the University of Washington and Meta, co-authored three relevant studies:

• the analysis showing that correct labels in demonstrations matter less than format (Min et al. 2022);
• work on zero-shot entity linking (Wu et al. 2020);
• work on zero-shot relation extraction (Levy et al. 2017).

Mike Lewis, of Meta, co-authored the same demonstration study (Min et al. 2022).

At Google, Denny Zhou led least-to-most prompting (al. 2022). Jason Wei, who later moved to OpenAI, was first author of chain-of-thought prompting (Wei et al. 2022) and of FLAN (Wei et al. 2021). Xuezhi Wang and Quoc Le were among the chain-of-thought co-authors.

Timo Schick and Hinrich Schütze of Ludwig-Maximilians-Universität München developed PET and its multi-token follow-up (Schick & Schütze 2021; Schick & Schütze 2021). Schick was also an author of Atlas (Izacard et al. 2022).

Other contributions include:

• Albert Webson, who later joined Google, co-authored T0 (Sanh et al. 2021) and tested whether models understand their prompts (Webson & Pavlick 2022). M Saiful Bari was also among the T0 authors.
• Sebastian Riedel, who later moved to Google DeepMind and is also at University College London, co-authored the study of prompt-order sensitivity (Lu et al. 2022).
• Jonathan Berant, at Google and Tel Aviv University, co-authored learned demonstration retrieval (Rubin et al. 2021).
• Zhuosheng Zhang, later at Shanghai Jiao Tong University, was first author of Auto-CoT (Zhang et al. 2022).

On the meta-learning side:

• Chelsea Finn, with Pieter Abbeel and Sergey Levine, introduced MAML (Finn et al. 2017).
• Jake Snell, Kevin Swersky and Richard Zemel proposed prototypical networks (Snell et al. 2017).
• Oriol Vinyals and colleagues proposed matching networks (Vinyals et al. 2016).
• Tom Brown and colleagues at OpenAI wrote the GPT-3 paper (Brown et al. 2020).

Among universities, three appear in this work:

• Peking University was home to the authors of the in-context learning survey (Dong et al. 2023).
• Tsinghua University produced pre-trained prompt tuning (Gu et al. 2022).
• Chelsea Finn later continued her work at Stanford University (Mitchell et al. 2023).

Open questions

Why in-context learning works at all is still unsettled. Several explanations compete:

• Induction heads, attention circuits that copy patterns seen earlier in the context (Olsson et al. 2022).
• Bayesian accounts in which demonstrations help the model infer a latent task (Dong et al. 2023).
• Interpretations of in-context learning as an implicit form of gradient descent (Dong et al. 2023).

The random-label result described under Methods (Min et al. 2022) fits the view that prompts mostly locate a skill already acquired in pretraining (Reynolds & McDonell 2021). Whether large models can also learn genuinely new input–label mappings from context remains an interpretability question as much as an empirical one.

A second issue is honest measurement. True few-shot evaluation shows that model selection with few examples is itself unsolved (Perez et al. 2021). Sensitivity to order and wording means a single reported number is rarely trustworthy (Zhao et al. 2021; Lu et al. 2022). Agreed protocols with fixed examples, multiple seeds and contamination checks are still emerging.

A third concerns what the written reasoning means. Chain-of-thought outputs can be wrong even when the answer is right (Wei et al. 2022). They can also hide the cues that really drove a decision (Turpin et al. 2023). This limits their use for auditing and for bias, fairness and safety. Related work shows that prompts may be followed for reasons other than their meaning (Webson & Pavlick 2022).

A fourth is cost and choice of paradigm. In-context learning reprocesses every demonstration for every query. Parameter-efficient fine-tuning can be both cheaper and more accurate (Liu et al. 2022), and the PET results described above point the same way (Schick & Schütze 2021). Which approach suits which task, budget and privacy constraint is not settled. In clinical settings, for example, data often cannot be sent to external APIs (Agrawal et al. 2022).

Finally, FSL ability is uneven across languages and modalities. Multilingual pretraining narrows the gap with English (Lin et al. 2021). Zero-shot transfer still falters for typologically distant languages, however (Lauscher et al. 2020).

Speech shows the same pattern. On FLEURS speech–text retrieval, the baseline averages a score of 76.9 from speech to text. Chinese, Japanese, Korean, Thai and Burmese, however, remain in single digits (Conneau et al. 2022).

Delivering on the field's founding promise, learning from a handful of examples where large datasets will never exist, remains open work. That holds for machine translation, speech and multimodal learning alike.

The library in figures