🎛️ Fine-tuning
Fine-tuning is the practice of taking a model already pretrained on a large, general dataset and continuing its training on a smaller, task- or domain-specific dataset. The model keeps the general knowledge it learned earlier and adapts it to the new purpose.
What it is
Training a large neural network from scratch takes huge amounts of text and compute. Fine-tuning avoids most of that cost. You start from a network whose weights have already been learned on general data, such as a language model trained on web text. You then keep training it with ordinary gradient descent on examples of the job you care about. That job might be text classification, named entity recognition (NER), question answering or answering a user in a chat. Most of what the model needs is already in its weights. The new data only steers it.
In natural language processing, this two-stage recipe of pretraining followed by adaptation is called sequential transfer learning. The NAACL 2019 tutorial Transfer Learning in Natural Language Processing set out two ways to carry out the adaptation stage (Ruder et al. 2019). In feature extraction, the pretrained network stays frozen and only supplies representations to a separate model. In fine-tuning, the pretrained weights themselves are updated. On CoNLL-2003 NER, frozen BERT-base features (its top four layers concatenated) scored 96.1 F1 on the development set. That is just 0.3 below fully fine-tuned BERT-base at 96.4 (Devlin et al. 2019).
The key idea
The central assumption is that general pretraining already captures most of the structure a task needs. Adaptation then needs only a small move away from the pretrained weights. BERT shows the idea in its simplest form. One bidirectional Transformer encoder is pretrained with masked-token and next-sentence objectives. Each downstream task then gets just one added output layer, and the whole network is trained end to end (Devlin et al. 2019). T5 pushed the same logic further by casting every task as text-to-text, so that a single loss and a single set of hyperparameters serve them all (Raffel et al. 2019).
Changing only a few parameters
Full fine-tuning updates every weight and produces a full copy of the model for each task. Parameter-efficient fine-tuning (PEFT) instead freezes the backbone and trains only a small "delta": a limited set of added or selected parameters. A Nature Machine Intelligence survey calls this approach delta-tuning. It groups the methods into three families: those that add parameters, those that select existing ones, and those that reparameterize the update (Ding et al. 2023). The best-known reparameterization is low-rank adaptation (LoRA) (Hu et al. 2021):
W = W0 + B A
Here W0 is a frozen pretrained weight matrix of size d × k, and W is the adapted matrix. B has size d × r and A has size r × k. The rank r is much smaller than d or k, so only B and A are trained. The product BA is multiplied by a constant scaling factor. B starts at zero, so training begins from the unchanged pretrained model. After training, the product can be merged into W0, which means LoRA adds no inference latency. For GPT-3, a rank of 1–2 was often enough, even though the hidden dimension reaches 12,288. Compared with Adam-based full tuning, trainable parameters fell by roughly ten thousand times and GPU memory by about three times (Hu et al. 2021).
Adapters take a different route. They insert small bottleneck layers into each block (Houlsby et al. 2019). In that original design they add about 3.6% of the pretrained model's parameters (Pfeiffer et al. 2020). The cost is extra latency at inference (Hu et al. 2021). K-Adapter injects factual and linguistic knowledge into a frozen RoBERTa (Wang et al. 2020). Prompt tuning learns continuous input embeddings on a frozen model (Lester et al. 2021). It approaches full fine-tuning only near 10B parameters. P-Tuning v2 closes that gap by placing prompts at every layer (Liu et al. 2022). QLoRA combines low-rank adapters with a backbone quantized to 4-bit precision to cut memory further (Dettmers et al. 2023).
History and founding papers
From domain adaptation to pretrained transfer
Long before deep learning, statistical NLP studied how to move a model from a source domain to a target domain. Structural correspondence learning used pivot features shared across domains to adapt a part-of-speech tagger from newswire to biomedical abstracts, with no target labels at all (Blitzer et al. 2006). It was later applied to sentiment across Amazon product categories (Blitzer et al. 2007). Instance weighting separated shifts in p(y|x) from shifts in p(x), where p(x) is the distribution of inputs and p(y|x) the label distribution given an input. It also showed that up-weighting a few labeled target examples helps a great deal (Jiang & Zhai 2007). In machine translation, selecting pseudo in-domain sentences from a general corpus beat training on all of it (Axelrod et al. 2011).
In vision, residual adapters let one ImageNet-pretrained ResNet serve ten domains. Only small per-domain modules were trained, so earlier domains were not forgotten (Rebuffi et al. 2017).
The pretrain-then-fine-tune era
Universal Language Model Fine-tuning (ULMFiT), by Jeremy Howard and Sebastian Ruder, showed that fine-tuning a pretrained language model works well for text classification (Howard & Ruder 2018). The NAACL tutorial credited it, together with BERT, with lifting results on many tasks within about two years. It compared this shift to the earlier arrival of word embeddings (Ruder et al. 2019). BERT set new results on eleven tasks and made fine-tuning the field's default (Devlin et al. 2019). T5 then compared pretraining objectives, corpora and fine-tuning strategies systematically (Raffel et al. 2019). In the same year, Ziegler and colleagues fine-tuned language models from human preference judgments (Ziegler et al. 2019). This line of work later produced InstructGPT.
How it is used today
Instruction and preference tuning
For large language models, fine-tuning now mostly shapes behavior rather than teaching a single task. InstructGPT used two stages. Supervised fine-tuning (SFT) was done on labeler-written demonstrations. A reward model trained on ranked responses then guided further training with proximal policy optimization (PPO). This whole pipeline is called reinforcement learning from human feedback (RLHF). Evaluators preferred outputs of the 1.3B InstructGPT over the 175B GPT-3 (Ouyang et al. 2022). Llama 2 openly released chat models fine-tuned with SFT and RLHF (Touvron et al. 2023).
Stronger models can also serve as teachers that write training answers. LLaMA trained on responses written by GPT-4 beat the earlier Alpaca data on helpfulness (Peng et al. 2023). Multitask prompted fine-tuning of BLOOM and mT5 transferred zero-shot skills to languages beyond English (Muennighoff et al. 2022). See multilingual natural language processing.
Toolkits such as LlamaFactory offer LoRA, QLoRA, SFT, RLHF and direct preference optimization (DPO) across more than 100 models. Combining its efficiency tricks brings training memory to about 0.6 bytes per parameter (Zheng et al. 2024). See also efficient models and compression.
Beyond text, and into specialized domains
Visual instruction tuning connected a CLIP image encoder (a vision model trained to match images with captions) to a language model. It then trained the pair on synthetic image dialogues (Liu et al. 2023). LLaVA-1.5 reached strong results with about one day of training on a single 8-GPU node (Liu et al. 2023). InstructBLIP updated only its Q-Former, the querying transformer from BLIP-2 that feeds image features to the language model. It kept the image encoder and language model frozen (Dai et al. 2023). In medicine, domain pretraining followed by fine-tuning showed that the genre of the pretraining text matters (Peng et al. 2019). ChatDoctor adapted LLaMA-7B on about 100,000 patient–doctor exchanges (Li et al. 2023).
Adapters also help low-resource languages. MAD-X swaps language adapters for zero-shot transfer (Pfeiffer et al. 2020).
How much data is enough
Data needs vary widely. Larger BERT models improved even on the Microsoft Research Paraphrase Corpus (MRPC), which has roughly 3,600 training examples (Devlin et al. 2019). T-Few beat the human baseline on RAFT, a benchmark of 11 real-world tasks, with 50 examples per task (Liu et al. 2022). Randomly cutting LLaVA-1.5's training mixture by up to 75% caused no significant drop (Liu et al. 2023). Very small sets are a risk, though: a biomedical benchmark with only 64 training pairs gave unstable results (Peng et al. 2019).
Fine-tuning or prompting?
The alternative to fine-tuning is prompt engineering or in-context few-shot learning. There, the model sees examples in its input but its weights do not change. T-Few argued that cheap parameter-efficient fine-tuning is both more accurate and far cheaper per prediction than in-context learning. A fine-tuned model need not reprocess demonstrations for every input. In-context results also move when the wording or order of examples changes (Liu et al. 2022). The trade-offs are roughly these:
| Fine-tuning | Prompting | |
|---|---|---|
| Weights changed | yes (all or a small delta) | no |
| Upfront cost | a training run | none |
| Cost per prediction | low (short inputs, no demonstrations) | demonstrations reprocessed each time |
| Sensitivity to wording and example order | lower | higher |
| Switching tasks | swap a checkpoint or adapter | edit the prompt |
Open questions
Catastrophic forgetting remains a central problem: fine-tuning on one task in sequence after another erodes earlier knowledge (Pfeiffer et al. 2020; Wang et al. 2020). Frozen-backbone adapters were designed partly to avoid this (Pfeiffer et al. 2020; Wang et al. 2020; Rebuffi et al. 2017). Aligning a model with RLHF can also cost accuracy on standard benchmarks, an effect called the "alignment tax". Mixing pretraining data back into training largely recovers it (Ouyang et al. 2022).
Several further questions are unresolved:
- Which parameter-efficient method to use. No single method wins everywhere, and these methods tend to converge more slowly than full tuning (Ding et al. 2023).
- Why adaptation has such low intrinsic rank, and how that rank relates to model scale (Hu et al. 2021).
- Side effects of the tuning data. Training on many short answers biases models toward brevity (Muennighoff et al. 2022).
- Hallucination, which persists after tuning in multimodal and medical models (Liu et al. 2023; Li et al. 2023).
- Social bias, which instruction tuning did not noticeably reduce (Ouyang et al. 2022). See bias, fairness and safety.
How to evaluate fine-tuned models fairly, across domains and languages, is still debated in evaluation and benchmarks.
Key papers
Related concepts
- Few-shot learning — Few-shot learning (FSL) is the study and practice of building machine-learning systems that can perform a new task, such as recognising a…
- Large language models — A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on very large text collections…
More papers
Topics: 🧠 Language models & pretraining · 🪶 Efficient models & compression · 🌱 Low-resource languages
Shelves: 🤖 Artificial Intelligence · 🤖 Artificial Intelligence
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/fine-tuning/