📈 Scaling laws
Scaling laws are empirical relationships, usually power laws, that describe how a neural network's performance, often its test loss, changes predictably as model size, training data or compute grows. They let researchers forecast large-model behaviour from small experiments and plan how to split a budget.
What it is
A scaling law summarises a regularity seen across many training runs. When one resource is increased and the others are not bottlenecks, loss falls along a smooth curve. That resource can be parameters, training examples or floating-point operations. On log-log axes the curve is close to a straight line, which is the signature of a power law.
The practical value is extrapolation. Researchers fit the curve on small, cheap models and then predict the loss of a much larger run before paying for it. They also compare architectures, datasets or objectives by the slope of their curves rather than by a single point.
The idea was worked out most fully for autoregressive language models (Kaplan et al. 2020). It was then extended to other generative domains (Henighan et al. 2020), to contrastive image-text models (Cherti et al. 2022), to mixtures of modalities (Aghajanyan et al. 2023) and to source code (Luo et al. 2026).
The key idea
The core mechanism is a power-law decline of loss as resources grow; in the fuller form, the decline approaches a floor. Two standard forms are shown below. The first is the single-variable form associated with Kaplan and colleagues. The second is the joint parametric form fitted in the Chinchilla work.
L(X) ≈ (X_c / X)^α_X with X = N, D or C
L(N, D) = E + A / N^α + B / D^β
In the first line, L is the test cross-entropy loss. X is the resource being scaled: N is the number of model parameters, D is the number of training tokens, and C is the training compute. X_c is a fitted constant that sets the scale. The exponent α_X says how fast loss falls as that resource grows. This form has no irreducible term, so taken literally it falls toward zero rather than toward a floor.
Kaplan and colleagues described separate laws of this kind, L(N), L(D) and L(C). Each holds when the other two resources do not limit training (Kaplan et al. 2020).
The second line adds the floor explicitly. E is the irreducible loss that remains even with infinite model and data. A, B, α and β are fitted constants. The two fractions are the excess loss due to finite model size and finite data (Hoffmann et al. 2022; Besiroglu et al. 2024).
Training compute for a dense transformer is roughly proportional to N times D. Minimising L(N, D) under a fixed compute budget therefore tells you how to divide that budget between a bigger model and more tokens. Hoffmann and colleagues concluded that parameters and tokens should grow roughly in step. This gives the widely quoted rule of about 20 training tokens per parameter. That is the ratio actually used for the 70B Chinchilla model, and it agrees with two of the paper's three estimation methods (Hoffmann et al. 2022; Besiroglu et al. 2024).
The earlier Kaplan analysis had instead suggested that model size should outpace data as compute increases (Kaplan et al. 2020; Porian et al. 2024).
A power law also implies diminishing returns. Each constant reduction in loss needs a multiplicative increase in resources. This is why test error falling only as a slow power of dataset size is seen as costly (Sorscher et al. 2022).
History and founding papers
The modern programme starts with the 2020 study of neural language models. That study measured loss across several orders of magnitude of size, data and compute, and it drew allocation advice from the fits (Kaplan et al. 2020). The same year, GPT-3 demonstrated what scaling a language model up could buy in few-shot ability (Brown et al. 2020).
Follow-up work found similar laws for images, video and other autoregressive generative tasks (Henighan et al. 2020). Another line expressed the benefit of pretraining in units of data. When fine-tuning data is scarce, the "effective data transferred" follows its own power law. For text-to-Python transfer, a tenfold larger model is worth roughly a hundredfold more fine-tuning data (Hernandez et al. 2021). A 2022 study then examined how repeated data distorts scaling. It found that even a small share of repeated data can markedly hurt performance, and it linked this harm to damage to the induction heads that support copying (Hernandez et al. 2022).
The 2022 Chinchilla paper shifted the field toward smaller models trained on more tokens (Hoffmann et al. 2022). Its numbers were later scrutinised.
- A replication rebuilt about 240 runs from the paper's figure. As published, the parametric fit of the third method implies about 70 tokens per parameter, at odds with the ratio used in practice. The refit is consistent with roughly 20, in line with the paper's other two estimation methods (Besiroglu et al. 2024).
- Another study traced the Kaplan–Chinchilla gap to three details. The first was uncounted output-head FLOPs. The second was warmup that was too long for small models. The third was learning rate, batch size and AdamW β2 left untuned per model size. In the authors' own reproduction of the Kaplan setup, the exponent of optimal size against compute was 0.835. Correcting the three details lowered it to 0.497, which matches the Chinchilla result (Porian et al. 2024).
A theoretical account, the quantization model, proposes that networks acquire discrete skills ("quanta") in order of how often each skill is useful. If those frequencies follow a Zipf-like distribution, smooth power-law loss emerges as an average of many abrupt steps (Michaud et al. 2023).
How it is used today
Scaling laws now guide decisions far beyond choosing N and D for pretraining. Typical uses include the following.
| Use | Representative finding |
|---|---|
| Inference-aware training | Counting expected serving cost moves the optimum away from Chinchilla ratios (Sardana et al. 2024) |
| Quantization | Across over 35,000 runs, 4-bit weights gave the best accuracy per model bit (Dettmers & Zettlemoyer 2022) |
| Vocabulary size | Optimal vocabulary parameters grow with exponent about 0.83 in non-vocabulary parameters (Tao et al. 2024) |
| Distillation | Student loss depends on the teacher only through the teacher's loss (Busbridge et al. 2025) |
| Code | Across 117 training runs, the compute-optimal tokens-per-parameter ratio is much larger for code than for natural language, and the gap widens with budget (Luo et al. 2026) |
Training schedules have also been revisited so that laws can be fitted without a separate run for each amount of training data (Hägele et al. 2024). Observational scaling laws go further: they predict capabilities from many existing public models rather than from new training runs (Ruan et al. 2024).
A fast-growing branch studies compute spent after training. Accuracy from repeated sampling and voting rises quickly but approaches a ceiling. On MATH-500, counted by number of samples, Llemma-7B beats Llemma-34B until roughly 128 samples are drawn. At matched compute, the 7B model with the REBASE tree search comes close to the 34B model's accuracy while spending about half the FLOPs (Wu et al. 2024). With a well-chosen search strategy and reward model, a 1B model can surpass a 405B model on MATH-500 (Liu et al. 2025).
When verification is priced fairly, simple majority voting wins at most budgets. Generative verification pays off only when compute is very large. On MATH it needed up to 8× more compute merely to equal majority voting, and about 128× more to beat it by 3.8%. In that high-budget regime, the optimal number of solutions grows as compute to the power 0.57, and the optimal number of verifications as compute to the power 0.39 (Singhi et al. 2025). For generating synthetic training data, samples from a weaker but cheaper model can beat samples from a stronger one at matched compute (Bansal et al. 2024).
Alignment has scaling laws too. Optimising a policy hard against a learned reward model eventually lowers true quality, and this overoptimization follows regular curves (Gao et al. 2022).
Open questions
Several questions remain open.
- How far do fitted laws carry over between datasets? CLIP trained on different datasets scales at different rates on different tasks (Cherti et al. 2022). In one setting, 7B models fine-tuned on Skywork-MathQA, a synthetic set of 2.5M problem–solution pairs, kept gaining math accuracy as data grew. Whether that lack of saturation holds for other models and corpora is unknown (Zeng et al. 2024).
- How much does architecture change the slope? Among ten families the vanilla Transformer scaled best, and comparisons at a single model size can mislead (Tay et al. 2023).
- How far can training objectives move the curve? A short extra phase with a mixture-of-denoisers objective cost about 0.1–1% of the original FLOPs. After it, the 540B model reached the final quality of PaLM 540B using roughly half the compute (Tay et al. 2022).
- Can data selection change the slope itself? In theory, with a high-quality pruning metric, error can fall exponentially rather than as a power law in the size of the pruned dataset. In practice, a self-supervised metric removed 20% of ImageNet without losing accuracy (Sorscher et al. 2022).
- How should context length be planned? Proposed laws relate the rotary-embedding base to how far a model can extrapolate beyond its training length (Liu et al. 2023).
Smooth loss curves also do not guarantee better behaviour. A prize contest received 99 submissions and selected 11 winning tasks that showed robust inverse scaling, where larger models did worse. The study traced this to four causes. Models may repeat memorised patterns instead of following the instructions in the prompt. They may imitate undesirable patterns in training data. They may be pulled off course by easy distractor tasks, or misled by unhelpful demonstrations. Some of these trends turned U-shaped at the largest scale, so extrapolating from small models is risky (McKenzie et al. 2023).
Capabilities that appear only at large scale also complicate forecasting. Chinchilla 70B can map a known answer onto its multiple-choice letter, a skill its 7B sibling lacks (Lieberum et al. 2023). Whether such capabilities can be predicted from aggregate loss remains a central unresolved question. So does how long power-law returns will continue as high-quality data runs short.
Key papers
Related concepts
- Language model — A language model is a probability distribution over sequences of tokens (words, subwords or characters). It is usually factored into…
- Transformer and attention — The Transformer is a neural network architecture that represents a sequence by letting each element gather information from all others…
- Minimum description length — Minimum description length (MDL) is a model-selection principle rooted in information theory. It says the best explanation of a dataset is…
- Subword tokenization — Subword tokenization is a way of splitting text into units smaller than words but usually larger than single characters. A vocabulary of…
More papers
Topics: 🧠 Language models & pretraining · 🪶 Efficient models & compression
Shelves: 🤖 AI & machine learning · 💬 Natural language processing
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/scaling-laws/