📉 Zipf's law
Zipf's law is the empirical regularity that, when the items of a collection such as the words of a text are ranked by frequency, each item's frequency is roughly inversely proportional to a power of its rank. A few items are therefore extremely common and most are rare.
What it is
Count every word in a large text and sort the words from most to least frequent. The most common word turns up about twice as often as the second, about three times as often as the third, and so on down a long tail. Plotted on logarithmic axes, rank against frequency falls close to a straight line. Its slope is roughly −s, where s is the exponent defined below. That shape is Zipf's law.
The practical consequence is a deep imbalance. A small head of very frequent items covers much of any sample. A vast tail of rare items, many seen only once (hapax legomena), makes up most of the inventory. Corpus statistics therefore stay sparse for rare items and combinations however large the sample. This is why methods that learn word preferences from text must generalise to predicate–argument pairs never observed (Van de Cruys 2014).
The same skew appears in other linguistic inventories:
- Word senses are Zipfian, so disambiguation systems drift toward the most frequent sense (Blevins & Zettlemoyer 2020).
- Subword vocabularies used in machine translation inherit the imbalance (Gowda & May 2020).
- Label sets in specialised domains, such as clinical diagnosis codes, follow a power law (Rios & Kavuluru 2018).
The key idea
The basic law, its best-known generalisation and a companion law are:
f(r) ∝ 1 / r^s (Zipf)
f(r) ∝ 1 / (r + q)^s (Zipf–Mandelbrot)
V(N) = K · N^β (Heaps)
Here r is an item's rank by decreasing frequency, f(r) is its frequency, ∝ means "proportional to", and s is the exponent. For word frequencies s is classically taken to be close to 1. Mandelbrot's shift parameter q ≥ 0 flattens the curve at the very top ranks, where pure Zipf overpredicts frequencies.
Heaps' law describes vocabulary growth. N is the number of tokens read so far, V(N) is the number of distinct types met, K is a corpus-dependent constant, and β lies between 0 and 1. New words therefore keep appearing, but ever more slowly. Under broad conditions a Zipfian rank–frequency curve implies Heaps-type growth.
As a result, the type/token ratio falls as N grows and is only comparable between samples of equal length. In morphologically rich languages, inflection produces many distinct word forms and so pushes this ratio up (Gerz et al. 2018).
Zipf also described the brevity law, or law of abbreviation: frequent words tend to be shorter. Articles, pronouns and prepositions dominate the head of the distribution, while long words cluster in the tail. Zipf read this as economy. Keeping the most-used words short saves the most effort over all the times they are used, an idea he later built into his principle of least effort (1949).
The law can also be restated as a law about the frequency of frequencies. Here the number of items occurring exactly k times falls off as a power of k. The exponent of this form, often written α, is not the same quantity as s. For a pure Zipf curve the two are linked by α = 1 + 1/s. Modern fits often use this form and follow a now-standard recipe:
- estimate the exponent by maximum likelihood;
- pick a lower cutoff by minimising Kolmogorov–Smirnov distance;
- weigh the power law against alternatives such as the log-normal with a likelihood-ratio test.
A recent study of image tokens applied exactly this procedure (Shin 2026).
History and founding papers
The regularity was noticed before Zipf. Jean-Baptiste Estoup remarked on it in French stenography, in the 1916 edition of his manual. Edward Condon (1928) observed it in English word counts. George Kingsley Zipf, a Harvard philologist, made it famous and general. In The Psycho-Biology of Language (1935) he documented the law across languages and texts and tied it to the brevity law. In a 1949 book he explained both through a "principle of least effort". This balanced a speaker's wish for few, short words against a listener's need for many distinct ones.
Several competing explanations followed:
- Mandelbrot (1953) derived his generalised form by treating word lengths and frequencies as an optimisation of information transmitted per unit cost.
- Herbert Simon (1955) gave a stochastic account. In it, text grows by repeating used words in proportion to past use and occasionally adding new ones. This rich-get-richer process produces power-law tails.
- George Miller (1957) showed that random typing, with a space bar among the keys, already yields a Zipf-like curve. This suggests the law may say little about deep linguistic organisation.
On vocabulary growth, Gustav Herdan (1960) and Harold Heaps (1978) formulated the sublinear law now usually called Heaps' law.
Zipf's ideas also fed into practical text processing. A. D. Booth's (1967) observations about low-frequency words led to the transition point technique. This method picks a mid-frequency cutoff from the number of hapax words, I₁, as TP = (√(8·I₁+1) − 1)/2. It has since been applied to choosing index terms for Arabic text categorisation (Abbas & Lichouri 2021).
How it is used today
In natural language processing, Zipf's law is chiefly a design constraint: it determines where models will be weak.
- Output vocabularies. Word-level language models learn one output vector per word, so the many rare forms get poorly trained parameters. Sharing character-level information across forms helps most where type/token ratios are high (Gerz et al. 2018).
- Subword segmentation. Vocabulary size trades class imbalance against sequence length. Small vocabularies make sequences longer, while large ones make the frequency distribution more skewed. One recommended rule is to pick the biggest BPE vocabulary for which no fewer than 95% of the target-side classes (the subword types the decoder predicts) still get 100 or more training examples (Gowda & May 2020).
- Long-tail evaluation. Aggregate metrics hide tail failures. Gloss-informed bi-encoders were trained on SemCor and tested on a standard English all-words benchmark. They reduced error on less frequent senses by 31.1% relative to prior systems, a gain of 15.6 absolute F1 points over the closest competitor. Yet their F1 on most-frequent-sense examples still exceeded that on rarer senses by over 40 points (Blevins & Zettlemoyer 2020).
- Rare and unseen labels. The ICD-9 diagnosis and procedure code set has roughly 17,000 codes. More than half of them never occur in the MIMIC III clinical records, and about 5,000 occur only between 1 and 10 times. This motivates scoring frequent, few-shot and zero-shot labels separately (Rios & Kavuluru 2018).
- Feature selection. On an Arabic news corpus, mid-frequency transition-point terms gave nearly the same categorisation accuracy as selection by term frequency or document count. They did so with a vocabulary of about 2,500 words instead of about 40,000 (Abbas & Lichouri 2021).
Zipf-like distributions are also reported well beyond language. Commonly cited examples include city populations, firm sizes, citation counts and web traffic. Fits vary in quality, and some, notably city sizes, are contested.
Rank–frequency analysis has been transferred to images by treating them as documents of visual tokens. The fitted quantity there is α, the maximum-likelihood tail exponent of the token-count distribution, not the rank exponent s.
- Geometric tokens. For low-level geometric tokens, α was 1.76 for generated images and 1.82 for real photographs. The two sets were statistically indistinguishable.
- Texture tokens. Both sets were better fitted by a log-normal than by a pure power law. The power-law α therefore serves only as a comparative measure of how steeply the tail falls. Generated images decayed more steeply (α 2.29 versus 1.95). In other words, synthetic images under-represent rare textures (Shin 2026).
A newer line of work links Zipf to neural scaling. The quantization model proposes that models learn discrete skills ("quanta") in order of usefulness. Suppose the k-th quantum helps on a share of samples that falls as k^−(α+1). Here α is the model's own exponent, unrelated to the image-token α. Then excess loss after learning n quanta scales as n^−α (Michaud et al. 2023). This would give a Zipfian origin for the empirical power-law learning curves seen across speech, vision, language modelling and translation (Hestness et al. 2017).
Measured on this library
On 23 October 2026 we counted the words of the 6,229 paper texts behind the published pages of this library: scientific English, 49.7 million word tokens and 551,917 distinct words. A word here is a run of letters, lower-cased. The texts were extracted from PDFs and keep their reference lists, whose names and abbreviations swell the rare tail; we did not estimate how much. The exponents are ordinary least-squares slopes on the log–log points, a quick estimate that is biased compared with the maximum-likelihood recipe above, so read them as rough.
| measure | value |
|---|---|
| Zipf exponent s, ranks 1–1,000 | 0.91 |
| Zipf exponent s, ranks 10–10,000 | 1.19 |
| Heaps' law fitted from 10,000 tokens on | K ≈ 7.3, β ≈ 0.63 |
| distinct words after 1 million tokens | 39,283 |
| distinct words after 20 million tokens | 289,818 |
The two Zipf ranges overlap yet give different slopes: the curve is not one straight line but flatter at the top and steeper further down, as the Zipf–Mandelbrot form allows. The fitted β is also lower than the 1/s ≈ 0.84 that the mid-range slope would imply, which points to a still steeper tail. Word length follows the brevity law at the head and then levels off: the ten most frequent words (the, and, of, in, a, to, for, is, we, on) average 2.2 letters, ranks 11–100 4.2 letters, ranks 101–1,000 6.4, and the ranks beyond stay between 6.6 and 6.7. This is one language and one register, so it shows the laws in scientific prose, not how they vary across languages.
Open questions
There is no agreed account of why Zipf's law holds. Least-effort optimisation, preferential attachment and random-typing models all reproduce the curve, so goodness of fit cannot choose between them. Whether the exponent carries cognitive meaning, or is largely a statistical by-product, remains debated.
Related long-range statistics face similar doubts. Dependence between linguistic units, measured as mutual information, decays with distance as a power law. Since Lin and Tegmark (2017), this has often been read as a sign of hidden hierarchical structure. Their derivation needs two conditions:
- inside a tree, correlation must drop exponentially as tree distance grows;
- trees must be balanced, making sequence distance rise exponentially with separation in the tree.
Real English and Japanese parse trees meet neither condition. Mutual information along tree paths decays as a power law, and strong branching bias makes sequence distance grow slowly. The hierarchical explanation therefore needs rethinking (Nakaishi et al. 2026).
Measurement is a further issue. In the image-token study, texture counts that look power-law-like were better described by a log-normal once fitted rigorously. Apparent power laws therefore need testing against alternatives (Shin 2026).
The connection to machine learning is also unsettled. Hestness et al. measured how generalisation error falls with training-set size. The exponents they observed, roughly −0.07 to −0.35 across domains, are shallower than the −0.5 or −1 that classical theory predicts. These values were unexplained when reported in 2017 (Hestness et al. 2017). The quantization model's estimates of skill frequencies in real language models are, by its authors' account, imprecise (Michaud et al. 2023).
If Zipfian data structure drives power-law scaling, reshaping the data distribution should change the law. Two curation studies bear on this, though neither targets Zipfian structure directly. One shows a change in the form of scaling; the other shows efficiency gains at a given scale:
- Pruning. In a student–teacher perceptron analysis, a well-tuned pruning fraction combined with a high-quality pruning metric makes error fall exponentially with pruned dataset size. Experiments on image benchmarks showed better-than-power-law scaling (Sorscher et al. 2022).
- De-duplication and diversification. At 6.7B parameters, embedding-based de-duplication and diversification needed about 20% (18–22%) fewer pretraining tokens than random document selection to reach the same validation perplexity. It also added about 2% average zero-shot accuracy across 16 tasks (Tirumala et al. 2023).
How far curation can counteract the long tail without losing rare but important knowledge is an open problem.
Key papers
Related concepts
- Corpus linguistics — Corpus linguistics is the empirical study of language through large, principled collections of naturally occurring text or transcribed…
- Minimum description length — Minimum description length (MDL) is a model-selection principle rooted in information theory. It says the best explanation of a dataset is…
- Subword tokenization — Subword tokenization is a way of splitting text into units smaller than words but usually larger than single characters. A vocabulary of…
- Language model — A language model is a probability distribution over sequences of tokens (words, subwords or characters). It is usually factored into…
More papers
Topics: 📖 Lexical semantics & word embeddings · ⏳ Historical linguistics & language change
Shelves: 🗣️ Linguistics & languages
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/zipfs-law/