📄 Evolution of the most common English words and phrases over the centuries

Summary

The paper asks whether the written English record shows the same self-organising statistics seen in social and technological systems. It also asks how stable the "most common" words and phrases have been since the early 1500s. Perc uses the English n-gram data released with the culturomics study by Michel et al. (Google Books), covering 1-grams through 5-grams.

He first confirms Zipf's law on this corpus. He then follows the yearly ranks of the top n-grams across centuries and measures how much those ranks wander. Finally, he tests whether frequent n-grams tend to gain further usage in proportion to how often they already occur.

The main findings:

• Top rankings were very short-lived in the 16th–17th centuries.
• Rankings became much steadier from about 1800 onward.
• Over the last two centuries, usage growth fits linear preferential attachment well.

Together with the known steady growth of the lexicon, this gives an empirical, network-science account of why Zipf scaling appears in text. It also suggests that today's frequency rankings are unlikely to shift much.

Key points

• Data. The corpus is the Google Books English n-gram dataset from Michel et al. (culturomics), with n = 1–5, from 1520 to 2008. Tables of the top n-grams per year are published at matjazperc.com/ngrams.
• Zipf test. The n-gram sets for n > 1 are treated as separate corpora, and 1-gram frequencies are counted inside each. Frequency falls with rank at a slope of about −1 on log-log axes, across several orders of magnitude.
• Rank tracking. The top 100 n-grams of 1520, 1604, 1700, 1800 and 1900 are followed until their rank passes 10,000 or the century ends. Popularity is fleeting before about 1800 and most persistent in the 20th century. Longer n-grams lose their rank faster than single words.
• Rank volatility measure. For every year, the average standard deviation of the rank trajectories of the top 1000 n-grams is computed over the next 100 years. It drops sharply around 1800. The result holds with a 50-year horizon and when the analysis runs backwards in time.
• Preferential attachment. Attachment rate rises linearly with current occurrence count for 1800–2008 (goodness-of-fit ≈ 0.8). The fit is poor for 1520–1800 (≈ 0.05). The author cites Michel et al. for the 33% growth of the lexicon during the 20th century.

Interesting ideas

• I1. Checking that a corpus reproduces the expected Zipf slope is a cheap sanity check before deeper analysis, a way to validate the data against a known statistical signature.
• I2. Stability of a ranking can be scored as the spread of rank trajectories over a fixed horizon, and the robustness of that score can be confirmed by changing the horizon and running the analysis backwards.
• I3. Longer units hold their rank less well than single words, and rankings from modern (post-1800) text are much more stable than early-modern ones.
• I4. Preferential attachment can be tested directly by relating each item's growth rate to its current count, checking linearity, and comparing fit quality between periods.
• I5. The author warns that apparent volatility in early centuries partly reflects having only a few books per year, so sparse slices can fake instability.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): run a Zipf-slope check on each new silver or generated corpus as a negative control; a broken slope would flag tokenizer bugs, heavy duplication or degenerate self-play text.
• I2 (learn): track the rank stability of frequent frames, repair rules or parser errors across training rounds, to tell genuine convergence apart from churn.
• I3 (tokens): frequency-based lemma and tagger priors drawn from modern text can stay largely fixed, while older texts such as early fables may need separate or softened frequency priors.
• I4 (learn): apply the same test to see whether self-induced rules or LLM-teacher labels reinforce themselves rich-get-richer style, which would point to a feedback loop rather than real evidence.
• I5 (answer): when evaluation sets are split by genre or source, report slice sizes and treat shifts in small slices with caution before blaming the model.