📄 Is language evolution grinding to a halt? The scaling of lexical turbulence in English fiction suggests it is not

Summary

The paper asks whether the English lexicon is still renewing itself. Earlier work by Petersen et al. reported that word births fell sharply and word deaths rose in the late 20th century. The authors re-examine that claim on the Google Books 2012 English Fiction corpus for 1800–2000. They chose this subset because their earlier study found the other Google Books sets heavily skewed toward scientific text.

They rebuilt the median-based birth/death criterion and moved the end year of the analysis window. This showed that the apparent rise in deaths comes mostly from where the arbitrary "present" is placed. Birth rates do fall, but true word death turns out to be very rare. As a sturdier alternative, they count how many words cross fixed relative-frequency thresholds between consecutive decades. They also use Jensen-Shannon divergence word shifts to show which words drive the change.

The overall Zipf distribution stays stable, yet beneath it there is constant "lexical turbulence." The lexicon keeps turning over, so language evolution is not stopping.

Key points

• Data: Google Books 2012 English Fiction 1-grams, 1800–2000. Each book is counted once, so the counts say nothing about how popular a book was or how often it was read. Corpus volume grows roughly exponentially, with dips around the American Civil War and both World Wars.
• Petersen et al.'s rule treats a word as born or dead when its frequency crosses 0.05× its own median frequency. The authors identify three weaknesses:
  • the threshold depends on the word and on the chosen year range;
  • rare words have a median of zero;
  • Google keeps only words seen at least 200 times, so a zero count does not mean the word was absent.
• When they rerun the analysis with end years from 1860 to 2000 in 20-year steps, birth rates decline roughly exponentially. Death rates, however, mainly reflect the cutoff year.
• Word flux across frequency thresholds grows superlinearly with word rank. Its scaling break lines up with the break in Zipf's law.
• JSD word shifts reveal both content and corpus problems:
  • Example: the 1960s→1970s shift at threshold 10⁻⁵ has a total JSD of 0.0031 bits.
  • Many prominent words point to scholarly criticism mixed into the fiction corpus, such as author names and "vols".
  • The authors recommend separating criticism from fiction in a future revision of the corpus.

Interesting ideas

• I1. Any event defined relative to the edge of a time window should be rerun with the edge shifted; if the result moves with the cutoff, it is an artifact rather than a finding.
• I2. Instead of making binary birth/death calls, count items crossing fixed frequency bands in both directions, which avoids per-item thresholds that are poorly defined.
• I3. Split a divergence between two distributions into ranked per-item contributions (a word shift), so that a single number becomes a readable explanation.
• I4. Inspect the items that dominate a measured change to discover what a corpus really contains, for example literary criticism hidden inside "fiction."
• I5. Treat frequency rank like a temperature: the most frequent words are nearly frozen, while rarer words change more and more.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluation could add a sliding-cutoff or shifted-split control, so that apparent gains or losses are checked for being produced by how the data was split.
• I2 (tokens): Lexicon and lemma coverage could be tracked per frequency band across text sources, showing where unknown-word handling and morphology fallbacks are under the most pressure.
• I3 (learn): Disagreements among witness parsers, or between silver and gold label distributions, could be broken down into ranked per-relation or per-lemma contributions, so repair rules target the items that matter most.
• I4 (tree): Before training gparse or the perceptron parser on a treebank or silver corpus, word-shift style audits could flag genre contamination that would bias the model.
• I5 (tokens): Closed-class and very frequent words can safely get hand-built, stable rules, while effort for robustness and updates goes to the volatile rare tail.