📄 The role of grammar in transition-probabilities of subsequent words in English text

Summary

Zipf's law for word frequencies can be produced by sample-space reducing (SSR) processes, in which each new word narrows the set of words that may follow. Pure SSR processes, however, yield triangular word-to-word transition matrices. Real English text has matrices that look almost random, close to a Bernoulli model with the same word frequencies.

Hanel and Thurner resolve this mismatch by treating text generation as two stacked processes. An SSR process first picks the words of a narrative. A toy grammar then rearranges them locally. Each lexicon word gets one of N_g grammatical labels (for example S, P, O), and a "neutral" label marks words that fuse with the next labelled word. The authors call this the grammatically ordered SSR (goSSR) model.

They test it on English novels from Project Gutenberg and compare it with the Simon, random typewriting and Bernoulli models using matrix-level statistics. Only three to five labels are needed to keep Zipf's law and to produce realistic transition matrices. Because local reordering hides the structure of word selection, adjacent-word statistics reveal little about how meaning is built.

Key points

• A pure SSR process with W ordered states, restarted repeatedly, visits state i with probability proportional to 1/i, which is exactly Zipf's law. Its transition matrix is triangular.
• The goSSR model assigns each word one of N_g roughly equally frequent labels, plus a neutral label 0 with probability p_n. Neutral words merge with the next non-neutral word into one unit.
• Using N_g = 3–5 labels is enough to make goSSR transition matrices resemble those of real text. The figures compare N_g = 2 with N_g = 5 and p_n = 0 with p_n = 0.2.
• Among the example books, "Five little peppers and how they grew" contains about 4,800 distinct words, roughly 88,000 word transitions and about 5,200 sentences.
• Two test statistics are used:
  • cos(A), the cosine between the all-ones vector v and Av, which serves as a proxy for the largest eigenvalue;
  • skew(A), the summed antisymmetry A_ij − A_ji over the upper triangle.

Interesting ideas

• I1. The null model shuffles every word in the text but puts sentence ends back at their original positions, so sentence-length structure survives while word order is destroyed.
• I2. Baselines are fitted fairly: the Simon model's new-word probability is adjusted over time so its output matches the target text's vocabulary size and length.
• I3. Choosing content and ordering it are modelled as separate stages, and the ordering step largely hides the selection structure from bigram statistics.
• I4. Function-like "neutral" words are glued to the next content word and moved as one block.
• I5. Whole transition matrices are compared with a few cheap scalars: convergence toward the stationary distribution (cos) and directional asymmetry (skew).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Use such sentence-preserving shuffles as a negative control. If a metric still rewards shuffled input, it is measuring frequencies rather than structure.
• I2 (learn): When comparing parsers or generators against trivial baselines, match the baselines on length and vocabulary so differences reflect structure rather than size effects.
• I3 (back): Generation can first select content from the world graph and then linearize it with grammatical slot rules. This also argues that adjacency statistics are a poor test of meaning preservation in the compressor.
• I4 (tree): This mirrors attaching det/case/aux to their content head. Reordering or comparing at the level of such head-plus-function chunks could simplify UD dialect conversion and word-order checks.
• I5 (back): Compute cos and skew on generated versus original text corpora as a quick sanity check that generation or translation output has natural sequential statistics.