📄 Toward a statistical mechanics of four letter words

Summary

The paper asks whether English spelling, which looks highly combinatorial and rule-bound, can be described by a simple model that uses only pairwise statistics between letter positions. Stephens and Bialek treat a four-letter word as a small network of interacting letters. They fit a maximum entropy model, equivalent to a Potts/Ising-style Boltzmann distribution with six pairwise potentials, to two corpora: Jane Austen novels from Project Gutenberg and the American National Corpus (ANC), with words checked against a dictionary. Generalized iterative scaling makes the model's pairwise marginals match the observed ones. The pairwise model recovers about 92% (Austen) and 89% (ANC) of the multi-information among letters. It also gives probability to plausible unseen strings, and some of these are real dictionary words. Read as an energy landscape, the model's local minima are mostly real words and cover nearly two-thirds of word usage. The result matters because low-order interactions turn out to explain most of a structure that looks irreducibly complex.

Key points

• Data: the Austen corpus has 676,302 tokens, 7,114 unique words and 763 distinct four-letter words, which occur 135,441 times. The ANC (second release, ~2×10⁷ words, restricted to words seen more than 100 times) gives 798 four-letter types and 2,179,108 occurrences.
• Entropies for Austen are 18.80 bits for uniform letters, 14.083 bits for the position-specific independent model, 6.92 bits for the true distribution and 7.48 bits for the pairwise model. This leaves 7.163 bits of multi-information, of which the pairwise model captures ~92%.
• The model has 4,050 parameters, compared with 456,976 possible strings. Its pairwise couplings link all position pairs, not just neighbours, so it is not a Markov chain.
• About 15% of model probability goes to strings absent from the corpus. Roughly one-fifth of that mass falls on genuine dictionary words, and the same ratio holds for the ANC.
• The ANC landscape has 136 local minima, 118 of which are real words, and these cover 63.5% of usage. English uses ~1/3700 of the possible four-letter strings. That restriction splits into a factor of ~1/26 from letter frequencies, ~1/100 from pairwise interactions and only ~1/1.5 from higher-order effects.

Interesting ideas

• I1. A pairwise maximum entropy model coupling every pair of positions, rather than only neighbours, captures long-range spelling constraints with few parameters, which suits any short structured string.
• I2. Reading the model as an energy landscape means words sitting in local minima can be recovered from any single-letter error by sliding downhill to the nearest minimum.
• I3. Multi-information gives a single number for how much of the data's structure a model of a given order captures, and it splits the total into contributions from first-order, pairwise and higher-order effects.
• I4. The authors check generalization by looking at the probability mass placed on unseen items and testing whether those items are valid against an external dictionary.
• I5. Strings the model rates as likely but that are absent from the corpus are still pronounceable and follow spelling conventions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): A per-length pairwise letter model could score out-of-vocabulary tokens for "English-likeness," helping the tagger tell rare real words and names apart from noise or typos.
• I2 (tokens): An energy-descent spelling repair step could suggest corrections before lemmatization, and its decision stays explainable as "this change lowers energy most."
• I3 (learn): A similar decomposition could show how much each feature order (unigram, pair, triple) adds to a parser or tagger model, telling us whether higher-order features are worth their cost.
• I4 (learn): Mova's evaluation could do the same by checking unseen outputs of trained components against WordNet or a dictionary, as a cheap control that separates real generalization from hallucination.
• I5 (back): Sampling from such a model could produce plausible but non-existent placeholder words or names for generated test texts without colliding with real vocabulary.