📄 Large language models and the entropy of English

Summary

The authors use modern LLMs to revisit Shannon's guessing game and ask how far back in a text the useful predictive information extends. Each model predicts the next token from real text, which gives two measurements: the mean code length (an upper bound on the true conditional entropy) and the model's own predictive entropy. They run this on context lengths up to roughly 10⁴ characters. Four decoder-only models are tested on nine English corpora, ranging from web text and Wikipedia to news, narratives, and poetry. Lengths are expressed in characters so that different tokenizers can be compared.

For most prose, code length keeps falling slowly out to about 10⁴ characters with no clear plateau, which implies dependencies across thousands of characters. A model-free check on character mutual information confirms weak but real long-range correlations. Poetry behaves differently: it is costlier to encode and appears to saturate.

These findings constrain statistical-physics models of language and of LLMs. They also suggest that long-range structure is learned late in training.

Key points

• Models and corpora. The models are OLMo 2 1B, Llama 3.2 1B, Qwen3 8B, and DCLM 1.7B. The authors trained DCLM 1.7B themselves, with about two orders of magnitude less data, a 2048-token context, and no mid- or post-training. The corpora are:

  • C4 (curated from over 3.65×10⁸ documents),
  • English Wikipedia and Simple English Wikipedia,
  • time-stamped BBC news,
  • a narrative/summary/analysis set,
  • the Gutenberg Poetry Corpus and Poetry Foundation poems.

  All 36 model–corpus pairs were analysed.

• Code length versus context. Tokens average about 4–5 characters (Table I gives the exact values). On C4, three of the four models agree closely for N > 10³ characters, and none plateaus by N ~ 10⁴. Below about 100 characters, however, the models differ noticeably.

• Mutual information without a model.

  • The analysis uses a 93-symbol alphabet and about 10⁹ samples, so the estimation floor is far below 10⁻⁴ bits.
  • Short-range correlations decay over about 5–10 characters.
  • In C4, I(d) follows roughly d^−0.12 over at least two decades.
  • Wikipedia shows smaller values and a sharper drop beyond about 10³ characters.

• "Emergent certainty." As context grows, a peak near zero entropy separates from the bulk of the distribution, while the bulk itself drifts lower. The two effects contribute about equally to the falling mean. The code-length distribution shows a shallow power-law divergence at small values and a near-exponential tail at large values.

• Training dynamics and calibration. During DCLM training, short-context performance saturates quickly, while long-context gains accumulate slowly. For Qwen3, predictive entropy sits systematically below code length, yet its code lengths still match the other models.

Interesting ideas

• I1. Express every measurement in characters rather than tokens, using a per-corpus characters-per-token ratio, so systems with different segmentations can be placed on one axis.
• I2. Plot how prediction quality improves as the allowed context window grows; this separates what local rules supply from what distant context adds.
• I3. Compare a model's self-reported uncertainty with the actual cost of the true answer; a persistent gap, as seen for Qwen3, reveals miscalibration even when accuracy looks fine.
• I4. Track short-range and long-range performance separately during training, since they converge at very different speeds.
• I5. Study the full distribution of per-symbol cost, especially the share of symbols that become almost free given context, instead of only the mean.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when witness parsers or LLM teachers tokenize differently, score agreement and error rates per character or per word span, so that segmentation choices do not bias the silver-data vote.
• I2 (frames): measure coreference and role-assignment accuracy as a function of how many preceding sentences the resolver may see, to find where long-range discourse actually pays off.
• I3 (learn): check each teacher tier's confidence against its real error on gold trees and frames, and down-weight overconfident witnesses when building silver data.
• I4 (tree): split gparse learning curves by dependency arc length to decide when training has really converged, rather than stopping once short arcs saturate.
• I5 (back): in the text → graph → text compressor, measure what fraction of words are near-certain given the meaning graph, to see what the graph already encodes and what must still be stored explicitly.