📄 A New Look at the Classical Entropy of Written English

Summary

Published figures for the information content of English per character disagree, mostly because earlier studies relied on indirect means such as human guessing, gambling games, compression or string matching. Guerrero instead computes entropy directly from Shannon's definitions by counting symbols. The data are 21 classic novels chosen from The Daily Telegraph's 2009 "100 novels everyone should read" list and taken from Project Gutenberg, about 20.3 million printable characters in total. Nothing is normalised: upper and lower case, punctuation and the space all count as distinct symbols. For every block length from 1 to 500 characters, block probabilities are estimated from the relative frequencies of non-overlapping blocks, and the entropy is averaged over all n possible starting offsets. Everything ran in Mathematica 6 on an ordinary laptop. The estimated entropy rate is 1.58 bits per character, with a redundancy of 74.86%. The paper also defines an "equiprobability distance": the block length beyond which every block in a sample is unique. Expressed in words, it marks where structural ties between words fade. This matters for plagiarism checkers and for length limits on short messages.

Key points

• The corpus contains 21 novels (Don Quixote, Ulysses, Moby Dick, Tess of the Durbervilles, and others) with 20,306,606 printable characters and 3,804,409 words. The weighted mean word length is 4.22 letters, and alphabet sizes range from 71 to 96 symbols per sample.
• The main result is 1.58 bits/character and 74.86% redundancy. Earlier estimates were 0.6–1.3 (Shannon, human prediction), 1.25–1.35 (Cover & King, gambling), 1.46 (Teahan & Cleary, PPM) and 1.77 (Kontoyiannis, string matching).
• Block entropy peaks at n between 8.31 and 11.43 characters, roughly 1.54–2.17 words. The author reports that this peak value alone is enough to obtain the entropy rate.
• Entropies computed at different shifts for the same n barely differ. For Tess at n = 5, the five shifts give values between 14.249 and 14.258 bits/symbol. In total, 125,250 shifted entropies are computed per sample for n up to 500.
• The samples with the lowest word dispersion ratio (WDR, distinct words over total words) show the highest redundancy. The word equiprobability distance d_aep ranges from about 5.2 words (sample 15) to 58.8 words (sample 11).

Interesting ideas

• I1. Each block length is measured at every phase offset and the results are averaged, which also shows for free how stable the estimate is; near-identical shift values signal a trustworthy statistic.
• I2. The equiprobability distance gives a length threshold: shared spans shorter than it are expected from ordinary language structure, while longer ones point to real copying.
• I3. Case, punctuation and spaces are treated as part of the language rather than cleaned away, and character blocks avoid the coarse length steps that word units impose.
• I4. WDR, a trivially cheap type/token ratio, tracks redundancy across texts, so it works as a quick proxy for how repetitive or predictable a text is.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): metrics on silver data could be recomputed over several shifted splits of the same documents, so that a score which moves a lot between shifts is flagged as unreliable before anyone trusts it.
• I2 (back): when testing the text → graph → text compressor, only shared spans longer than such a threshold would count as verbatim reuse, separating genuine paraphrase from parroting the source.
• I3 (tokens): character n-gram statistics over raw, unnormalised text could help the tokenizer judge where abbreviations, quotes and sentence ends actually split.
• I4 (answer): fables and school-test passages could be tagged with WDR, so reader accuracy can be reported separately for repetitive and lexically dense texts.