📄 Entropy Rate Constancy in Text

Summary

Genzel and Charniak ask whether people writing text keep the amount of information per word roughly steady. Information theory says a constant rate is the most efficient way to use a noisy channel. They split a word's conditioning history into two parts: the words earlier in the same sentence, and all preceding sentences. If total conditional entropy stays flat, then entropy measured without the earlier sentences should grow as a text goes on, because those sentences carry more and more useful information.

They test this corollary on Wall Street Journal text using three estimators:

• an n-gram model,
• the Charniak (2001) statistical parser used as a language model,
• the non-parametric estimator of Kontoyiannis et al. (1998).

All three show context-free sentence entropy rising with sentence position.

A part-of-speech breakdown, combined with a cache model, shows two causes of the rise. One is lexical (which words recur) and the other is non-lexical (how words are used). The work gives an indirect, measurable signature of discourse context. It suggests language modeling as a tool for studying how context shapes text.

Key points

• The n-gram and parser experiments train on Penn Treebank sections 0–20 and test on sections 21–24. Each article is a separate text, and scores are averaged by sentence number. The parser gives lower (better) entropy estimates than the n-gram model, but both rise in the same way.
• The non-parametric estimate uses 3M words of WSJ 1988 for training and 23M words of WSJ 1987 for testing. It confirms the trend with one exception: second sentences are unusually easy to predict. The authors tie this to corpus quirks, such as about 1.5% of second sentences opening with the same stock phrase about undisclosed terms.
• Per-word entropy rises with sentence number even for closed-class words (determiners, prepositions), not only for nouns and verbs. This points to non-lexical effects.
• A cache model (λ = 0.1) mixed with a smoothed trigram has the following effects:
  • It lowers entropy clearly for nouns and slightly for verbs.
  • It has no effect on closed-class words.
  • So lexical repetition explains part of the effect, but only for open-class words.
• Late sentence positions have fewer samples, because most articles are short, so estimates there are less reliable.

Interesting ideas

• I1. Test an unmeasurable quantity through a measurable corollary. Good models of discourse context did not exist, so the authors derived a prediction about context-free entropy, which simple models can estimate. Testing the consequence instead of the hard quantity is a reusable strategy whenever the target variable is out of reach.
• I2. Assume model bias is roughly uniform and compare trends. Cross-entropy overestimates true entropy, but if the overestimate is similar across conditions, relative changes are still meaningful. The same logic applies when comparing positions, genres, or time periods with imperfect models.
• I3. Check robustness with several estimators. Agreement among an n-gram model, a parser-based model, and a non-parametric match-length estimator makes the finding less likely to be an artifact of any one model. It also helps reveal corpus-specific anomalies, like the second-sentence dip.
• I4. Use a cache as a diagnostic. Comparing a model with and without a word cache separates simple repetition effects from deeper usage effects. Splitting the result by part of speech adds a natural control, since closed-class vocabulary barely changes.
• I5. Position-indexed entropy curves as a discourse probe. Plotting sentence-level entropy against position in a document could be used to study how genres front-load or distribute information. The authors suggest such measurements might detect contextual relations like the given-new distinction.