📄 An Analysis of Letter Dynamics in the English Alphabet

Summary

The paper asks whether letter frequencies in English differ by genre and whether they can be used to tell genres apart. The authors built a small corpus of four categories: 20 novels and 20 plays from Project Gutenberg, 20 news articles from US outlets, and 20 scientific papers. In MATLAB, they counted each letter while ignoring case, punctuation and spaces, then turned the counts into percentages. They averaged these percentages per category, sorted them to build cumulative coverage curves, and estimated how much information each letter carries. They also defined a "distance, d" between one document's letter profile and a category's average profile, which acts as a nearest-reference classifier. All four genres share the same broad three-peaked letter distribution. A handful of letters covers half of all text. On held-out documents, the distance usually pointed to the correct genre, although the news category did not separate significantly from the others. The work matters as a very cheap, transparent baseline for sorting documents and for compact information transmission.

Key points

• Corpus: 20 novels and 20 plays (gutenberg.org), 20 news articles (CNN, Fox News, AP, Yahoo News, Just the News), and 20 science articles, some historical (Fleming 1929, Watson and Crick 1953, Newton's Principia). Everything was processed in MATLAB R2019a.
• Letter coverage is highly skewed: about 23% of the alphabet accounts for roughly half of all letters, and about 62% accounts for roughly 90%.
• Evaluation used 5 test documents per genre, compared against each n=20 category average, with Kruskal-Wallis and the Dunn-Sidak correction.
• Novels and science articles were significantly closer to their own reference than to two of the other references.
• Plays were closest to their own reference on average. News was closest to news, but not significantly so, and news test 1 actually lay nearer the science reference (1.78 vs 2.18).

Interesting ideas

• I1. A reference profile per class, plus a single distance to each profile, gives a classifier whose decision can be explained by showing which letters deviate most.
• I2. The authors suggest that a large deviation from the shared letter distribution may signal text produced under unusual influence rather than normal writing.
• I3. Each test group is scored against every class reference, not only its own, and the gaps are checked with a significance test.
• I4. Sorting items by frequency and plotting cumulative coverage shows how few units carry most of the mass.
• I5. A cheap, transparent genre signal can be computed before any deep analysis.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): a letter-profile check in crate en could flag input whose character statistics look unusual, such as broken PDF extraction or code fragments, before tagging, instead of silently parsing garbage.
• I2 (learn): per-source letter or character profiles could be used to screen silver data and set aside documents whose profile is far from normal prose before the witness parsers vote on them.
• I3 (learn): evaluations of genre-specific models, such as a fable reader versus a news reader, could report performance against all the "wrong" references as negative controls, with a test showing the gap is real.
• I4 (tokens): the same curve over lemmas or morphological forms in the treebanks could decide which items deserve hand-written lexicon entries first.
• I5 (frames): a coarse genre guess (play vs. narrative vs. scientific prose) could set priors for speech-act and dialogue handling in crate prag. It would only route the input and would never stand in for meaning.