Elamur Research⌕
☰
home / papers / 2025 / Shakespeare, Entropy and Educated Monkeys

Shakespeare, Entropy and Educated Monkeys

Paper · 2025 · Ioannis Kontoyiannis

Summary

The paper revisits the infinite monkey theorem, which says that endless random typing will eventually reproduce Shakespeare. It asks how much faster this would happen if the "monkey" still typed at random but only produced statistically typical English. The author does not model English directly. He uses Shannon's Asymptotic Equipartition Property: given an entropy rate h, there are about 2^{ℓh} typical texts of length ℓ, so a specific one appears after roughly 2^{ℓh} keystrokes on average.

To turn keystrokes into years, he assumes an average human typing speed of 52 wpm, five characters per word and nonstop typing. He also uses h = 0.863 bits per character, an estimate obtained with Mistral 7B. The result is a compact rule of thumb for waiting times, which he compares with a uniform typist using a 27-symbol alphabet.

The educated monkey wins enormously on short phrases: minutes or hours instead of cosmological spans. Full works stay hopeless either way. The paper is a clear, playful demonstration of entropy rate as a practical measure of how constrained natural language really is.

Key points

  • Waiting-time rules of thumb:

    • Educated monkey: about 7.3 × 10^{0.26ℓ−9} years for ℓ characters.
    • Random monkey (27 symbols: lowercase letters plus space): about 7.3 × 10^{1.43ℓ−9} years.

  • The typing rate is 13/3 characters per second, which gives 1.367 × 10^8 characters per year. All target phrases are first reduced to the 27-character form.

  • Example waiting times (educated vs random monkey):

    TargetEducated monkeyRandom monkey
    Shakespeare's line preferring arrival hours early to being even a minute late~73 thousand years~2.7 × 10^63 years
    "To be or not to be"~3 hours~4.2 × 10^17 years
    All of Hamlet10^42,277 years10^232,784 years

  • The appendix surveys entropy estimates for English:

    • Shannon's n-gram and guessing methods: 0.6–1.3 bpc.
    • Cover and King's betting method: 1.29–1.9 bpc.
    • A 2019 guessing replication: ≈1.22 bpc.
    • PPM with extrapolation: ≈1.13 bpc.
    • Hutter prize on enwik9: 0.887 bits/byte, which is specific to that file.
    • LLM-based estimates (Llama 2, Chinchilla, Mistral 7B, GPT-2, OPT-IML): the 0.863 bpc figure comes from this group.

  • Historical context covered: Borel (1913), Collingwood, Borges, the Woodcock & Falletta "finite monkey theorem", the Monkey Shakespeare Simulator, and the Plymouth macaque experiment.

Interesting ideas

  • I1. A single scalar, the entropy rate, can stand in for an unknown full distribution: by the AEP, about 2^{ℓh} equally likely typical strings exist, which yields order-of-magnitude estimates with no explicit model of the language.
  • I2. The paper treats a language model's compression performance as a measuring instrument for text entropy rather than as a generator, the move behind the most trusted modern estimates.
  • I3. Every result is reported against two baselines, a uniform random typist and a statistics-constrained typist, which makes the size of the gain obvious.
  • I4. Before any computation, all targets are normalized to one fixed 27-symbol alphabet (lowercase letters plus space) so the comparisons are fair.
  • I5. The author warns that compression figures on enwik9 depend on that file's markup and mixed content and need not represent typical English.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): the text → graph → text compressor could report its output in bits per character and compare it with published entropy figures, giving a principled measure of how much information the meaning graph keeps or loses.
  • I2 (learn): a frozen LM's per-character cross-entropy could act as an independent typicality witness for silver data, flagging generated or repaired sentences that are statistically implausible.
  • I3 (learn): evaluations could routinely include a uniform-random and a frequency-only negative control, so improvements credited to structure (trees, frames) are measured against both chance and pure statistics.
  • I4 (tokens): when comparing tokenizers or taggers across corpora, a declared canonical normalization would keep metrics comparable and avoid gains that come only from formatting.
  • I5 (world): knowledge compiled from Wikipedia/Wikidata dumps should carry provenance, so statistics derived from it are not mistaken for properties of ordinary English text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom256 works · 123,373 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksMathematics · 189 worksComputational Theory and Mathematics · 321 worksStatistics and Probability · 19 works

Cite

Cite as Ioannis Kontoyiannis. 2025. Shakespeare, Entropy and Educated Monkeys. arXiv:2512.11880.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2512.11880 · arXiv 2512.11880

@misc{kontoyiannis2025shakespeare,
  title = {{Shakespeare, Entropy and Educated Monkeys}},
  author = {Ioannis Kontoyiannis},
  year = {2025},
  doi = {10.48550/arxiv.2512.11880},
  eprint = {2512.11880},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2512.11880}
}

References

35 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Acocella. The Typing Life: How Writers Used to Write. New Yorker, no. 9, April 2, 2007. find
  2. J.D. Barrow. Outer Space: Monkey Business. Plus Magazine, University of Cambridge, March 1, 2005. Available at plus.maths.org/content/outer-space-monkey-business. find
  3. É. Borel. La mécanique statique et l’irréversibilité. J. Phys. Theor. Appl., 3(1):189–196, March 1913. ↗
  4. J.L. Borges. La biblioteca total. Sur, 59, August 1939. find
  5. J. Bowery, M. Mahoney, and M. Hutter. Human Knowledge Compression Contest (Hutter Prize). prize.hutter1.net. Accessed: 2025-11-20. find
  6. J. Bowery, M. Mahoney, and M. Hutter. Hutter Prize Winners (enwik9). fx2-cmix, 110,793,128 bytes, Sept. 3, 2024. en.wikipedia.org/wiki/Hutter_Prize. Accessed: 2025-11-20. find
  7. S.K. Choi et al. The Atacama Cosmology Telescope: A measurement of the cosmic microwave background power spectra at 98 and 150 GHz. Journal of Cosmology and Astroparticle Physics, 2020(12):045, December 2020. find
  8. R.G. Collingwood. The principles of art, volume 11. Clarendon Press, Oxford, U.K., 1958. find
  9. T.M. Cover and R. King. A convergent gambling estimate of the entropy of English. IEEE Trans. Inform. Theory, 24(4):413–421, July 1978. ↗
  10. T.M. Cover and J.A. Thomas. Elements of information theory. John Wiley & Sons, New York, NY, second edition, 2012. find
  11. R. Dawkins. The blind watchmaker. Oxford University Press, Oxford, U.K., 1986. find
  12. G. Delétang, A. Ruoss, P.-A. Duquenne, E. Catt, T. Genewein, C. Mattern, J. Grau-Moya, L.K. Wenliang, M. Aitchison, L. Orseau, M. Hutter, and J. Veness. Language modeling is compression. In 12th International Conference on Learning Representations, ICLR ’24, Vienna, Austria, May 2024. ↗
  13. V. Dhakal, A.M. Feit, P.O. Kristensson, and A. Oulasvirta. Observations on typing from 136 million keystrokes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1–12, Montréal, Canada, April 2018. find
  14. S. Frautschi. Entropy in an expanding universe. Science, 217(4560):593–599, August 1982. ↗
  15. P. Guo, Z. Li, H. Hu, C. Huang, M. Li, and R. Zhang. Ranking LLMs by compression. arXiv e-prints, 2406.14171 [cs.AI], June 2024. find
  16. A.Q. Jiang et al. Mistral 7B. arXiv e-prints, 2310.06825 [cs.CL], October 2023. find
  17. C. Kittel and H. Kroemer. Thermal physics. Macmillan, 1980. find
  18. I. Kontoyiannis. The complexity and entropy of literary styles. NSF Technical Report no. 97, Department of Statistics, Stanford University, June 1996. Available at www.dpmms.cam.ac.uk/person/ik355. find
  19. I. Kontoyiannis, P.H. Algoet, Yu.M. Suhov, and A.J. Wyner. Nonparametric entropy estimation for stationary processes and random fields, with applications to English text. IEEE Trans. Inform. Theory, 44(3):1319–1327, May 1998. ↗
  20. Z. Li, C. Huang, X. Wang, H. Hu, C. Wyeth, D. Bu, Q. Yu, W. Gao, X. Liu, and M. Li. Lossless data compression by large models. Nat. Mach. Intell., 7, May 2025. ↗
  21. I.S. MacKenzie and R.W. Soukoreff. Text entry for mobile computing: Models and methods, theory and practice. Hum.-Comput. Interact., 17(2-3):147–198, 2002. ↗
  22. Wired magazine. Monkeys Don’t Write Shakespeare. WIRED, 11(05), May 9, 2003. find
  23. M. Mahoney. About the Test Data (Hutter Prize). www.mattmahoney.net/dc/textdata.html, September 1, 2011. find
  24. T. Nannicelli. Mass AI-art: A moderately skeptical perspective. The Journal of Aesthetics and Art Criticism, page kpaf026, 2025. ↗
  25. R. Perlman. The Infinite-Monkey Theorem: Field Notes. New Yorker, no. 13, January 9, 2023. find
  26. G. Ren, S. Takahashi, and K. Tanaka-Ishii. Entropy rate estimation for English via a large cognitive experiment using mechanical turk. Entropy, 21(12), 2019. ↗
  27. H. Ritchie. Monkeys Will Never Type Shakespeare, Study Finds. BBC News, November 1, 2024. Available at www.bbc.co.uk/news/articles/c748kmvwyv9o. find
  28. C.E. Shannon. A mathematical theory of communication. Bell System Tech. J., 27(3):379–423, 623–656, 1948. ↗
  29. C.E. Shannon. Prediction and entropy of printed English. Bell System Tech. J., 30:50–64, 1951. ↗
  30. I. Stewart. Professor Stewart’s hoard of mathematical treasures. Basic Books, New York, NY, 2010. find
  31. D.F. Styer. Statistical mechanics (version compiled on 2025-11-21), 2025. The Physics Library. Available at phys.libretexts.org. find
  32. R. Takahira, K. Tanaka-Ishii, and Ł. Dębowski. Entropy rate estimates for natural language – a new extrapolation of compressed large-scale corpora. Entropy, 18(10), 2016. BBC, May 9, 2003. Available at find
  33. K. Thomson-Jones and S. Moser. The philosophy of digital art. Stanford Encyclopedia of Philosophy, 2023. Available at plato.stanford.edu. find
  34. Wikipedia. Infinite monkey theorem. Wikipedia, The Free Encyclopedia, 2025. en.wikipedia.org/wiki/Infinite_monkey_theorem. Accessed: 2025-11-20. find
  35. S. Woodcock and J. Falletta. A numerical evaluation of the Finite Monkeys Theorem. Franklin Open, 9:100171, December 2024. Available at ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kontoyiannis-2025-shakespeare-entropy-educated-monkeys/