Elamur Research⌕
☰
home / papers / 2018 / A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics

A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics

Paper · 2018 · Martin Gerlach and Francesc Font-Clos

Summary

Project Gutenberg (PG) has been a core resource in statistical language research for over two decades, yet no agreed-upon full version of it exists. Most studies rely on a few hand-picked books (Moby Dick shows up almost everywhere) and on inconsistent, poorly documented preprocessing. This threatens both reproducibility and generalizability. To tackle this, Gerlach and Font-Clos introduce a freely available four-step pipeline, which they name the Standardized Project Gutenberg Corpus (SPGC):

  • mirror all of PG,
  • strip headers and boilerplate,
  • tokenize with NLTK,
  • store each book at several levels of granularity.

Every book is linked to PG metadata (author, life dates, language, subjects, downloads) and to volunteer-curated "bookshelves." The authors show that bookshelves overlap less than the official subject categories. Anyone can rebuild the SPGC locally, refresh it as PG grows, and use it in any language. A frozen snapshot is also released for exact replication. Example analyses based on Jensen-Shannon divergence show how word usage varies across authors, periods and genres. The result is a large, reusable resource for corpus linguistics, NLP and information retrieval.

Key points

  • Static snapshot SPGC-2018-07-18 (DOI 10.5281/zenodo.2422560):
    • 55,905 books and more than 3×10⁹ word tokens.
    • Each book is available as raw text, cleaned text, a token sequence and word counts.
    • Duplicates and non-UTF-8 files were removed.
  • Processing tools:
    • rsync mirror of aleph.gutenberg.org,
    • NLTK sentence tokenizer plus TreebankWordTokenizer,
    • filtering to alphabetic tokens, lowercasing,
    • Python collections.Counter for the counts.
  • Languages:
    • 56 languages in total; English accounts for 81% of books and 2.8×10⁹ tokens, over ten times the British National Corpus (~10⁸).
    • Next largest: French (2,864 books), Finnish (1,903) and German (1,644).
    • 13 non-English languages exceed 100 books.
  • Book length: most books are longer than 10⁴ tokens. The length distribution is heavy-tailed, and many books exceed 10⁵ tokens.
  • Dating and copyright:
    • Publication years are approximated from author lifespans (birth + 20 < t < death).
    • Output peaks around 1900.
    • The 1998 Copyright Term Extension Act restricts works published after 1923.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northwestern UniversityEvanston · United States46 works · 2,850 citations🎓University of MilanMilan · Italy7 works · 38,856 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksGeneral Social Sciences · 254 worksLiterature and Literary Theory · 541 works

Cite

Cite as Martin Gerlach and Francesc Font-Clos. 2018. A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics. arXiv:1812.08092.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1812.08092 · arXiv 1812.08092

@misc{gerlach2018standardized,
  title = {{A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics}},
  author = {Martin Gerlach and Francesc Font-Clos},
  year = {2018},
  doi = {10.48550/arxiv.1812.08092},
  eprint = {1812.08092},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1812.08092}
}

References

57 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Altmann EG, Gerlach M. Statistical laws in linguistics. In: Degli Esposti M, Altmann EG, Pachet F, editors. Creativity and Universality in Language. Springer; 2016. p. 7–26. find
  2. Ferrer i Cancho R, Solé RV. Least effort and the origins of scaling in human language. Proceedings of the National Academy of Sciences. 2003;100(3):788–791. find
  3. Petersen AM, Tenenbaum JN, Havlin S, Stanley HE, Perc M. Languages cool as they expand: Allometric scaling and the decreasing need for new words. Scientific reports. 2012;2:943. find
  4. Tria F, Loreto V, Servedio VDP, Strogatz SH. The dynamics of correlated novelties. Scientific reports. 2014;4:5890. find
  5. Corominas-Murtra B, Hanel R, Thurner S. Understanding scaling through history-dependent processes with collapsing sample space. Proceedings of the National Academy of Sciences. 2015;112:5348–5353. find
  6. Font-Clos F, Corral A. Log-log convexity of type-token growth in Zipf’s systems. Phys Rev Lett. 2015;114:238701. find
  7. Cocho G, Flores J, Gershenson C, Pineda C, Sánchez S. Rank Diversity of Languages: Generic Behavior in Computational Linguistics. PLoS One. 2015 Apr;10(4):e0121898. find
  8. Lippi M, Montemurro MA, Esposti MD, Cristadoro G. Natural Language Statistical Features of LSTM-generated Texts. arXiv:180404087;. find
  9. Mazzolini A, Gherardi M, Caselle M, Cosentino Lago-marsino M, Osella M. Statistics of shared components in complex component systems. Physical Review X. 2018;8:021023. find
  10. Dorogovtsev SN, Mendes JF. Language as an evolving word web. Proc R Soc B. 2001;268:2603–2606. find
  11. Solé RV, Corominas-Murtra B, Valverde S, Steels L. Language networks: Their structure, function, and evolution. Complexity. 2010;15(6):20–26. find
  12. Amancio DR, Altmann EG, Oliveira ON, Costa LDF. Comparing intermittency and network measurements of words and their dependence on authorship. New Journal of Physics. 2011;13:123024. find
  13. Choudhury M, Chatterjee D, Mukherjee A. Global topology of word co-occurrence networks: Beyond the two-regime power-law. In: Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010); 2010. p. 162–170. pgcorpus/gutenberg, while python-based jupyter notebooks that re… find
  14. Cong J, Liu H. Approaching human language with complex networks. Physics of Life Reviews. 2014;11:598–618. find
  15. Bochkarev V, Solovyev V, Wichmann S. Universals versus historical contingencies in lexical evolution. Journal of the Royal Society Interface. 2014;11:20140841. find
  16. Ghanbarnejad F, Gerlach M, Miotto JM, Altmann EG. Extracting information from S-curves of language change. Journal of the Royal Society Interface. 2014;11:20141044. find
  17. Feltgen Q, Fagard B, Nadal JP. Frequency patterns of semantic change: Corpus-based evidence of a near-critical dynamics in language change. Royal Society Open Science. 2017;4:170830. find
  18. Gonçalves B, Loureiro-Porto L, Ramasco JJ, Sánchez D. Mapping the Americanization of English in space and time. PLoS ONE. 2018;13:e0197741.
  19. Amato R, Lacasa L, Dı́az-Guilera A, Baronchelli A. The dynamics of norm change in the cultural evolution of language. Proc Natl Acad Sci U S A. 2018;115:8260–8265. find
  20. Karjus A, Blythe RA, Kirby S, Smith K. Challenges in detecting evolutionary forces in language change using diachronic corpora. arXiv:181101275. 2018;. find
  21. Montemurro MA, Zanette DH. Towards the quantification of the semantic information encoded in written language. Advances in Complex Systems. 2010;13:135. find
  22. Takahira R, Tanaka-Ishii K, Debowski L. Entropy rate estimates for natural language—A new extrapolation of compressed large-scale corpora. Entropy. 2016;18:364. find
  23. Febres G, Jaffé K. Quantifying structure differences in literature using symbolic diversity and entropy criteria. Journal of Quantitative Linguistics. 2017;24:16–53. find
  24. Bentz C, Alikaniotis D, Cysouw M, Ferrer-i Cancho R. The entropy of words—Learnability and expressivity across more than 1000 languages. Entropy. 2017;19:275. find
  25. Ferrer i Cancho R, Solé R, Köhler R. Patterns in syntactic dependency networks. Physical Review E. 2004;69:51915. find
  26. Kulig A, Kwapień J, Stanisz T, Drożdż S. In narrative texts punctuation marks obey the same statistics as words. Inf Sci. 2016;375:98–113. find
  27. Michel JB, Shen YK, Aiden AP, Veres A, Gray MK, Team TGB, et al. Quantitative analysis of culture using millions of digitized books. Science. 2011;331(6014):176– 182. find
  28. Masucci AP, Kalampokis A, Eguı́luz VM, Hernández-Garcı́a E. Wikipedia information flow analysis reveals the scale-free architecture of the semantic space. PLoS One. 2011;6:e17333. find
  29. Yasseri T, Kornai A, Kertész J. A practical approach to language complexity: A Wikipedia case study. PloS One. 2012;7:e48386. find
  30. Dodds PS, Harris KD, Kloumann IM, Bliss CA, Danforth CM. Temporal patterns of happiness and information in a global social network: Hedonometrics and twitter. PLoS One. 2011;6:e26752. find
  31. Morse-Gagné EE. Culturomics: statistical traps muddy the data. Science. 2011;332(6025):35. find
  32. Pechenick EA, Danforth CM, Dodds PS. Characterizing the Google Books corpus: Strong limits to inferences of socio-cultural and linguistic evolution. PLoS One. 2015;10:e0137041. find
  33. Hart M. Project Gutenberg; 1971. find
  34. Ebeling W, Pöschel T. Entropy and long-range correlations in literary English. Europhys Lett. 1994;26:241– 246. find
  35. Schurmann T, Grassberger P. Entropy estimation of symbol sequences. Chaos. 1996;6:414–427. find
  36. Baayen RHH. The effects of lexical specialization on the growth curve of the vocabulary. Comput Linguist. 1996;22:455–480. find
  37. Altmann EG, Cristadoro G, Esposti MD. On the origin of long-range correlations in texts. Proc Natl Acad Sci U S A. 2012;109:11582–11587. find
  38. Moreno-Sánchez I, Font-Clos F, Corral Á. Large-scale analysis of Zipf’s law in English texts. PLoS One. 2016;11:e0147073. find
  39. Williams JR, Bagrow JP, Danforth CM, Dodds PS. Text mixing shapes the anatomy of rank-frequency distributions. Phys Rev E. 2015;91:052811. find
  40. Tria F, Loreto V, Servedio V. Zipf’s, Heaps’ and Taylor’s Laws are determined by the expansion into the adjacent possible. Entropy. 2018;20:752. find
  41. Hughes JM, Foti NJ, Krakauer DC, Rockmore DN. Quantitative patterns of stylistic influence in the evolution of literature. Proceedings of the National Academy of Sciences. 2012;109:7682–7686. find
  42. Reagan AJ, Mitchell L, Kiley D, Danforth CM, Dodds PS. The emotional arcs of stories are dominated by six basic shapes. EPJ Data Science. 2016;5:31. find
  43. Dheeru D, Karra Taniskidou E. UCI Machine Learning Repository; 2017. Available from: http://archive.ics.uci.edu/ml. ↗
  44. Davies M. The Corpus of Contemporary American English (COCA): 560 million words, 1990-present.; 2008. Available from: https://corpus.byu.edu/coca/. ↗
  45. Leech G. 100 million words of English. English Today. 1993;9:9–15. Available from: http://www.natcorp.ox.ac.uk. ↗
  46. Cattuto C, Loreto V, Pietronero L. Semiotic dynamics and collaborative tagging. Proc Natl Acad Sci U S A. 2007;104:1461–1464. find
  47. Loper E, Bird S. NLTK: The Natural Language Toolkit. In: Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics - Volume 1. ETMTNLP ’02. Stroudsburg, PA, USA: Association for Computational Linguistics; 2002. p. 6… find
  48. Manning CD, Raghavan P, Schütze H. Introduction to Information Retrieval. Cambridge University Press; 2008. find
  49. Gerlach M, Font-Clos F, Altmann EG. Similarity of symbol frequency distributions with heavy tails. Phys Rev X. 2016;6:021009. find
  50. McInnes L, Healy J. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv:180203426. 2018;.
  51. ;. We select these 20 authors from the authors of the 100 most downloaded books in order to avoid authors such as “Anonymous”. find
  52. Juola P. Authorship attribution. Foundations and Trends R in Information Retrieval. 2008;1:233–334. find
  53. Stamatatos E. A survey of modern authorship attribution methods. Journal of the American Society for Information Science and Technology. 2009;60:538–556. find
  54. Ioannidis JPA. Why most published research findings are false. PLoS Medicine. 2005;2:e124. find
  55. Open Science Collaboration. Estimating the reproducibility of psychological science. Science. 2015;349:aac4716. find
  56. Camerer CF, Dreber A, Holzmeister F, Ho TH, Huber J, Johannesson M, et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour. 2018;2:637– 644. find
  57. Yucesoy B, Wang X, Huang J, Barabási AL. Success in books: A big data approach to bestsellers. EPJ Data Science. 2018;7:7. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gerlach-2018-standardized-project-gutenberg-corpus-statistical/