Elamur Research⌕
☰
home / papers / 2022 / Computational analyses of the topics, sentiments, literariness, creativity and beauty of texts in a large Corpus of English Literature

Computational analyses of the topics, sentiments, literariness, creativity and beauty of texts in a large Corpus of English Literature

Paper · 2022 · Arthur M. Jacobs and Annette Kinder

Summary

Jacobs and Kinder analyze nearly all of the Gutenberg Literary English Corpus (GLEC). Earlier quantitative work had covered only its poetry subset. They selected 2722 texts by 112 authors across six categories: children and youth literature, essays, novels, plays, poems and short stories.

Each text was POS-tagged and lemmatized with TreeTagger and reduced to content words. SentiArt then assigned every word lexical and affective values. On top of this, the authors ran three studies:

  1. LDA topic modelling combined with sentiment scoring.
  2. Text-level indices of literariness, creativity and beauty, built from measures of semantic complexity found in the literature.
  3. Text-category classification and authorship recognition using only these five complexity features.

The main findings:

  • Essays are the most positive category and plays the most negative.
  • Plays score as the most literary category, and poems and plays as the most creative.
  • Of Austen's novels, Emma is predicted to be the most beautiful.
  • The five features reach accuracies between .75 and .97.

The study gives reading psychology and digital humanities baselines and testable predictions for other book corpora.

Key points

  • Corpus: GLEC has about 2950 texts, roughly 12 million sentences and 250 million words. The study kept only authors with at least five texts in a category, so that 5-fold cross-validation was possible. This left 2722 texts.
  • Topic modelling:
    • The model was LDA via gensim, with 50 topics, 50 words per topic and 2500 iterations.
    • Only nouns (without proper names) and verbs occurring at least 100 times were used. This choice was based on topic-coherence pilots.
  • Sentiment results:
    • Whitman (novels) and Keats (poetry) came out as the most positive authors, and Kipling as the most negative in both.
    • Shakespeare's plays were the most negative and fearful.
    • Vocabulary decay appears in all six categories. It is linked to time pressure, not to the number of works an author wrote.
  • Five semantic-complexity features:
    • Literariness: intra-textual variance (ITV) and stepwise distance (SWD), from van Cranenburgh et al.
    • Creativity: forward flow (FF), from Gray et al.
    • Beauty: two Kintsch-inspired indices, HARM and VARI.
  • Classification with an untuned MLP:
    • Authorship recognition was harder than category classification, with misclassification rates from 0 to 28%.
    • Story authors were easiest to recognize and novel authors hardest.
    • SWD and ITV were usually the most diagnostic features.

Interesting ideas

  • I1. Restricting classes to those with enough items for every cross-validation fold makes the benchmark sound by construction, instead of patching sparse classes afterwards.
  • I2. Attaching lexicon-based valence to discovered topics gives each theme an emotional tone, not just a word list.
  • I3. Forward flow and stepwise distance measure how far meaning moves between consecutive units, and so treat creativity or literariness as a property of the semantic trajectory.
  • I4. Intra-textual variance captures how spread out a text's meaning is across its parts, a cheap global coherence signal.
  • I5. A handful of interpretable features, with importances and per-author "footprints", rivals heavier models while staying explainable.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation sets for genre or speaker recognition could enforce a minimum count per class, so that each fold contains every class and scores stay comparable.
  • I2 (frames): Valence could be aggregated over frame participants and events rather than raw words, giving an explainable mood profile per agent or episode.
  • I3 (back): Generated text could be checked by comparing its step-to-step semantic drift with that of the source, which would flag outputs that are too flat or too jumpy.
  • I4 (world): Variance could be computed over the entities and states introduced per sentence, to detect scene shifts or fragmented world models in a story.
  • I5 (answer): Small named-feature profiles could explain why a reader module assigns a text to a genre or author, which suits Mova's explainability requirement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Freie Universität BerlinBerlin · Germany10 works · 283 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works General Social Sciences · 87 works Literature and Literary Theory · 340 works

🧭 Topics 2 English · 1,373 works Sentiment & emotion · 608 works

Cite

Cite as Arthur M. Jacobs and Annette Kinder. 2022. Computational analyses of the topics, sentiments, literariness, creativity and beauty of texts in a large Corpus of English Literature. arXiv:2201.04356.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2201.04356 · arXiv 2201.04356

@misc{jacobs2022computational,
  title = {{Computational analyses of the topics, sentiments, literariness, creativity and beauty of texts in a large Corpus of English Literature}},
  author = {Arthur M. Jacobs and Annette Kinder},
  year = {2022},
  doi = {10.48550/arxiv.2201.04356},
  eprint = {2201.04356},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2201.04356}
}

References

44 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Appel, M., & Hanauer, D. (2021). The Psychological and Social Effects of Literariness: Formal Features and Paratextual Information. In: D. Kuiken & A.M. Jacobs (eds.), Handbook of Empirical Literary Studies. Berlin: de Gruyter Bird, S., Klein, E., & Loper, E. (2009). Natural language processing with Python. OReilly Media. ↗
  2. Boucher, J., and Osgood, C. E. (1969). The pollyanna hypothesis. J. Verb. Learn. Verb. Behav. 8: 1–8. doi: 10.1016/S0022-5371(69)80002-2 ↗
  3. Cafiero, F. & Camps, J. P. (2019). Why Molière most likely did write his plays, Science Advances, 27 Novembre 2019, DOI : 10.1126/sciadv.aax5489 ↗
  4. Eder, M. (2015). Does size matter? Authorship attribution, small samples, big problem. Digital Scholarship in the Humanities, 30(2): 167–82. find
  5. Elliott, J. (2016). Vocabulary Decay in Category Romance. Digital Scholarship in the Humanities, Vol. 31, No. 2, 321-332. doi:10.1093/llc/fqu069 ↗
  6. Gambino, R., Pulvirenti, G., Sylvester, T., Jacobs, A.M., & Lüdtke, J. (2020). The Foregrounding Assessment Matrix: An Interface for Qualitative-Quantitative Interdisciplinary Research. Enthymema, 26, pp. 254-277. http://dx.doi.org/ 10.13130/2037-2426/14387 ↗
  7. Gray, K., Anderson, S., Chen, E. et al., (2019). Forward flow: A new measure to quantify free thought and predict creativity, American Psychologist, Vol. 74, No. 5, 539–554, http://dx.doi.org/10.1037/amp0000391 ↗
  8. Herrmann, J.B., Jacobs, A.M., & Piper, A. (2021). Computational Stylistics. In: D. Kuiken & A.M. Jacobs (eds.), Handbook of Empirical Literary Studies. Berlin: de Gruyter Hills TT, Proto E, Sgroi D, Seresinhe CI (2019) Historical analysis of national subjective wellbeing using millions of digitized books. Nature Human Behaviour 3:1271–1275, URL http://dx.doi.org/10. 1038/s41562-019-0750-z. ↗
  9. Hsu, C. T., Jacobs, A. M., Citron, F., & Conrad, M. (2015). The emotion potential of words and passages in reading Harry Potter - An fMRI study. Brain and Language, 142, 96-114. ↗
  10. Kintsch, W. (2012). Musing about beauty. Cogn. Sci. 36, 635–654. doi: 10.1111/j. 1551-6709.2011.01229.x Jacobs AM (2015) Neurocognitive poetics: methods and models for investigating the neuronal and cognitive-affective bases of literature reception. Front. Hum. Neurosci. 9:186. doi: 10.3389/fnhum.2015.00186 find
  11. Jacobs AM (2017) Quantifying the Beauty of Words: A Neurocognitive Poetics Perspective. Front. Hum. Neurosci. 11:622. doi: 10.3389/fnhum.2017.00622 ↗
  12. Jacobs AM (2018a) The Gutenberg English Poetry Corpus: Exemplary Quantitative Narrative Analyses. Front. Digit. Humanit. 5:5. doi: 10.3389/fdigh.2018.00005 ↗
  13. Jacobs, A. M. (2018b). (Neuro-)Cognitive Poetics and Computational Stylistics. Scientific Study of Literature, 8:1, pp. 165-208. https://doi.org/10.1075/ssol.18002.jac. ↗
  14. Jacobs AM (2019) Sentiment Analysis for Words and Fiction Characters From the Perspective of Computational (Neuro-)Poetics. Front. Robot. AI 6:53. doi: 10.3389/frobt.2019.00053 ↗
  15. Jacobs, A. M., & Kinder, A. (2018). What makes a metaphor literary? Answers from two computational studies, Metaphor and Symbol, 33:2, 85-100, DOI: 10.1080/10926488.2018.1434943 ↗
  16. Jacobs, A. M., & Kinder, A. (2019b). Computing the Affective-Aesthetic Potential of Literary Texts, Artifical Intelligence, 1:1, 11–27; doi:10.3390/ai1010002 ↗
  17. Jacobs, A. M., Schuster, S., Xue, S. & Lüdtke, J. (2017). Whats in the brain that ink may character ....: A Quantitative Narrative Analysis of Shakespeares 154 Sonnets for Use in Neurocognitive Poetics, Scientific Study of Literature 7:1, 4–51 . doi 10.1075/ssol.7.1.02jac. ↗
  18. Jacobs AM, Herrmann B, Lauer G, Lüdtke J and Schroeder S (2020) Sentiment Analysis of Children and Youth Literature: Is There a Pollyanna Effect? Front. Psychol. 11:574746. doi: 10.3389/fpsyg.2020.574746 ↗
  19. Kim, E., Padó, S., & Klinger, R. (2017). Investigating the relationship between literary genres and emotional plot development. Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, 17–26. find
  20. Kucera, H., & Francis, W. N. Computational analysis of present-day American English. Providence: Brown University Press, 1967. find
  21. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211. doi: 10.1037/0033-295X.104.2.211 ↗
  22. Lüdtke, J., Meyer-Sickendiek, B., and Jacobs, A. M. (2014). Immersing in the stillness of an early morning: testing the mood empathy hypothesis in poems. Psychol. Aesthet. 8, 363–377. doi: 10.1037/a0036826 ↗
  23. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. Retrieved from https://arxiv.org/abs/ 1301.3781 ↗ our card
  24. Mosteller, R. F. and Wallace, D. L. (1964). Inference and Disputed Authorship: The Federalist. Reading, MA: Addison-Wesley. find
  25. Neidorf, L., Krieger, M. S., Yakubek, M., Chaudhuri, P., & Dexter, J. P. (2019). Large-scale quantitative profiling of the Old English verse tradition. Nat Hum Behav, 3(6), 560–567. Retrieved from http://www.nature.com/ articles/s41562-019-0570-1 ↗
  26. Reagan, AJ, Mitchell, L, Kiley, D, Danforth, CD, & Dodds, PS (2016). The emotional arcs of stories are dominated by six basic shapes. EPJ Data Science 5(1):31. ↗
  27. Rehurek, R., & Sojka, P. (2010). Software framework for topic modelling with large corpora. In R. Witte, H. Cunningham, J. Patrick, E. Beisswanger, E. Buyko, U. Hahn, K. Verspoor, & A. R. Coden (Eds.), Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks (pp. 45–50). Valletta, Malta: ELRA. find
  28. Reisenzein, R. (2009). »Emotions as metarepresentational states of mind: Naturalizing the belief– desire theory of emotion«. Cognitive Systems Research, 10, 6–20 ↗
  29. Röder, M., Both, A., and Hinneburg, A. (2015). “Exploring the Space of Topic Coherence Measures”. In: Proceedings of the Eight International Conference on Web Search and Data Mining. Shanghai. http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf . ↗ our card
  30. Salgaro, M. (2018). Historical introduction to the special issue on literariness. Scientific Study of Literature, 8(1), 6–18. https://doi.org/10.1075/ssol.00005.sal ↗
  31. Samothrakis, S., & Fasli, M. (2015). Emotional sentence annotation helps predict fiction genre. PloS one 10(11):e0141922. ↗
  32. Schmid, H. (1994). Probabilistic Part-of-Speech Tagging Using Decision Trees. Proceedings of International Conference on New Methods in Language Processing, Manchester, UK. find
  33. Schmidhuber, J. (1997). Low-complexity art. Leonardo, Journal of the International Society for the Arts, Sciences, and Technology, 30, 97–103. ↗
  34. Schöch, C. (2017). Topic modeling genre: an exploration of french classical and enlightenment drama. Digit. Human. Q. doi: 10.5281/zenodo.166356. ↗
  35. Schroeder, S., Würzner, K. M., Heister, J., Geyken, A., and Kliegl, R. (2015). childLex: a lexical database of German read by children. Behav. Res. Methods 47, 1085–1094. doi: 10.3758/s13428-014-0528-1 ↗
  36. Simonton, D. K. (1990). Lexical choices and aesthetic success: A computer content analysis of 154 Shakespeare sonnets. Computers and the Humanities, 24, 254–264. ↗
  37. Shawn, G., Weingart, S., and Milligan, I. (2012). “Getting Started with Topic Modeling and MALLET”. In: The Programming Historian. http://programminghistorian.org/lessons/topicmodelingandmallet . ↗
  38. Strobl, C., Malley, J., and Tutz, G. (2009). An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychol. Methods 14, 323–348. doi: 10.1037/a0016973 ↗
  39. Xue, S., Lüdtke, J., Sylvester, T., & Jacobs, A. M. (2019). Reading Shakespeare Sonnets : Combining Quantitative Narrative Analysis and Predictive Modeling — an Eye Tracking Study. Journal of Eye Movement Research, 12(5). https://doi.org/10.16910/jemr.12.5.2 ↗
  40. Xue S, Jacobs AM and Lüdtke J (2020) What Is the Difference? Rereading Shakespeares Sonnets — An Eye Tracking Study. Front. Psychol. 11:421. doi: 10.3389/fpsyg.2020.00421 ↗
  41. Willems, R. & Jacobs, A.M. (2016). Caring about Dostoyevsky: The untapped potential of studying literature. Trends in Cognitive Sciences, 20, 243-245. https://doi.org/10.1016/j.tics.2015.12.009 ↗
  42. van Cranenburgh, A., van Dalen-Oskam, K., & van Zundert, J. (2019). Vector space explorations of literary language. Language Resources and Evaluation, 53(4), 625–650. https://doi.org/10.1007/s10579-018-09442-4 ↗
  43. Vendler, H. (1997). The art of Shakespeare’s sonnets. Cambridge, MA: Harvard University Press. ↗
  44. Wehbe, L. et al. (2014). Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PLoS ONE 9, e112575. Figure Captions Figure 1a-f. Top topics for each of the six text categories (alphabetical order from top to bottom: *Figure 2. ‘happiness’ and ‘fear’ scores as computed by SentiArt for the six text categories.* Figure 3a-f. ‘happiness’ and ‘fear’ scores as computed by SentiArt for the all authors in the six text categories. Figure 4a-f. Top topics for selected authors: a) Alcott (top row, left), b) Bunyan (top row right), c) Barrie (middle row, left), d) Defoe (middle row, right), e) Kipling (bottom row, left) and f) Keats (bottom row, right). Figure 5a-c. Entropy (H) as a function of text segment (1-100) for selected authors. *Figure 6. Entropy (H) as a function of text segment for the six text categories.* *Figure 7. ITV and SWD measures for authors in the six text categories.* *Figure 8. SWD (1 st and 2 nd principal components) for three selected authors of poetry.* *Figure 9. Harmony and Variety for Jane Austen’s six novels.* *Figure 10. Receiver Operating Characteristic (ROC) curves for the four text categories* (validation set). *Figure 11. Category Footprints based on Feature Importances/FI* *Figure 12. Author Footprints based on FI values (probability that a given feature is important* for authorship recognition) for three exemplary authors from the POE category. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jacobs-2022-computational-analyses-topics-sentiments-literariness/