Elamur Research⌕
☰
home / papers / 2020 / Quasi Error-free Text Classification and Authorship Recognition in a large Corpus of English Literature based on a Novel Feature Set

Quasi Error-free Text Classification and Authorship Recognition in a large Corpus of English Literature based on a Novel Feature Set

Paper · 2020 · Arthur M. Jacobs and Annette Kinder

Summary

Jacobs and Kinder test whether a small, interpretable set of document-level measures can do two jobs: sort literary texts into broad categories and identify their authors. Their data is the Gutenberg Literary English Corpus (GLEC), from which they use 2722 texts drawn from six groups: children's and youth literature, essays, novels, plays, poetry and short stories. Each text becomes a vector of ten numbers. Five describe style: mean log token frequency, type-token ratio, a sentence syllable index, sentence count and mean sonority. Five describe affective content and come from word2vec similarities computed in the SentiArt tool: affective-aesthetic potential, plus closeness to disgust, fear, happiness and surprise. A boosted two-layer MLP, run in JMP with 5-fold cross-validation, misclassified fewer than 1.3% of held-out texts in the six-way category task. A Naive Bayes baseline was wrong for about a third of them. Authorship recognition with the identical feature set was also close to error-free. Frequency, TTR, sonority and surprise were the most diagnostic features. The result offers cheap, transparent predictors that work for texts of any length, from short poems to long novels, and can feed experiments in reading psychology.

Key points

  • GLEC holds about 900 novels, 500 short stories, 300 children's texts, 200 poetry items, 100 plays and 500 non-fiction works, totalling roughly 12M sentences and 250M words. The study used 2722 of these texts.
  • The ten features were selected from a larger SentiArt feature pool. JMP 14 Pro predictor screening picked the five strongest style features and the five strongest content features.
  • MLP setup:
    • one hidden layer of 100 TanH units, followed by one of 25 linear units;
    • boosting with up to 10 component models and learning rate 0.1;
    • k = 5 cross-validation and a fixed seed of 1.
  • On a test set of 544 texts, the MLP's misclassification rate was below .013, compared with .33 for Naive Bayes.
  • Feature importance was measured as each feature's total effect in a sensitivity analysis, with values above 0.1 counted as important. TTR, FREQ, SONSCORE and SURPRISE ranked highest.

Interesting ideas

  • I1. The authors run Naive Bayes, which assumes features are independent, alongside a nonlinear MLP; the large accuracy gap shows that the features mainly work in combination.
  • I2. Emotion scores come from vector similarity to emotion label words, and AAP is the mean similarity to 60 positive labels minus the mean similarity to 60 negative ones; this avoids the coverage gaps of a fixed emotion dictionary.
  • I3. A simple mean sonority score built from phonemes captures a sound-level property of words that earlier work linked to perceived word beauty.
  • I4. Feature importance is reported as a total effect from sensitivity analysis, covering each feature's contribution both alone and through interactions, on a 0–1 scale.
  • I5. A ten-number style and affect profile is enough to tell plays, poems, essays and children's stories apart almost perfectly.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Use a feature-independent baseline as a standard control in evaluations, so we can see whether a component's gain comes from feature interactions or from single cues.
  • I2 (frames): Attach such signed affect scores to frame participants and events, rather than to the whole text, to support irony and implicature cues while keeping each score traceable to its label words.
  • I3 (tokens): Store a sonority value as an optional lexical attribute in the en crate, so poetry-oriented readers and stylistic checks can query it.
  • I4 (learn): Apply the same total-effect analysis to the features of our perceptron parser or repair rules, so we can explain which signals actually drive decisions.
  • I5 (answer): Run a cheap genre check first and use it to route each input to the matching reader, such as the fable/tale QA reader or a school-test reader, before deeper analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Freie Universität BerlinBerlin · Germany10 works · 283 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works General Social Sciences · 87 works Literature and Literary Theory · 340 works

🧭 Topics 1 English · 1,373 works

Cite

Cite as Arthur M. Jacobs and Annette Kinder. 2020. Quasi Error-free Text Classification and Authorship Recognition in a large Corpus of English Literature based on a Novel Feature Set. arXiv:2010.10801.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2010.10801 · arXiv 2010.10801

@misc{jacobs2020quasi,
  title = {{Quasi Error-free Text Classification and Authorship Recognition in a large Corpus of English Literature based on a Novel Feature Set}},
  author = {Arthur M. Jacobs and Annette Kinder},
  year = {2020},
  doi = {10.48550/arxiv.2010.10801},
  eprint = {2010.10801},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2010.10801}
}

References

36 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacobs AM (2018) The Gutenberg English Poetry Corpus: Exemplary Quantitative Narrative Analyses. Front. Digit. Humanit. 5:5. doi: 10.3389/fdigh.2018.00005 ↗
  2. Kucera, H., & Francis, W. N. Computational analysis of present-day American English. Providence: Brown University Press, 1967. find
  3. Neidorf, L., Krieger, M. S., Yakubek, M., Chaudhuri, P., & Dexter, J. P. (2019). Large-scale quantitative profiling of the Old English verse tradition. Nat Hum Behav, 3(6), 560–567. Retrieved from http://www.nature.com/ articles/s41562-019-0570-1 ↗
  4. Samothrakis, S., & Fasli, M. (2015). Emotional sentence annotation helps predict fiction genre. PloS one 10(11):e0141922. ↗
  5. Jacobs, A. M. (2018). (Neuro-)Cognitive Poetics and Computational Stylistics. Scientific Study of Literature, 8:1, pp. 165-208. https://doi.org/10.1075/ssol.18002.jac. ↗
  6. Cafiero, F. & Camps, J. P. (2019). Why Molière most likely did write his plays, Science Advances, 27 Novembre 2019, DOI : 10.1126/sciadv.aax5489 ↗
  7. Schöch, C. (2017). Topic modeling genre: an exploration of french classical and enlightenment drama. Digit. Human. Q. doi: 10.5281/zenodo.166356. ↗
  8. Reagan, AJ, Mitchell, L, Kiley, D, Danforth, CD, & Dodds, PS (2016). The emotional arcs of stories are dominated by six basic shapes. EPJ Data Science 5(1):31. ↗
  9. Hills TT, Proto E, Sgroi D, Seresinhe CI (2019) Historical analysis of national subjective wellbeing using millions of digitized books. Nature Human Behaviour 3:1271–1275, URL http://dx.doi.org/10. 1038/s41562-019-0750-z. ↗
  10. Jacobs, A. M., Schuster, S., Xue, S. & Lüdtke, J. (2017). Whats in the brain that ink may character ....: A Quantitative Narrative Analysis of Shakespeares 154 Sonnets for Use in Neurocognitive Poetics, Scientific Study of Literature 7:1, 4–51. doi 10.1075/ssol.7.1.02jac. ↗
  11. Xue, S., Lüdtke, J., Sylvester, T., & Jacobs, A. M. (2019). Reading Shakespeare Sonnets : Combining Quantitative Narrative Analysis and Predictive Modeling — an Eye Tracking Study. Journal of Eye Movement Research, 12(5). https://doi.org/10.16910/jemr.12.5.2 ↗
  12. Hsu, C. T., Jacobs, A. M., Citron, F., & Conrad, M. (2015). The emotion potential of words and passages in reading Harry Potter - An fMRI study. Brain and Language, 142, 96-114. ↗
  13. Lüdtke, J., Meyer-Sickendiek, B., and Jacobs, A. M. (2014). Immersing in the stillness of an early morning: testing the mood empathy hypothesis in poems. Psychol. Aesthet. 8, 363– 377. doi: 10.1037/a0036826 ↗
  14. Xue S, Jacobs AM and Lüdtke J (2020) What Is the Difference? Rereading Shakespeares Sonnets — An Eye Tracking Study. Front. Psychol. 11:421. doi: 10.3389/fpsyg.2020.00421 ↗
  15. Jacobs AM (2019) Sentiment Analysis for Words and Fiction Characters From the Perspective of Computational (Neuro-)Poetics. Front. Robot. AI 6:53. doi: 10.3389/frobt.2019.00053 ↗
  16. Jakobson, R., and Lévi-Strauss, C. (1962). “Les chats” de charles baudelaire. LHomme 2, 5–21. doi: 10.3406/hom.1962.366446 ↗
  17. Jockers, M. and Witten, D. (2010). A comparative study of machine learning methods for authorship attribution. Literary and Linguistic Computing, 25: 215–23 ↗
  18. Kessler, B., Nunberg, G., & Schutze, H. (1997). Automatic detection of text genre. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics. Madrid, Spain, pages 32–38. ↗ our card
  19. Kim, E., Padó, S., & Klinger, R. (2017). Investigating the relationship between literary genres and emotional plot development. Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, 17–26. find
  20. Jacobs, A. M., & Kinder, A. (2019). Computing the Affective-Aesthetic Potential of Literary Texts, Artifical Intelligence, 1:1, 11–27; doi:10.3390/ai1010002 ↗
  21. Bestgen, Y. (1994). Can emotional valence in stories be determined from words? Cogn. Emot. 8, 21–36. doi: 10.1080/02699939408408926 ↗
  22. Westbury, C., Keith, J., Briesemeister, B. B., Hofmann, M. J. & Jacobs, A. M. (2015). Avoid violence, rioting, and outrage; approach celebration, delight, and strength: Using large text corpora to compute valence, arousal, and the basic emotions. Quarterly Journal of Experimental Psychology 68, 1599–1622, doi: 10.1080/17470218.2014.970204. ↗
  23. Jacobs AM (2017) Quantifying the Beauty of Words: A Neurocognitive Poetics Perspective. Front. Hum. Neurosci. 11:622. doi: 10.3389/fnhum.2017.00622 ↗
  24. Jacobs, A. M., & Kinder, A. (2018). What makes a metaphor literary? Answers from two computational studies, Metaphor and Symbol, 33:2, 85-100, DOI: 10.1080/10926488.2018.1434943 ↗
  25. Klare, G. R. (1974–1975). Assessing readability. Reading Research Quarterly, 10, 62-102. find
  26. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. Retrieved from https://arxiv.org/abs/1301.3781 ↗ our card
  27. Strobl, C., Malley, J., and Tutz, G. (2009). An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychol. Methods 14, 323–348. doi: 10.1037/a0016973 ↗
  28. Mosteller, R. F. and Wallace, D. L. (1964). Inference and Disputed Authorship: The Federalist. Reading, MA: Addison-Wesley. find
  29. Eder, M. (2015). Does size matter? Authorship attribution, small samples, big problem. Digital Scholarship in the Humanities, 30(2): 167–82. find
  30. Jacobs AM (2015) Neurocognitive poetics: methods and models for investigating the neuronal and cognitive-affective bases of literature reception. Front. Hum. Neurosci. 9:186. doi: 10.3389/fnhum.2015.00186 ↗
  31. Willems, R. & Jacobs, A.M. (2016). Caring about Dostoyevsky: The untapped potential of studying literature. Trends in Cognitive Sciences, 20, 243-245. https://doi.org/10.1016/j.tics.2015.12.009 ↗
  32. Schroeder, S., Würzner, K. M., Heister, J., Geyken, A., & Kliegl, R. (2015). childLex: A lexical database of German read by children. Behavior Research Methods, 47(4), 1085-1094. ↗
  33. Jacobs AM, Hofmann MJ and Kinder A (2016) On Elementary Affective Decisions: To Like Or Not to Like, That Is the Question. Front. Psychol. 7:1836. doi: 10.3389/fpsyg.2016.01836 ↗
  34. Bird, S., Klein, E., & Loper, E. (2009). Natural language processing with Python. OReilly Media. find
  35. Schmid, H. (1994). Probabilistic Part-of-Speech Tagging Using Decision Trees. Proceedings of International Conference on New Methods in Language Processing, Manchester, UK. find
  36. Saltelli, A. (2002). Sensitivity analysis for importance assessment. Risk Anal. 22, 579–590. doi: 10.1111/0272-4332.00040 Acknowledgements None Author contributions A.J. designed and performed the study. A.K. and A.J. performed the ML part and analysed the results. AJ drafted the manuscript, A.J. and A.K contributed both to the final version of the paper. Competing interests The authors declare no competing interests. Additional information ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jacobs-2020-quasi-error-free-text-classification/