Elamur Research⌕
☰
home / papers / 2023 / T5 meets Tybalt: Author Attribution in Early Modern English Drama Using Large Language Models

T5 meets Tybalt: Author Attribution in Early Modern English Drama Using Large Language Models

Paper · 2023 · Rebecca M. M. Hicke and David Mimno

Summary

The paper asks whether a fine-tuned large language model can identify the author of very short passages from Early Modern English plays. Classic stylometry handles whole plays well but has trouble with small fragments. The authors combined plays from the Folger EMED collection and the Shakespeare His Contemporaries corpus. They normalized spelling and characters so that source-specific quirks could not leak, then cut each play into speaker utterances of 5–450 words. T5 was fine-tuned to fill in a masked author slot placed in front of each utterance. It was compared with logistic regression, linear SVM, cosine delta and the decoder models Pythia and Falcon. t5-large beat every baseline on short samples. However, it tended to dump uncertain passages onto a few "scapegoat" authors, and it seemed to favour Shakespeare, likely because he appears in its pre-training data. The study shows both the promise and the risks of using LLMs for style analysis. The authors advise against forensic use.

Key points

  • The corpus started from 367 EMED plays plus 181 more from SHC. After filtering, the primary set has 253 single-author plays by 23 authors, and a separate subcorpus holds 23 plays of disputed authorship. Test utterances average only 28.2 words.
  • Split design per author:
    • One play per author is held out entirely, and 200 samples from it go into the test set.
    • Each remaining play contributes 235 training, 15 validation and 50 test samples.
  • T5 small, base and large were trained for 10 epochs with batch sizes 16, 8 and 4. Pythia-1B and Falcon-1B hallucinated heavily, producing 10,221 and 9,180 distinct "author" strings, and were dropped. Cosine delta used a 5,000-unigram vocabulary.
  • On plays co-written by Shakespeare, t5-large gave most samples to him. For The Two Noble Kinsmen, 54% went to Shakespeare and 8.5% to Fletcher. Cosine delta never gave Shakespeare more than 25% of any of these plays.
  • The authors applied the model to post-Early Modern plays, 74 from EMED/SHC plus 67 from Project Gutenberg. The share of samples labelled Shakespeare rose over the centuries, and by majority vote 97% of 1900s plays went to him. The authors read this as relative similarity among known authors, not true resemblance.

Interesting ideas

  • I1. Phrase a classification task the way the model saw text in pre-training: here the label became a masked span marked with T5's own sentinel tag, so no extra classifier head was needed.
  • I2. Before training, remove features that reveal which corpus a text came from, such as spelling variants, odd characters and line-break conventions, so the model cannot take shortcuts.
  • I3. Look at where wrong predictions pile up: confident errors gathered on "scapegoat" classes whose word use is close to the corpus average and whose vocabularies are large.
  • I4. A model's pre-training exposure can inflate results on famous material, as with the co-authored plays the model assigned almost entirely to Shakespeare.
  • I5. The LLM and cosine delta made different kinds of mistakes, so combining them may be more reliable than either alone.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Requests to LLM teacher tiers for silver labels, such as frame roles or relation types, could be written as span-filling in the model's native format, which should reduce malformed or invented outputs.
  • I2 (tree): When mixing treebanks or converting UD dialects, punctuation, tokenization and casing could be normalized so the parser learns syntax rather than treebank identity; a negative-control classifier could check that the source is no longer predictable.
  • I3 (learn): Evaluation could report sink labels, such as a default dependency relation or a catch-all VerbNet class, as an error profile separate from overall accuracy.
  • I4 (answer): Fable and tale QA should include unfamiliar or rewritten stories, so that an LLM teacher or baseline answering from memory can be told apart from real reading of the text.
  • I5 (learn): This supports the witness-parser setup; deliberately choosing witnesses with different inductive biases, such as the perceptron parser and gparse, should make their agreement a stronger silver signal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cornell UniversityIthaca · United States120 works · 59,940 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Rebecca M. M. Hicke and David Mimno. 2023. T5 meets Tybalt: Author Attribution in Early Modern English Drama Using Large Language Models. arXiv:2310.18454.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.18454 · arXiv 2310.18454

@misc{hicke2023t5,
  title = {{T5 meets Tybalt: Author Attribution in Early Modern English Drama Using Large Language Models}},
  author = {Rebecca M. M. Hicke and David Mimno},
  year = {2023},
  doi = {10.48550/arxiv.2310.18454},
  eprint = {2310.18454},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.18454}
}

References

29 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Zehe, L. Konle, L. K. Dümpelmann, E. Gius, A. Hotho, F. Jannidis, L. Kaufmann, M. Krug, F. Puppe, N. Reiter, et al., Detecting scenes in fiction: A new segmentation task, in: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, online, 2021. doi: 10.18653/v1/2021.eacl-main.276 . ↗
  2. J. Pagel, N. Sihag, N. Reiter, Predicting Structural Elements in German Drama, in: Proceedings of the Second Conference on Computational Humanities Research, Monastery of the Grauwzusters, Antwerp, Belgium, 2021. doi: 10.2307/1145292 . ↗
  3. H. Yoo, J. Jin, J. Son, J. Bak, K. Cho, A. Oh, HUE: Pretrained model and dataset for understanding hanja documents of Ancient Korea, in: Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, United States, 2022. doi: 10.18653/v1/2022.findings-naacl.140 . ↗
  4. P. W. Smith, W. Aldridge, Improving authorship attribution: optimizing Burrows’ Delta method, Journal of Quantitative Linguistics 18 (2011) 63–88. doi: 10.1080/09296174.2011.533591 . ↗
  5. M. Eisen, A. Ribeiro, S. Segarra, G. Egan, Stylometric analysis of Early Modern period English plays, Digital Scholarship in the Humanities 33 (2018) 500–528. doi: 10.1093/llc/fqx059 . ↗
  6. A. B. Duque, F. d. A. T. de Carvalho, R. Vimieiro, A Multiview Clustering Approach for Mining Authorial Affinities in Literary Texts, in: 2019 8th Brazilian Conference on Intelligent Systems (BRACIS), Salvador, Bahia, Brazil, 2019. doi: 10.1109/BRACIS.2019.00146 . ↗
  7. A. S. Arefin, R. Vimieiro, C. Riveros, H. Craig, P. Moscato, An Information Theoretic Clustering Approach for Unveiling Authorship Affinities in Shakespearean Era Plays and Poems, PLoS ONE 9 (2018) e111445. doi: 10.1371/journal.pone.0111445 . ↗
  8. O. A. Rosso, H. Craig, P. Moscato, Shakespeare and other English Renaissance authors as characterized by Information Theory complexity quantifiers, Physica A: Statistical Mechanics and its Applications 388 (2009) 916–926. doi: 10.1016/j.physa.2008.11.018 . ↗
  9. L. M. Naeni, H. Craig, R. Berretta, P. Moscato, A novel clustering methodology based on modularity optimisation for detecting authorship affinities in Shakespearean era plays, PloS ONE 11 (2016) e0157988. doi: 10.1371/journal.pone.0157988 . ↗
  10. P. Plecháč, Relative contributions of Shakespeare and Fletcher in Henry VIII: An analysis based on most frequent words and most frequent rhythmic patterns, Digital Scholarship in the Humanities 36 (2021) 430–438. doi: 10.1093/llc/fqaa032 . ↗
  11. G. Taylor, Empirical Middleton: Macbeth, Adaptation, and Microauthorship, Shakespeare Quarterly 65 (2014) 239–272. doi: 10.1353/shq.2014.0030 . ↗
  12. G. Taylor, J. V. Nance, Imitation or collaboration? Marlowe and the early Shakespeare canon, Shakespeare, Origins and Originality 68 (2015) 32–47. doi: 10.1017/cbo9781316258736.003 . ↗
  13. D. Freebury-Jones, Unsound deductions in early modern attribution: the case of Thomas Watson, ANQ: A Quarterly Journal of Short Articles, Notes and Reviews 33 (2020) 164–171. doi: 10.1080/0895769x.2019.1612231 . ↗
  14. P. Rizvi, The problem of microattribution, Digital Scholarship in the Humanities 34 (2019) 606–615. doi: 10.1093/digitalsh/fqy066 . ↗
  15. M. Koppel, J. Schler, S. Argamon, Authorship attribution in the wild, Language Resources and Evaluation 45 (2011) 83–94. doi: 10.1007/s10579-009-9111-2 . ↗
  16. H. Zhang, P. Nie, Y. Wen, X. Yuan, Authorship attribution for short texts with author-document topic model, in: Knowledge Science, Engineering and Management: 11th International Conference, KSEM 2018, Changchun, China, 2018. doi: 10.1007/978-3-319-99365-2_3 . ↗
  17. R. Layton, P. Watters, R. Dazeley, Authorship attribution for twitter in 140 characters or less, in: 2010 Second Cybercrime and Trustworthy Computing Workshop, Ballarat, Victoria, Australia, 2010. doi: 10.1109/ctc.2010.17 . ↗
  18. H. Azarbonyad, M. Dehghani, M. Marx, J. Kamps, Time-aware authorship attribution for short text streams, in: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Santiago, Chile, 2015. doi: 10.1145/2766462.2767799 . ↗
  19. J. Huertas-Tato, A. Huertas-Garcia, A. Martin, D. Camacho, PART: Pre-trained Authorship Representation Transformer, arXiv paper, 2022. doi: 10.48550/ARXIV.2209.15373 . ↗
  20. M. Najafi, E. Tavan, Text-to-Text Transformer in Authorship Verification Via Stylistic and Semantical Analysis, in: CLEF 2022 – Conference and Labs of the Evaluation Forum, Belmeloro University Complex, Bologna, Italy, 2022. doi: 10.18653/v1/w17-4914 . ↗
  21. M. Mueller, Shakespeare His Contemporaries: a corpus of Early Modern Drama 1550-1650, Online corpus, 2015. URL: https://github.com/JonathanReeve/corpus-SHC. ↗
  22. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al., Pythia: A suite for analyzing large language models across training and scaling, in: Proceedings of the 40th International Conference on Machine Learning, Honolulu, Hawaii, USA, 2023, pp. 2397–2430. doi: 10.48550/arXiv.2304.01373 . ↗
  23. G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei, J. Launay, The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only (2023). find
  24. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, Journal of Machine Learning 21 (2020) 5485–5551. doi: 10.48550/ARXIV.1910.10683 . ↗
  25. J. Burrows, ‘Delta’: a measure of stylistic difference and a guide to likely authorship, Literary and linguistic computing 17 (2002) 267–287. doi: 10.1093/llc/17.3.267 . ↗
  26. T. Wood, faststylometry: Burrows Delta, 2021. URL: https://github.com/fastdatascience/faststylometry.git, python package version 0.5. ↗
  27. Oxford Dictionary of National Biography, Oxford University Press, Oxford, England, 2004. find
  28. A. Augustyn, Authors of the Medieval and Renaissance Eras, 1100 to 1660, Encyclopaedia Britannica, Inc, New York, New York, 2014. find
  29. E. H. C. Oliphant, Problems of Authorship in Elizabethan Dramatic Literature, Modern Philology 8 (1911) 411–459. doi: 10.1086/386843 . ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hicke-2023-t5-meets-tybalt-author-attribution/