Elamur Research⌕
☰
home / papers / 2026 / Evaluating Humanities Theory Alignment in Large Language Models: Incremental Prompting and Statistical Assessment

Evaluating Humanities Theory Alignment in Large Language Models: Incremental Prompting and Statistical Assessment

Summary

The paper asks whether LLMs behave in line with particular humanities theories even when no one tells them to, and whether their behavior shifts as they receive more and more information about a theory. The authors propose a top-down, black-box behavioral alignment workflow. Each theory's core assumption is turned into conditional rules, and from these rules they hand-build small test sets. They apply this to three theories: Davidson's truth-conditional semantics (radical interpretation), Lewis's truth in fiction, and Iser's textual gaps (Leerstelle). Four instruction-tuned models receive three kinds of prompts: sentence-completion ("basic"), theory-uninformed classification, and theory-enriched classification, plus a variant that removes notation. Significance tests are then used to compare the prompt types. Theory-uninformed prompts generally did better for Lewis and Iser, while theory input helped on the Davidson task. Gap detection proved much harder than the binary truth judgments. Feeding theory step by step gave unstable effects that depended on the model. The authors conclude that theory alignment cannot be judged from enriched prompts alone. Reliable LLM use in Computational Literary Studies requires precise operationalizations and reference data.

Key points

  • Gemini 3 Flash scored best across tasks and corpora and stayed robust as theoretical and notational complexity grew. Claude degraded when given formalized theory with logical notation.
  • Performance on the Iser gap task was clearly lower than on the binary tasks:
    • best mean F1 was 0.768 for Iser versus 0.924 for the binary tasks;
    • per-class F1 on Iser was about 0.27.
  • Removing formal symbols lowered performance, which suggests the models may use notation as structural scaffolding. This effect was tested only in the Davidson setting.
  • The datasets are small and manually constructed:
    • 170 items for Davidson;
    • 68–74 items per Lewis setting;
    • 60 Iser items per corpus.
  • The authors queried the models through OpenRouter's API, running each condition just 2–3 times at a temperature of 0.1, which is why they describe the results as indicative. Basic completion prompts did poorly across all metrics because instruction-tuned models do not behave like plain next-token predictors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · History & humanities

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.latechclfl-1.27

@inproceedings{pichler2026evaluating,
  title = {{Evaluating Humanities Theory Alignment in Large Language Models: Incremental Prompting and Statistical Assessment}},
  author = {Pichler, Axel and Pagel, Janis},
  booktitle = {Proceedings of the 10th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature 2026},
  year = {2026},
  doi = {10.18653/v1/2026.latechclfl-1.27},
  url = {https://doi.org/10.18653/v1/2026.latechclfl-1.27}
}

References

34 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. 2025. Playing repeated games with large language models. Nature Human Behaviour, 9(7):1380–1390. find
  2. David Bamman, Sabrina Baur, Mackenzie Ha.nh Cramer, Anna Ho, and Tom McEnaney. 2025. Measuring the Stories in Contemporary Songs. Anthology of Computers and the Humanities, 3:820–844. find
  3. Rebecca M. M. Hicke, Yuri Bizzoni, Pascale Feldkamp, and Ross Deans Kristensen-McLachlan. 2025a. Says Who? Effective Zero-Shot Annotation of Focalization. Anthology of Computers and the Humanities, 3:739–755. find
  4. David Bamman, Kent K. Chang, Li Lucy, and Naitian Zhou. 2024. On Classification with Large Language Models in Cultural Analytics. In CHR2024, pages 494–527, Aarhus. find
  5. Rebecca M. M. Hicke, Brian W. Haggard, Mia Ferrante, Rayhan Khanna, and David Mimno. 2025b. Are You There God? Lightweight Narrative Annotation of Christian Fiction with LMs. Anthology of Computers and the Humanities, 3:1012–1035. find
  6. Boris Beizer. 1995. Black Box Testing: Techniques for Functional Testing of Software and Systems. John Wiley, New York. find
  7. Gregory Currie. 1990. The Nature of Fiction, 1 edition. Cambridge University Press. find
  8. Donald Davidson. 1973. Radical interpretation. Dialectica, 27(3/4):313–328. find
  9. Donald Davidson. 2001. Epistemology Externalized, 1 edition, pages 193–204. Oxford University PressOxford. find
  10. Julia Flanders and Fotis Jannidis, editors. 2018. The Shape of Data in the Digital Humanities: Modeling Texts and Text-based Resources, 1 edition. Routledge, Abingdon, Oxon ; New York, NY : Routledge, 2019. \| Series: Digital research in the arts and humanities. find
  11. Evelyn Gius, Stefanie Messner, and Axel Pichler. 2025. How are Literary Histories written? An LLM-based Analysis of Objects and Perspectives in German Literary History. Anthology of Computers and the Humanities, 3:1090–1107. find
  12. Arianna Graciotti, Franziska Pannach, Valentina Presutti, and Federico Pianzola. 2025. Llamas Don’t Understand Fiction: Application and Evaluation of Rebecca M. M. Hicke and David Mimno. 2025. Looking for the inner music : Probing LLMs’ understanding of literary style. Computational Humanities Resea… find
  13. Saniya Irfan and Syed Juned Ali. 2025. QaLLM: An LLM-based NER Dataset Curation, Annotation and Evaluation in Historical Urdu Elegies. Anthology of Computers and the Humanities, 3:922–937. find
  14. Janina Jacke. 2025. Operationalization and interpretation dependence in computational literary studies. JCLS, 4(1). find
  15. Fotis Jannidis, Rabea Kleymann, Julian Schröter, and Heike Zinsmeister. 2025. Do Large Language Models Understand Literature? Case Studies and Probing Experiments on German Poetry. JCLS, 4(1). find
  16. Jannis Klähn, Janos Borst-Graetz, and Manuel Burghardt. 2025. From dictionaries to LLMs – an evaluation of sentiment analysis techniques for German language data. Computational Humanities Research, 1:e4. find
  17. Leonard Konle, Merten Kröncke, Fotis Jannidis, and Simone Winko. 2024. On the Unity of Literary Change. The Development of Emotions in German Poetry, Prose, and Drama between 1850 and 1920 as a Test Case. In CHR2024, pages 282–300, Aarhus. find
  18. Tilmann Köppe. 2014. 8. Fiktive Tatsachen. In Tobias Klauk and Tilmann Köppe, editors, Fiktionalität, pages 190–208. DE GRUYTER. find
  19. David Lewis. 1983. Truth in Fiction, 1 edition, pages 261–280. Oxford University PressNew York. find
  20. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35. find our card
  21. Pritha Majumdar, Franziska Pannach, Arianna Graciotti, and Johan Bos. 2025. ‘... like a needle in a haystack”: Annotation and Classification of Comparative Statements. In Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and… find
  22. Eric Margolis and Stephen Laurence. 2023. Concepts. In Edward N. Zalta and Uri Nodelman, editors, The Stanford Encyclopedia of Philosophy, Fall 2023 edition. Metaphysics Research Lab, Stanford University. find
  23. Gaspard Michel, Elena V. Epure, Romain Hennequin, and Christophe Cerisara. 2025. Evaluating LLMs for Quotation Attribution in Literary Texts: A Case Study of LLaMa3. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human L… find
  24. Franco Moretti. 2014. “Operationalizing”: or, the Function of Measurement in Modern Literary Theory. The Journal of English Language and Literature, 60:3– 19. find
  25. Janis Pagel, Axel Pichler, and Nils Reiter. 2024. Evaluating In-Context Learning for Computational Literary Studies: A Case Study Based on the Automatic Recognition of Knowledge Transfer in German Drama. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Herita… find
  26. Axel Pichler and Janis Pagel. 2025. Investigating Conceptual Plasticity: On Detecting a Re-Conceptualization of Focalization with Large Language Models. Proceedings of DH2025. find
  27. Axel Pichler, Janis Pagel, and Nils Reiter. 2025. Evaluating LLM-Prompting for Sequence Labeling Tasks in Computational Literary Studies. In Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2025),… find
  28. Axel Pichler and Nils Reiter. 2022. From concepts to texts and back: Operationalization as a core activity of digital humanities. Journal of Cultural Analytics, 7(4). find
  29. Jørgen Sneis. 2018. Phänomenologie und Textinterpre-tation: Studien zur Theoriegeschichte und Methodik der Literaturwissenschaft. De Gruyter. find
  30. Crina Tudor, Beata Megyesi, and Robert Östling. 2025. Prompting the Past: Exploring Zero-Shot Learning for Named Entity Recognition in Historical Texts Using Prompt-Answering LLMs. In Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Hu… find
  31. Xintong Wang, Xiaoyu Li, Xingshan Li, and Chris Biemann. 2024. Probing Large Language Models from a Human Behavioral Perspective. In Proceedings of the Workshop: Bridging Neurons and Symbols for Natural Language Processing and Knowledge Graphs Reasoning (NeusymBridge) @ LREC-COLING-2024, pages 1–7,… find
  32. Nicolas Werner and Nils Reiter. 2025. Between Woolf and Homer: An Explorative Approach to Intertextuality Detection using Large Language Models. Anthology of Computers and the Humanities, 3:382–435. find
  33. Marcus Willand. 2015. Isers impliziter Leser im praxeologischen Belastungstest: Ein literaturwis-senschaftliches Konzept zwischen Theorie und Methode. In Andrea Albrecht, Lutz Danneberg, Olav Krämer, and Carlos Spoerhase, editors, Theorien, Methoden und Praktiken des Interpretierens, pages 237–270.… find
  34. Yaru Wu, Yuri Bizzoni, Pascale Moreira, and Kristoffer Nielbo. 2024. Perplexing Canon: A study on GPT-based perplexity of canonical and non-canonical literary works. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Li… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pichler-2026-evaluating-humanities-theory-alignment-large/