Elamur Research⌕
☰
home / papers / 2026 / Can Large Language Models Facilitate Qualitative Political Narrative Analysis?

Can Large Language Models Facilitate Qualitative Political Narrative Analysis?

Paper · 2026 · Luke Stephens, Clare Llewellyn, Lauren Rogers et al.

Summary

Political narrative research is mostly interpretive and labour-intensive. Earlier NLP work has largely pulled out isolated narrative components such as events, characters and entities, instead of treating narratives as coherent stories. This paper asks whether LLMs can serve as assistants in interpretive narrative analysis. The authors study European Union State of the Union (SOTEU) speeches from 2010–2023, focusing on two themes: migration and solidarity. Two expert narrative scholars independently wrote analyses of each speech and of the whole corpus. Mistral, Llama, ChatGPT-4o and DeepSeek received equivalent prompts that cast them as narrative scholars. The outputs were compared using lexical and semantic similarity metrics and checked through structured human validation of structural, factual and narrative aspects. The two scholars agreed closely in meaning while differing in wording. Model–human similarity was broadly on par with scholar–scholar similarity, often differing only marginally. However, validation revealed weaknesses that the metrics missed. The authors therefore argue that similarity scores alone cannot judge interpretive quality. They propose a Human-LLM Co-Analysis approach for applying qualitative analysis to larger corpora.

Key points

  • The corpus consists of English-language SOTEU addresses by three Commission Presidents: Barroso, Juncker and von der Leyen. The text describes 13 speeches of about 75,000 words, but Table 1 lists 12 addresses totalling 76,046 words.
  • The task design had each scholar and model write a 300-word analysis per speech for each topic. Each also produced one 600-word analysis of how the topics evolve across the full corpus.
  • Four models were benchmarked: Mistral, Llama, ChatGPT-4o and DeepSeek. Agreement between the two scholars served as the reference for interpretive consistency.
  • Human validation found model-specific failures, including factual errors, overly structural or abstract readings, and weak treatment of less prominent narrative threads.
  • The authors acknowledge a limitation: the human benchmark reflects a single interpretive community, and scholars from other traditions or national contexts might read the speeches differently.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 worksEducation · 203 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Luke Stephens, Clare Llewellyn, Lauren Rogers, Constantine Kyritsopoulos, Arman Prangere, Feiteng Long, Peyton Snyder and Laura Cram. 2026. Can Large Language Models Facilitate Qualitative Political Narrative Analysis?. Proceedings of the 3rd Workshop on Natural Language Processing for Political Sciences (PoliticalNLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4rwv2gsphhck

@inproceedings{stephens2026can,
  title = {{Can Large Language Models Facilitate Qualitative Political Narrative Analysis?}},
  author = {Luke Stephens, Clare Llewellyn, Lauren Rogers, Constantine Kyritsopoulos, Arman Prangere, Feiteng Long, Peyton Snyder and Laura Cram},
  booktitle = {Proceedings of the 3rd Workshop on Natural Language Processing for Political Sciences (PoliticalNLP 2026)},
  year = {2026},
  doi = {10.63317/4rwv2gsphhck},
  url = {https://doi.org/10.63317/4rwv2gsphhck}
}

References

25 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muneera Bano, Didar Zowghi, and Jon Whittle. 2023. Exploring qualitative research using llms. find
  2. Kenneth Benoit, Scott De Marchi, Conor Laver, Michael Laver, and Jinshuai Ma. 2025. Using large language models to analyze political texts through natural language understanding. American Journal of Political Science. find
  3. Natalia Chaban, Alister Miskimmon, and Ben O’Loughlin. 2019. Understanding eu crisis diplomacy in the european neighbourhood: strategic narratives and perceptions of the eu in ukraine, israel and palestine. European Security, 28(3):235–250. find
  4. Thomas Colley and Carolijn {van Noort}. 2022. Strategic Narratives, Ontological Security and Global Policy: Responses to China’s Belt and Road Initiative. Palgrave Studies in International Relations. Palgrave Macmillan Ltd., United Kingdom. find
  5. Elahe Rahimtoroghi, Jiaqi Wu, Ruimin Wang, Pranav Anand, and Marilyn Walker. 2017. Modelling protagonist goals and desires in first-person narrative. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 360–369. find
  6. Niels Dekker, Tobias Kuhn, and Marieke van Erp. 2019. Evaluating named entity recognition tools for extracting social networks from novels. PeerJ Computer Science, 5:e189. find
  7. Georg Rehm, Julian Moreno Schneider, Peter Bourgonje, Ankit Srivastava, Jan Nehring, Armin Berger, Luca König, Sören Räuchle, and Jens Gerth. 2017. Event detection and semantic storytelling: Generating a travelogue from a large collection of personal letters. In Proceedings of the Events and Stories… find
  8. Zackary Okun Dunivin. 2025. Scaling hermeneutics: a guide to qualitative coding with llms for reflexive content analysis. EPJ Data Science, 14(1):28. find
  9. Linus Hagström and Karl Gustafsson. 2019. Narrative power: how storytelling shapes east asian international politics. Cambridge Review of International Affairs, 32(4):387–406. find
  10. Alexandra Homolar and Oliver Turner. 2024. Narrative alliances: the discursive foundations of international order. International Affairs, 100(1):203– 220. find
  11. Zhichao Hu and Marilyn Walker. 2017. Inferring narrative causality between event pairs in films. In Proceedings of the 18th annual SIGdial meeting on discourse and dialogue, pages 342–351. find
  12. Muhammad Nihal Hussain, Kiran Kumar Bandeli, Samer Al-khateeb, and Nitin Agarwal. 2018. Analyzing shift in narratives regarding migrants in europe via blogosphere. Text2Story@ ECIR, 585:33–40. find
  13. Sarah Jenner, Dimitris Raidos, Emma Anderson, Stella Fleetwood, Ben Ainsworth, Kerry Fox, Jana Kreppner, and Mary Barker. 2025. Using large language models for narrative analysis: A novel application of generative ai. Methods in Psychology, 12:100183. find
  14. Alessandro Maisto. 2025. Collaborative storytelling and llm: A linguistic analysis of automatically-generated role-playing game sessions. arXiv preprint arXiv:2503.20623. — [arXiv:2503.20623](https://arxiv.org/abs/2503.20623) find
  15. Sebastian Michelmann, Manoj Kumar, Kenneth A. Norman, and Mariya Toneva. 2023. Large language models can segment narrative events similarly to humans. find
  16. Andrew Piper and Sunyam Bagga. 2025. NarraDe-tect: An annotated dataset for the task of narrative detection. In Proceedings of the The 7th Workshop on Narrative Understanding, pages 1–7, Albuquerque, New Mexico. Association for Computational Linguistics. find
  17. Paul Ricoeur. 1984. Time and narrative, Volume 3, volume 3. University of Chicago press. find
  18. Benjamin M. Schmidt. 2015. Plot arceology: A vector-space model of narrative structure. In 2015 IEEE International Conference on Big Data (Big Data), pages 1667–1672. find
  19. Margaret R. Somers. 1994. The narrative constitution of identity: A relational and network approach. Theory and Society, 23(5):605–649. find
  20. Evangelia Spiliopoulou, Eduard Hovy, and Teruko Mitamura. 2017. Event detection using frame-semantic parser. In Proceedings of the Events and Stories in the News Workshop, pages 15–20, Vancouver, Canada. Association for Computational Linguistics.
  21. Dominik Stammbach, Maria Antoniak, and Elliott Ash. 2022. Heroes, villains, and victims, and GPT-3: Automated extraction of character roles without training data. In Proceedings of the 4th Workshop of Narrative Understanding (WNU2022), pages 47–56, Seattle, United States. Association for Computation… find
  22. Brent J Steele. 2008. Ontological security in international relations: Self-identity and the IR state. Routledge. find
  23. Yidan Sun, Qin Chao, and Boyang Li. 2024. Event causality is key to computational story understanding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3493–3511, Mexico Ci… find
  24. Nicholas Sloss Treynor and Joshua McCoy. 2025. A case study on user perception of parameterized llm-generated narratives. In 2025 IEEE Conference on Games (CoG), pages 1–7. IEEE. find
  25. H. White. 1987. The Content of the Form: Narrative Discourse and Historical Representation. Johns Hopkins University Press. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stephens-2026-can-large-language-models-facilitate/