Elamur Research⌕
☰
home / papers / 2026 / Decoding News Narratives: A Critical Analysis of Large Language Models in Framing Detection

Decoding News Narratives: A Critical Analysis of Large Language Models in Framing Detection

Paper · 2026 · Valeria Pastorino, Jasivan Alex Sivakumar and Nafise Sadat Moosavi

Summary

Framing in news means highlighting some aspects of an event and downplaying others to steer how readers interpret it. Detecting it automatically is hard: manual annotation is expensive, and supervised classifiers tend to fail outside the topic they were trained on. The authors test whether instruction-following LLMs (GPT-4, GPT-3.5-Turbo, Llama 3 and FLAN-T5) can do this as a binary task, deciding whether a headline is framed or not. They compare zero-shot, few-shot and explanation-based prompts. Experiments use the Gun Violence Frame Corpus (GVFC) and a new out-of-domain set of headlines on many topics, released as the ITW dataset. LLMs, and GPT-4 in particular, transfer across domains better than fine-tuned models. However, results change a lot with prompt wording, and models make systematic mistakes on ambiguous headlines. GPT-4 often treats emotionally charged wording as framing, which produces false positives. The authors also show that agreement across several models can flag disputed gold labels. This gives a cheap way to audit datasets when resources are limited.

Key points

  • GVFC setup: only headlines relevant to gun violence are used, and nine frame categories are collapsed into a single framed/not-framed label. The evaluation set has 2,594 headlines (1,293 framed, 1,301 not framed), so the majority baseline is 50.15%. Another 22 headlines are held out as few-shot examples.
  • Models and prompts: GPT-4-0613 and GPT-3.5-turbo-0613 are run at temperature 0, alongside open-source Llama 3 and FLAN-T5. Few-shot variants use focused in-domain, varied in-domain, cross-domain and mixed-domain examples.
  • FLAN-T5 results: without fine-tuning, FLAN-T5 stays around chance accuracy (about 50%). Fine-tuned FLAN-T5 Base reaches an F1 of about 79.8 on GVFC but falls to about 63.9 on ITW, which illustrates how domain shift hurts supervised models.
  • Main error pattern: GPT-4 tends to confuse emotional language with framing.
  • Resources: the ITW dataset is available at github.com/vpastorino/ITW-dataset, and the paper proposes cross-model consensus as a method for annotation auditing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SheffieldSheffield · United Kingdom113 works · 21,542 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

Cite

Cite as Valeria Pastorino, Jasivan Alex Sivakumar and Nafise Sadat Moosavi. 2026. Decoding News Narratives: A Critical Analysis of Large Language Models in Framing Detection. Proceedings of the 3rd Workshop on Natural Language Processing for Political Sciences (PoliticalNLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2quz7g7fh7u4

@inproceedings{pastorino2026decoding,
  title = {{Decoding News Narratives: A Critical Analysis of Large Language Models in Framing Detection}},
  author = {Valeria Pastorino, Jasivan Alex Sivakumar and Nafise Sadat Moosavi},
  booktitle = {Proceedings of the 3rd Workshop on Natural Language Processing for Political Sciences (PoliticalNLP 2026)},
  year = {2026},
  doi = {10.63317/2quz7g7fh7u4},
  url = {https://doi.org/10.63317/2quz7g7fh7u4}
}

References

31 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dallas Card, Amber E. Boydstun, Justin H. Gross, Philip Resnik, and Noah A. Smith. 2015. The Media Frames Corpus: Annotations of Frames Across Issues. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lang…
  2. Marwa Abdulhai, Gregory Serapio-Garcia, Clément Crepy, Daria Valter, John Canny, and Natasha Jaques. 2023. Moral Foundations of Large Language Models. ArXiv:2310.15337 [cs].
  3. Afra Feyza Akyürek, Lei Guo, Randa Elanwar, Prakash Ishwar, Margrit Betke, and Derry Tanti Wijaya. 2020. Multi-Label and Multilingual News Framing Analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8614–8624, Online. Association for Computatio… find
  4. Minje Choi, Jiaxin Pei, Sagar Kumar, Chang Shu, and David Jurgens. 2023. Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11370–11403, Singa… find
  5. Mohammad Ali and Naeemul Hassan. 2022. A Survey of Computational Framing Analysis Approaches. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9335–9348, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  6. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan… find
  7. Eric Baumer, Elisha Elovic, Ying Qin, Francesca Polletta, and Geri Gay. 2015. Testing and Comparing Computational Approaches for Identifying the Language of Framing in Political News. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics… find
  8. Vibhu Bhatia, Vidya Prasad Akavoor, Sejin Paik, Lei Guo, Mona Jalal, Alyssa Smith, David Assefa Tofu, Edward Edberg Halim, Yimeng Sun, Margrit Betke, Prakash Ishwar, and Derry Tanti Wijaya. 2021. OpenFraming: Open-sourced Tool for Computational Framing Analysis of Multilingual Data. In Proceedings o… find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding.
  10. Marco Binotto and Marco Bruno. 2018. Spazi me-diali delle migrazioni. framing e rappresentazioni del confine nell’informazione italiana. Lingue e Linguaggi, 25(0). find
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  12. Paul DiMaggio, Manish Nag, and David Blei. 2013. Exploiting affinities between topic modeling and the sociological perspective on culture: Application to newspaper coverage of u.s. government arts funding. Poetics, 41(6):570–606. Topic Models and the Cultural Sciences. find
  13. Amber E. Boydstun, Dallas Card, Justin Gross, Paul Resnick, and Noah A. Smith. 2014. Tracking the Development of Media Frames within and across Policy Issues. find
  14. Robert M. Entman. 1993. Framing: Toward Clarification of a Fractured Paradigm. Journal of Communication, 43(4):51–58. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1460-2466.1993.tb01304.x. — [doi:10.1111/j.1460-2466.1993.tb01304.x](https://doi.org/10.1111/j.1460-2466.1993.tb01304.x) ↗
  15. Bjorn Burscher, Rens Vliegenthart, and Claes H. de Vreese. 2016. Frames beyond words: Applying cluster and sentiment analysis to news coverage of the nuclear power issue. Social Science Computer Review, 34(5):530–545. find
  16. Robert M. Entman. 2007. Framing Bias: Media in the Distribution of Power. Journal of Communication, 57(1):163–173. find
  17. Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Framing and Agenda-setting in Russian News: a Computational Analysis of Intricate Political Strategies. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3…
  18. Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023b. SemEval-2023 Task 3: Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multi-lingual Setup. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval…
  19. Fabrizio Gilardi, Charles R. Shipan, and Bruno Wüest. 2021. Policy diffusion: The issue-definition stage. American Journal of Political Science, 65(1):21–35. find
  20. Erving Goffman. 1974. Frame Analysis: An Essay on the Organization of Experience. Harper colophon books. Harvard University Press. find
  21. Antonina Sinelnik and Dirk Hovy. 2024. Narratives at conflict: Computational analysis of news framing in multilingual disinformation campaigns. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 225–237, Bangkok, Th… find
  22. Siyi Liu, Lei Guo, Kate Mays, Margrit Betke, and Derry Tanti Wijaya. 2019. Detecting Frames in News Headlines and Its Application to Analyzing News Framing Trends Surrounding U.S. Gun Violence. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 504–514, H… find
  23. David Sun, Artem Abzaliev, Hadas Kotek, Christopher Klein, Zidi Xiu, and Jason Williams. 2023. DELPHI: Data for Evaluating LLMs’ Performance in Handling Controversial Issues. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 820–827, Sin… find
  24. Julia Mendelsohn, Ceren Budak, and David Jurgens. 2021. Modeling Framing in Immigration Discourse on Social Media. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2219– 2263, Online. Association…
  25. Isidora Tourni, Lei Guo, Taufiq Husada Daryanto, Fabian Zhafransyah, Edward Edberg Halim, Mona Jalal, Boqi Chen, Sha Lai, Hengchang Hu, Margrit Betke, Prakash Ishwar, and Derry Tanti Wijaya. 2021. Detecting Frames in News Headlines and Lead Images in U.S. Gun Violence Coverage. In Findings of the As… find
  26. Nona Naderi and Graeme Hirst. 2017. Classifying Frames at the Sentence Level in News Articles. In RANLP 2017 - Recent Advances in Natural Language Processing Meet Deep Learning, pages 536–542. Incoma Ltd. Shoumen, Bulgaria. find
  27. Viet-An Nguyen, Jordan Boyd-Graber, Philip Resnik, and Kristina Miler. 2015. Tea party in the house: A hierarchical ideal point topic model and its application to republican legislators in the 112th congress. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics… find
  28. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
  29. Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models. ArXiv:2303.10420 [cs]. find
  30. OpenAI. 2023. Gpt-4 technical report.
  31. Jakub Piskorski, Nicolas Stefanovitch, Valerie-Anne Bausier, Nicolo Faggiani, Jens Linge, Sopho Kharazi, Nikolaos Nikolaidis, Giulia Teodori, Bertrand De Longueville, Brian Doherty, et al. 2023a. News categorization, framing and Qi Yu. 2022. “Again, Dozens of Refugees Drowned”: A Computational Study… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pastorino-2026-decoding-news-narratives-critical-analysis/