Elamur Research⌕
☰
home / papers / 2026 / Large Language Models Unpack Complex Political Opinions through Target-Stance Extraction

Large Language Models Unpack Complex Political Opinions through Target-Stance Extraction

Paper · 2026 · Ozgur Togay, Javier Garcia-Bernardo, Florian Kunneman et al.

Summary

Most computational studies of polarization sort political talk into coarse partisan bins. This misses how beliefs about specific policies, people and issues relate to one another. The authors test whether instruction-tuned LLMs can perform Target-Stance Extraction (TSE), which means finding what a post is about and the attitude expressed toward it, without task-specific fine-tuning.

They built a corpus of 1,084 comments from r/NeutralPolitics, a strictly moderated, evidence-oriented subreddit, retrieved through the Pushshift Archives. Trained annotators labelled the comments using an iteratively developed codebook of 138 targets plus an open "Other" option. An expert then validated a balanced gold set of 200 posts. Proprietary and open-weight models (GPT-4.1, o3, Gemini, LLaMA, Qwen, Gemma and others) were evaluated with zero-shot, few-shot and conversational-context prompts against the full target list.

The strongest system reached F1 of 0.76 for targets and 0.87 for stance, roughly matching skilled human annotators. It also held up on posts where the annotators disagreed. This suggests LLMs can support fine-grained, scalable opinion measurement in political text with little supervision.

Key points

  • Dataset: 1,084 annotated r/NeutralPolitics comments, of which 200 form an expert-validated gold set. The gold set is balanced: 50 posts with no target and 50 per stance class, covering 58 targets. Both sets are released at github.com/zgrtgy/llm-tse.
  • Text length: Posts average 106 words, far longer than PStance (30) and SemEval 2016 (17). This makes the task harder than tweet-based benchmarks.
  • Human agreement: Krippendorff's α for targets was 0.48. Annotators chose the same target for 568 posts, and for 385 posts they matched on the target and the stance together. Neutral stance caused the most disagreement.
  • Error analysis (o3): Performance was best for concrete categories such as Key Politicians (F1 0.93) and weakest for diffuse ones such as Trust & Institutions (0.60). Neutral stance was the hardest class (F1 0.76).
  • Disagreement cases: On 20 posts where annotators disagreed, o3 matched the expert label 12 times, gave plausible labels 3 times and was wrong 5 times. Its typical error was reading too much into weak cues.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Utrecht UniversityUtrecht · Netherlands39 works · 1,603 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Ozgur Togay, Javier Garcia-Bernardo, Florian Kunneman and Anastasia Giachanou. 2026. Large Language Models Unpack Complex Political Opinions through Target-Stance Extraction. Proceedings of the 3rd Workshop on Natural Language Processing for Political Sciences (PoliticalNLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3ivad4jwtca2

@inproceedings{togay2026large,
  title = {{Large Language Models Unpack Complex Political Opinions through Target-Stance Extraction}},
  author = {Ozgur Togay, Javier Garcia-Bernardo, Florian Kunneman and Anastasia Giachanou},
  booktitle = {Proceedings of the 3rd Workshop on Natural Language Processing for Political Sciences (PoliticalNLP 2026)},
  year = {2026},
  doi = {10.63317/3ivad4jwtca2},
  url = {https://doi.org/10.63317/3ivad4jwtca2}
}

References

43 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abu Ubaida Akash, Ahmed Fahmy, and Amine Trabelsi. 2025. Can Large Language Models Address Open-Target Stance Detection? (arXiv:2409.00222).
  2. American National Election Studies. 2021. find
  3. Anes 2020 time series study full release [dataset and documentation]. Available at https://www.electionstudies.org. ↗
  4. Christopher A. Bail, Lisa P. Argyle, Taylor W. Brown, John P. Bumpus, Haohan Chen, M. B. Fallin Hunzaker, Jaemin Lee, Marcus Mann, Friedolin Merhout, and Alexander Volfovsky. 2018. Exposure to opposing views on social media can increase political polarization. Proceedings of the National Academy of… find
  5. Eytan Bakshy, Solomon Messing, and Lada A. Adamic. 2015. Exposure to ideologically diverse news and opinion on Facebook. Science, 348(6239):1130–1132. find
  6. Pablo Barberá, John T. Jost, Jonathan Nagler, Joshua A. Tucker, and Richard Bonneau. 2015. Tweeting From Left to Right: Is Online Political Communication More Than an Echo Chamber? Psychological Science, 26(10):1531–1542. find
  7. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The Pushshift Reddit Dataset. Proceedings of the International AAAI Conference on Web and Social Media, 14:830–839. find
  8. Data Availability The datasets, the codebook, the code, and other supplementary materials associated with this study are publicly available at the following link https://github.com/zgrtgy/llm-tse. Ethical Statement This project received ethical approval from the Ethical Review Board of the Faculty o… ↗
  9. Pedro Brum, Matheus Cândido Teixeira, Renato Vimieiro, Eric Araújo, Wagner Meira Jr, and Gisele Lobo Pappa. 2022. Political polarization on Twitter during the COVID-19 pandemic: A case study in Brazil. Social Network Analysis and Mining, 12(1):140. find
  10. Michael Burnham. 2024. Stance Detection: A Practical Guide to Classifying Political Beliefs in Text. (arXiv:2305.01723). — [arXiv:2305.01723](https://arxiv.org/abs/2305.01723) find
  11. This study received funding from a research grant awarded by the Applied Data Science focus area at Utrecht University. Generative AI tools were used only for formatting issues and grammatical checks. find
  12. Kareem Darwish. 2020. Quantifying Polarization on Twitter: The Kavanaugh Nomination. (arXiv:2001.02125). — [arXiv:2001.02125](https://arxiv.org/abs/2001.02125) find
  13. Michael Davern, Rene Bautista, Jeremy Freese, Pamela Herd, and Stephen L. Morgan. 2024. General social survey 1972-2024 [machine-readable data file]. Principal Investigator: Michael Davern; Co-Principal Investigators: Rene Bautista, Jeremy Freese, Pamela Herd, and Stephen L. Morgan. Sponsored by the… find
  14. Daniel DellaPosta. 2020. Pluralistic Collapse: The “Oil Spill” Model of Mass Opinion Polarization. American Sociological Review, 85(3):507–536. find
  15. James N. Druckman, Donald P. Green, and Shanto Iyengar. 2024. Does Affective Polarization Contribute to Democratic Backsliding in The ANNALS of the American America? Academy of Political and Social Science. find
  16. Venkata Rama Kiran Garimella and Ingmar Weber. 2017. A Long-Term Analysis of Polarization on Twitter. Proceedings of the International AAAI Conference on Web and Social Media, 11(1):528–531. find
  17. Shalmoli Ghosh, Prajwal Singhania, Siddharth Singh, Koustav Rudra, and Saptarshi Ghosh. 2019. Stance Detection in Web and Social Media: A Comparative Study. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 75–87. Springer International Publishing. find
  18. Valentin Hofmann, Hinrich Schütze, and Janet B. Pierrehumbert. 2022. The Reddit Politosphere: A Large-Scale Text and Network Resource of Online Political Discourse. Proceedings of the International AAAI Conference on Web and Social Media, 16:1259–1267. find
  19. Yingjie Li, Krishna Garg, and Cornelia Caragea. 2023. A New Direction in Stance Detection: Target-Stance Extraction in the Wild. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10071–10085. Association for Computational Lingui…
  20. Yingjie Li, Tiberiu Sosea, Aditya Sawant, Ajith Jayaraman Nair, Diana Inkpen, and Cornelia Caragea. 2021. P-Stance: A Large Dataset for Stance Detection in Political Domain. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2355–2365. Association for Computational…
  21. Jennifer McCoy, Tahmina Rahman, and Murat Somer. 2018. Polarization and the Global Crisis of Democracy: Common Patterns, Dynamics, and Pernicious Consequences for Democratic Polities. American Behavioral Scientist, 62(1):16–42. find
  22. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 Task 6: Detecting Stance in Tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41. Association for Computational Linguistics.
  23. Lillio Mok, Michael Inzlicht, and Ashton Anderson. 2023. Echo Tunnels: Polarized News Sharing Online Runs Narrow but Deep. Proceedings of the International AAAI Conference on Web and Social Media, 17:662–673. find
  24. N Newman, A Ross Arguedas, CT Robertson, RK Nielsen, and R Fletcher. 2025. Digital news report 2025. Technical report, Reuters Institute for the Study of Journalism. find
  25. Shanto Iyengar, Yphtach Lelkes, Matthew Levendusky, Neil Malhotra, and Sean J. Westwood. 2019. The Origins and Consequences of Affective Polarization in the United States. Annual Review of Political Science, 22(Volume 22, 2019):129–146. find
  26. Fuqiang Niu, Min Yang, Ang Li, Baoquan Zhang, Xiaojiang Peng, and Bowen Zhang. 2024. A Challenge Dataset and Effective Models for Conversational Stance Detection. (arXiv:2403.11145).
  27. Dilek Küçük and Fazli Can. 2021. Stance Detection: A Survey. ACM Computing Surveys, 53(1):1–37. find
  28. Yunus Emre Orhan. 2022. The relationship between affective polarization and democratic backsliding: Comparative evidence. Democratization, 29(4):714–735. find
  29. Stephen Lacy, Brendan R. Watson, Daniel Riffe, and Jennette Lovejoy. 2015. Issues and Best Practices in Content Analysis. Journalism & Mass Communication Quarterly, 92(4):791– 811. find
  30. J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1):159–174. find
  31. Xingyu Peng, Zhenkun Zhou, Chong Zhang, and Ke Xu. 2024. Online Social Behavior Enhanced Detection of Political Stances in Tweets. Proceedings of the International AAAI Conference on Web and Social Media, 18:1207–1219. find
  32. Kai Ruggeri, Friederike Stock, S. Alexander Haslam, Valerio Capraro, Paulo Boggio, Naomi Ellemers, Aleksandra Cichocka, Karen M. Douglas, David G. Rand, Sander van der Linden, Mina Cikara, Eli J. Finkel, James N. Druckman, Michael J. A. Wohl, Richard E. Petty, Joshua A. Tucker, Azim Shariff, Michele… find
  33. Maryam Shafiei, Hossein Rahmani, Amirhossein Derakhshan, and Milad Allahgholi. 2025. find
  34. Caskow: Context-Aware Stance Detection Using External Knowledge-Augmented LLM. In 2025 11th International Conference on Web Research (ICWR), pages 123–129. find
  35. Elizabeth Suhay, Emily Bello-Pardo, and Brianna Maurer. 2018. The Polarizing Effects of Online Partisan Criticism: Evidence from Two Experiments. The International Journal of Press/Politics, 23(1):95–115. find
  36. Anissa Tanweer, Emily Kalah Gade, P. M. Krafft, and Sarah Dreier. 2021. Why the Data Revolution Needs Qualitative Thinking. Harvard Data Science Review, 3(3). find
  37. Petter Törnberg. 2024. Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages. Social Science Computer Review. find
  38. Felicity M. Turner-Zwinkels, Jochem Van Noord, Rebekka Kesberg, Efrain García-Sánchez, Mark J. Brandt, Toon Kuppens, Matthew J. Easterbrook, Lien Smets, Paulina Gorska, Marta Marchlewska, and Tomas Turner-Zwinkels. 2023. Affective Polarization and Political Belief Systems: The Role of Political Iden… find
  39. Jochem Van Noord, Felicity M Turner-Zwinkels, Rebekka Kesberg, Mark J Brandt, Matthew J Easterbrook, Toon Kuppens, and Bram Spruyt. 2024. The nature and structure of European belief systems: exploring the varieties of belief systems across 23 European countries. European Sociological Review. find
  40. Veniamin Veselovsky and Ashton Anderson. 2023. Reddit in the Time of COVID. Proceedings of the International AAAI Conference on Web and Social Media, 17:878–889. find
  41. Stefan Sylvius Wagner, Maike Behrendt, Marc Ziegele, and Stefan Harmeling. 2024. The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions. (arXiv:2406.12480). — [arXiv:2406.12480](https://arxiv.org/abs/2406.12480) find
  42. Muheng Yang, Xidao Wen, Yu-Ru Lin, and Lingjia Deng. 2017. Quantifying Content Polarization on Twitter. In 2017 IEEE 3rd International Conference on Collaboration and Internet Computing (CIC), pages 299–308. IEEE. find
  43. Xinshu Zhao, Jun S. Liu, and Ke Deng. 2013. Assumptions behind Intercoder Reliability Indices. Communication Yearbook, 36(1):419–480. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/togay-2026-large-language-models-unpack-complex/