📄 Council of LLMs: Evaluating Capability of Large Language Models to Annotate Propaganda

Summary

This study asks whether large language models can reliably annotate propaganda, a subjective task where human labels are expensive and inconsistent. The authors compare three annotation setups. In the single-annotator setup, one model labels each sentence. In the multi-annotator setup, models label independently and then each model re-annotates after anonymously seeing all candidate labels and rationales. In the hybrid setup, human and LLM labels are combined. They test proprietary GPT models and open-source models on balanced 200-sentence samples from a news corpus (PTC) and a social media corpus (JMBX), averaging three runs per setting.

No LLM configuration reached human-level accuracy. Hybrid annotation consolidated by GPT-4o came closest to the expert labels, followed by single LLM annotators. Multi-annotator consolidation performed worst, although it helped reasoning models. The models tend to over-detect propaganda, producing high recall but many false positives. A qualitative typology of their explanations shows that some disagreements with the gold labels are defensible interpretations rather than errors. The work supports using LLMs as aids within human-supervised annotation pipelines rather than as unsupervised labelers.

Key points

• Data: Propaganda Techniques Corpus (PTC; news, sentence-level labels) and Journalist Media Bias on X (JMBX; tweets from journalists at biased outlets). There were 200 sentences per dataset, split 50/50 between propaganda and non-propaganda, with results averaged over three runs.
• Models: GPT-4o, GPT-4.1 and GPT-o3, plus open-source models including Llama 3.3 70B, QwQ, Qwen and DeepSeek R1. Prompts used the technique definitions of Martino et al. (2020).
• GPT results (P/R/F): GPT-4.1 scored 0.77/0.74/0.74 standalone and 0.78/0.71/0.70 as consolidator. GPT-4o dropped from 0.79/0.74/0.73 to 0.78/0.70/0.68. Only the reasoning model GPT-o3 gained slightly (0.68 → 0.69 F1). Among open-source models, QwQ was the one that benefited from consolidation.
• Frequent false-positive labels: Loaded Language was applied excessively more often than any other label, with Name Calling and Doubt next in line. Open-source models also sometimes assigned Causal Oversimplification to neutral text.
• Error typology of LLM reasoning: the authors sorted disagreements into three types:
  • overgeneralization, where a definition is matched on the surface but without real persuasive intent;
  • defensible divergence, where the LLM's reading is plausible;
  • unconvincing or unsupported reasoning, based on shallow lexical cues.