Elamur Research⌕
☰
home / papers / 2026 / Comparing BERT Sentence-Pair Classification and Few-Shot LLM Prompting for Detecting Threat and Solution Framing in German Climate News

Comparing BERT Sentence-Pair Classification and Few-Shot LLM Prompting for Detecting Threat and Solution Framing in German Climate News

Paper · 2026 · Raven Adam, David Maier and Marie Kogler

Summary

The paper looks at automatically detecting whether sentences in German-language climate reporting stress dangers, remedies, both, or neither. Communication scholars care about this balance because it shapes how audiences react, but coding large collections by hand is too slow. The authors built two pairs of independent binary detectors, one for threat and one for solution. The first pair fine-tunes deepset/gbert-large using BERT's sentence-pair input, with the previous sentence supplied as the second segment for context. The second pair prompts the open-weights Llama 4 Maverick with eight examples, chain-of-thought reasoning, structured output with a confidence value, and German-language coding rules grounded in speech-act categories. Both were evaluated on 440 hand-coded articles from two Austrian dailies. The fine-tuned encoder scores F1 0.83 on both tasks, against 0.78 for the LLM. Removing the preceding-sentence context costs BERT about ten F1 points. The practical takeaway is that a small supervised encoder still wins when labelled data exists, while prompting is a workable fallback when it does not.

Key points

  • Corpus: 440 articles published January–June 2024, from Kronen Zeitung (304) and Der Standard (136).
    • Articles were retrieved from WISO (GBI-Genios) with the query klima* OR CO2.
    • They were coded in MAXQDA using a scheme developed with linguists.
    • Threat and solution are treated as independent dimensions, not mutually exclusive classes.
  • Main result: gbert-large with sentence-pair input reaches F1 0.83 for both threat and solution. Few-shot CoT prompting of Llama 4 Maverick reaches 0.78.
  • Context ablation: dropping the previous sentence lowers BERT's F1 by roughly ten points.
  • Disagreements: the two systems disagree on about 20% of sentences, and BERT matches the gold label about twice as often as the LLM in those cases.
    • The LLM's full-article context causes "context bleeding", which inflates false positives for solution framing.
    • The LLM's confidence averaged above 0.93 despite error rates over 20%, and no confidence threshold improved F1.
  • Caveats acknowledged by the authors:
    • The train/test split was done at sentence level rather than article level, which may inflate BERT's scores.
    • Results come from a single random split.
    • No intercoder reliability figures are reported.
    • Only one model per paradigm was tested.

Interesting ideas

  • I1. Rather than one four-way classifier, the authors run two independent yes/no detectors whose outputs combine into "both" or "neither"; this keeps each decision simple and handles overlapping labels naturally.
  • I2. Supplying just the previous sentence as a second input segment is a very cheap way to add discourse context, yet it accounted for about ten F1 points here.
  • I3. The finding that whole-article context made the LLM worse ("context bleeding") shows that more context can blur local judgments rather than sharpen them.
  • I4. The authors tested whether the LLM's self-reported confidence could serve as a quality filter by sweeping thresholds against gold labels, and found that it could not.
  • I5. Reading chain-of-thought outputs side by side across structurally similar cases exposed reasoning that contradicted the final labels.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): speech-act, implicature and irony flags in prag could each be a separate detector, so that one sentence can legitimately carry several pragmatic functions.
  • I2 (frames): coreference and frame-role decisions could take the preceding sentence's tree as a paired input, giving the decision local context without processing the whole document.
  • I3 (world): when assigning states or attitudes to entities, evidence should be scoped to the sentence or clause it comes from, so that material elsewhere in the text does not leak into locally neutral statements.
  • I4 (learn): LLM teacher tiers should not be trusted on their own stated certainty; witness-parser agreement or a threshold sweep against held-out gold should decide which silver labels are kept.
  • I5 (answer): test explanations with minimal-pair inputs and check that the explanation stays consistent with the answer, which acts as a faithfulness control on the readers' justifications.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Graz University of TechnologyGraz · Austria3 works · 5 citations🎓University of GrazGraz · Austria2 works

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Raven Adam, David Maier and Marie Kogler. 2026. Comparing BERT Sentence-Pair Classification and Few-Shot LLM Prompting for Detecting Threat and Solution Framing in German Climate News. arXiv:2606.26489.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2606.26489 · arXiv 2606.26489

@misc{adam2026comparing,
  title = {{Comparing BERT Sentence-Pair Classification and Few-Shot LLM Prompting for Detecting Threat and Solution Framing in German Climate News}},
  author = {Raven Adam, David Maier and Marie Kogler},
  year = {2026},
  doi = {10.48550/arxiv.2606.26489},
  eprint = {2606.26489},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.26489}
}

References

48 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adam, R., Kogler, M., and Scholger, M. (2023). Aufwind in der Berichterstattung zum Klimaschutz: Langfristige Entwicklung von Themen und Stimmungsbildern in österreichischen Zeitungen. Zeitschrift für Digitale Geisteswissenschaften, 8. ↗
  2. Ali, M. N. and Hassan, N. (2022). A survey of computational framing analysis approaches. In Proceedings of EMNLP 2022. find
  3. Alizadeh, M., Kubli, M., Samei, Z., et al. (2025). Open-source LLMs for text annotation: A practical guide for model setting and fine-tuning. Journal of Computational Social Science, 8:17. ↗
  4. Badullovich, N., Grant, W. J., and Colvin, R. M. (2020). Framing climate change for effective communication: A systematic map. Environmental Research Letters, 15(12):123002. ↗
  5. Barez, F., Wu, T. Y., Arcuschin, I., Lan, M., Wang, V., Siegel, N., Collignon, N., Neo, C., Lee, I., Paren, A., and Bibi, A. (2025). Chain-of-thought is not explainability. Preprint, alphaXiv. find
  6. Boumans, J. W. and Trilling, D. (2016). Taking stock of the toolkit: An overview of relevant automated content analysis approaches and techniques for digital journalism scholars. Digital Journalism, 4(1):8–23. find
  7. Brown, T. B., Mann, B., Ryder, N., et al. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), volume 33. ↗
  8. Bucher, M. J. J. and Martini, M. (2024). Fine-tuned ‘small’ LLMs (still) significantly outperform zero-shot generative AI models in text classification. arXiv preprint arXiv:2406.08660. ↗
  9. Card, D., Boydstun, A. E., Gross, J. H., Resnik, P., and Smith, N. A. (2015). The media frames corpus: Annotations of frames across issues. In Proceedings of ACL-IJCNLP 2015, pages 438–444.
  10. Carlson, D. et al. (2026). The use of LLMs to annotate data in management research: Foundational guidelines and warnings. Strategic Management Journal. ↗
  11. Chan, B., Schweter, S., and Möller, T. (2020). German’s next language model. In Proceedings of COLING 2020, pages 6788–6796.
  12. Cohan, A., Beltagy, I., King, D., Dalvi, B., and Weld, D. (2019). Pretrained language models for sequential sentence classification. In Proceedings of EMNLP-IJCNLP 2019, pages 3693–3699. find
  13. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL 2020, pages 8440–8451. find
  14. Dablander, F., Wimmer, S., et al. (2025). Media coverage of climate activist groups in Germany. Climatic Change, 178(8):144. ↗
  15. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019, pages 4171–4186.
  16. Dunivin, Z. O. (2025). Scaling hermeneutics: A guide to qualitative coding with LLMs for reflexive content analysis. EPJ Data Science, 14:28. ↗
  17. Edwards, A. and Camacho-Collados, J. (2024). Language models for text classification: Is in-context learning enough? In Proceedings of LREC-COLING 2024. ↗
  18. Entman, R. M. (1993). Framing: Toward clarification of a fractured paradigm. Journal of Communication, 43(4):51–58. ↗
  19. Farjam, M., Meyer, H., and Lohkamp, M. (2025). A practical guide and case study on how to instruct LLMs for automated coding during content analysis. Social Science Computer Review. OnlineFirst. ↗
  20. Feldman, L. and Hart, P. S. (2018). Is there any hope? How climate change news imagery and text influence audience emotions and support for climate mitigation policies. Risk Analysis, 38(3):585–602. find
  21. Feldman, L., Hart, P. S., and Milosevic, T. (2015). Polarizing news? Representations of threat and efficacy in leading US newspapers’ coverage of climate change. Public Understanding of Science, 26(4):481–497. find
  22. Galke, L. and Scherp, A. (2024). Are we really making much progress in text classification? A comparative review. arXiv preprint arXiv:2204.03954v6. ↗
  23. Gilardi, F., Alizadeh, M., and Kubli, M. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences (PNAS), 120(30):e2305016120. ↗
  24. Grimmer, J. and Stewart, B. M. (2013). Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 21(3):267– 297. ↗
  25. Guenther, L., Brüggemann, M., and Elkobros, S. (2022). From global doom to sustainable solutions: International news magazines’ multimodal framing of our future with climate change. Journalism Studies, 23(1):131–148. ↗
  26. Guenther, L., Jörges, S., Mahl, D., and Brüggemann, M. (2024). Framing as a bridging concept for climate change communication: A systematic review based on 25 years of literature. Communication Research. ↗
  27. Hart, P. S. and Feldman, L. (2014). Threat without efficacy? Climate change on U.S. network news. Science Communication, 36(3):325–351. find
  28. Hase, V., Mahl, D., Schäfer, M. S., and Keller, T. R. (2021). Climate change in news media across the globe: An automated analysis of issue attention and themes in climate change coverage in 10 countries (2006–2018). Global Environmental Change, 70:102353. ↗
  29. Hovsepian, K., Liu, D., and Murugesan, S. (2024). Label with confidence: Effective confidence calibration and ensembles in LLM-based data labeling. In GenAI for E-Commerce Workshop. find
  30. Krippendorff, K. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3):411–433. ↗
  31. Liu, S., Guo, L., Mays, K., Betke, M., and Wijaya, D. T. (2019). Detecting frames in news headlines and its application to analyzing news framing trends surrounding U.S. gun violence. In Proceedings of CoNLL 2019, pages 504–514. find
  32. Luoma, J. and Pyysalo, S. (2020). Exploring cross-sentence contexts for named entity recognition with BERT. In Proceedings of COLING 2020, pages 904–914. ↗
  33. Maier, D. (2025). Automated detection of threat- and solution-oriented climate change framing in German-language newspapers. Master’s thesis, University of Graz. find
  34. Matthes, J. and Kohring, M. (2008). The content analysis of media frames: Toward improving reliability and validity. Journal of Communication, 58(2):258–279. ↗
  35. Mei, L., Yao, J., Ge, Y., Wang, Y., Bi, B., Cai, Y., Liu, J., Li, M., Li, Z., Zhang, D., et al. (2025). A survey of context engineering for large language models. arXiv preprint arXiv:2507.13334. ↗
  36. Meyer, H., Farjam, M., Rauxloh, H., and Brüggemann, M. (2025). From disruptive protests to disrupted news frames: Comparing German news on climate protests. Journalism. ↗
  37. Mondshine, I., Paz-Argaman, T., and Tsarfaty, R. (2025). Beyond English: The impact of prompt translation strategies across languages and tasks in multilingual LLMs. In Proceedings of LoResMT 2025 (ACL Workshop), pages 81–104. ↗
  38. Nisbet, M. C. (2009). Communicating climate change: Why frames matter for public engagement. Environment: Science and Policy for Sustainable Development, 51(2):12– 23. ↗
  39. Piskorski, J., Stefanovitch, N., Da San Martino, G., and Nakov, P. (2023). SemEval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup. In Proceedings of SemEval-2023, pages 2343–2361.
  40. Schulhoff, S., Ilie, M., Balepur, N., et al. (2024). The prompt report: A systematic survey of prompt engineering techniques. arXiv preprint arXiv:2406.06608. ↗
  41. Stammbach, D., Webersinke, N., Bingler, J. A., Kraus, M., and Leippold, M. (2023). A dataset for detecting real-world environmental claims. In Findings of ACL 2023. ↗
  42. Thier, K. and Lin, T.-T. (2022). How solutions journalism shapes support for collective climate change adaptation. Environmental Communication, 16(8):1027–1045. ↗
  43. Thier, K. and Wu, X. (2024). Framing climate solutions: An exploratory quantitative content analysis. Environmental Communication, 19(3):359–375. ↗
  44. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems 30 (NeurIPS), pages 5998–6008.
  45. Webersinke, N., Kraus, M., Bingler, J. A., and Leippold, M. (2022). ClimateBERT: A pretrained language model for climate-related text. In Proceedings of the AAAI Fall Symposium. arXiv:2110.12010. ↗
  46. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 24824–24837.
  47. Wolf, T., Debut, L., Sanh, V., et al. (2020). Transformers: State-of-the-art natural language processing. In Proceedings of EMNLP 2020: System Demonstrations, pages 38–45.
  48. Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., and Yang, D. (2024). Can large language models transform computational social science? Computational Linguistics, 50(1):237–291. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/adam-2026-comparing-bert-sentence-pair-classification/