Elamur Research⌕
☰
home / papers / 2026 / Propaganda Signals in LLMs: Perspectival Divergence and Narrative Framing in the Russia-Ukraine War

Propaganda Signals in LLMs: Perspectival Divergence and Narrative Framing in the Russia-Ukraine War

Paper · 2026 · Ofir Shabat, Ido Guy and Kira Radinsky

Summary

LLMs are increasingly used to translate and explain news about ongoing wars. Common bias tests, such as left–right scores or demographic disparities, reveal little about whether a model reproduces one side's narrative of a conflict. The authors address this gap for the Russia-Ukraine war by measuring "perspectival divergence": how closely a model's output matches Russian-oriented versus Ukrainian-oriented accounts of the same incident.

They assembled 100 real events. Each has a manually written neutral English statement and short RU- and UA-oriented reference snippets drawn from news outlets and Telegram channels. Eight LLMs translated the neutral statements into Russian and Ukrainian under five prompt styles. An LLM judge (Llama-3.1-8B-Instruct) then scored each translation's semantic distance to the reference in the same language. A five-category propaganda scheme was used to explain how any alignment is expressed.

Each model showed a consistent leaning and a characteristic mix of techniques that held across prompt styles, and factuality-focused metrics did not detect these patterns. The work shows that a model can appear neutral under conventional evaluation while still reproducing conflict-specific propaganda.

Key points

  • Dataset: 100 event triples, each pairing a neutral English statement (about 15.5 words on average) with RU and UA reference snippets (about 33–35 words). Annotators agreed 98% on whether RU/UA pairs describe the same event and 92% on event-core consistency of the full triples. The dataset, prompts and code are released on GitHub.
  • Models and prompts: eight models were tested: mistral-7b-instruct-v0.3, qwen2.5-7b, llama-3.1-8b-instruct, gemini-3-pro-preview, deepseek-llm-7b-chat, falcon-7b-instruct, moonshot-v1-8k and gpt-5.1. The five prompt groups were baseline, social influence, news, journalist voice and country-aligned.
  • Judge robustness: RU-vs-UA differences were tested with paired t-tests and BH-FDR correction. Re-scoring with GPT-5.1 and Falcon-7B-Instruct as alternative judges gave high agreement on which side's reference was closer (Cohen's κ = 0.89 and 0.86).
  • Propaganda taxonomy: the scheme sorts techniques into five groups: Toponymy & Naming, Source & Credibility, Framing & Narrative, Social Pressure & Identity, and Emotional Manipulation. Models differ in which category drives their divergence. Examples:
    • Falcon rendered a Russian withdrawal from Snake Island as a "liberation."
    • Mistral inserted the slur "rashist."
    • Qwen labelled POWs as "Ukrainian-Nazi."
  • Conclusions and limits: the authors argue that divergence has two separate dimensions, direction and mechanism, so mitigation should target each model's dominant techniques. The study covers only one conflict and two languages, and only the translation task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technion – Israel Institute of TechnologyHaifa · Israel49 works · 5,306 citations🎓Ben-Gurion University of the NegevBeersheba · Israel40 works · 3,099 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 worksSociology and Political Science · 577 works

Cite

Cite as Ofir Shabat, Ido Guy and Kira Radinsky. 2026. Propaganda Signals in LLMs: Perspectival Divergence and Narrative Framing in the Russia-Ukraine War. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.557

@inproceedings{shabat2026propaganda,
  title = {{Propaganda Signals in LLMs: Perspectival Divergence and Narrative Framing in the Russia-Ukraine War}},
  author = {Ofir Shabat, Ido Guy and Kira Radinsky},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.557},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.557}
}

References

8 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Carl I Hovland and Walter Weiss. 1951. The influence of source credibility on communication effectiveness. Public opinion quarterly, 15(4):635–650. \#\# Acknowledgments Alberto Acerbi and Joseph M Stubbersfield. 2023. Large language models show human-like content biases in transmission chain experimen… find
  2. Yejin Bang, Delong Chen, Nayeon Lee, and Pascale Fung. 2024. Measuring political bias in large lan-Tiancheng Hu, Yara Kyrychenko, Steve Rathje, Nigel Collier, Sander van der Linden, and Jon Roozenbeek. 2024. Generative language models exhibit social identity biases. Preprint, arXiv:2310.15819. — [arXiv:2310.15819](https://arxiv.org/abs/2310.15819) find
  3. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. find
  4. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment (2023). arXiv preprint arXiv:2303.16634, 12.
  5. Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. 2024. More human than human: measuring chatgpt political bias. Public Choice, 198(1):3–23. find
  6. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. Preprint, arXiv:2009.09025.
  7. Ayako Sato, Kyotaro Nakajima, Hwichan Kim, Zhousi Chen, and Mamoru Komachi. 2024. Tmu-hit’s submission for the wmt24 quality estimation shared task: Is gpt-4 a good evaluator for machine translation? In Proceedings of the Ninth Conference on Machine Translation, pages 529–534. find
  8. Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering. Proceedings of the ACM on Software Engineering, 2(ISSTA):1955– 1977. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shabat-2026-propaganda-signals-llms-perspectival-divergence/