Elamur Research⌕
☰
home / papers / 2023 / Automated multilingual detection of Pro-Kremlin propaganda in newspapers and Telegram posts

Automated multilingual detection of Pro-Kremlin propaganda in newspapers and Telegram posts

Paper · 2023 · Veronika Solopova, Oana-Iuliana Popescu, Christoph Benzmuller et al.

Summary

Since Russia's full-scale invasion of Ukraine, pro-Kremlin narratives have spread in volumes that no team of human fact-checkers can review. The authors test whether such propaganda can be flagged automatically and transparently across languages. They scraped 18,229 newspaper articles in Ukrainian, Russian, Romanian, English and French (23 February–4 April 2022), as well as Russian- and Ukrainian-language Telegram posts. Every text was labelled by its outlet's stance rather than by manual annotation. Two approaches are compared. The first uses classical classifiers (decision tree, linear regression, SVM, neural network) on 41 handcrafted stylistic, syntactic and emotion features plus 116 keywords. The second trains a linear model on multilingual BERT embeddings. Experiments also probe transfer to an unseen language (French) and to the far shorter Telegram genre. BERT embeddings win by only about 0.04 F1, but they are opaque and produce more false positives. The feature-based SVM remains competitive and interpretable. The released data and code support moderation in under-resourced languages, and the paper also weighs the ethical risks of such tools.

Key points

  • Corpus: 18,229 news texts (8,872 pro-Western, 9,357 pro-Kremlin) from outlets such as Ukrainska Pravda, BBC, digi24, RIA, RT and Strana.ua. Telegram posts come from channels such as InformNapalm and Rybar. Labels are source-based, with no manual annotation.
  • Features: fake-news cues (negations, passive voice, modal verbs, etc.), NRC Word-Emotion Association Lexicon emotion counts and neutrality-related dictionaries, all normalised by text length. They are compared against multilingual BERT embeddings.
  • Experimental design: stratified 5-fold cross-validation was used throughout.
    • Experiment 1 trains on four languages and then tests on French.
    • Experiment 2 adds French and applies the model to Telegram posts, which are 10–20× shorter.
    • Experiment 3 trains on news and Telegram together.
  • Best feature-based model: SVM with RBF kernel (gamma=100, C=46). It is only ~0.04 F1 behind the BERT-based model and more stable on out-of-distribution data. Keywords helped it mainly on new data.
  • Findings and ethics: pro-Kremlin media in Ukraine and Russia share rhetorical strategies, with weaker overlap with Romanian fake-news sites. Ukrainian outlets also show signs of war propaganda, so a clean propaganda-versus-neutral split is not achievable. The authors warn the classifier could equally be misused to block pro-Western content.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 works

Cite

Cite as Veronika Solopova, Oana-Iuliana Popescu, Christoph Benzmuller and Tim Landgraf. 2023. Automated multilingual detection of Pro-Kremlin propaganda in newspapers and Telegram posts. arXiv:2301.10604.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.10604 · arXiv 2301.10604

@misc{solopova2023automated,
  title = {{Automated multilingual detection of Pro-Kremlin propaganda in newspapers and Telegram posts}},
  author = {Veronika Solopova, Oana-Iuliana Popescu, Christoph Benzmuller and Tim Landgraf},
  year = {2023},
  doi = {10.48550/arxiv.2301.10604},
  eprint = {2301.10604},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.10604}
}

References

46 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [1] Smith, B. L. propaganda. Encyclopedia Britannica 2022, 24 . find
  2. [2] propaganda. n. OED Online. Oxford University Press 2022, 11 . https://github.com/anonrep/pro-kremlin_propaganda ↗
  3. [3] Khvostunova, O. Do Russians Really “Long for War” in Ukraine? Foreign Policy Research Institute 2022, [4] Roth, A. Kremlin reverts to type in denial of alleged war crimes in Ukraine’s Bucha. find
  4. The Guardian 2022, [5] Heritage, T. Russia launches new media to lead "propaganda war" with West. find
  5. Reuters 2014, [6] Paul, K. Flood of Russian misinformation puts tech companies in the hot seat. find
  6. The Guardian 2022, [7] Steiger, M.; Bharucha, T. J.; Venkatagiri, S.; Riedl, M. J.; Lease, M. The Psychological Well-Being of Content Moderators. 2021. find
  7. [8] of Excellence, N. S. C. C. The manipulative techniques of Russian’s Information Campaign, Euro-Atlantic values and Russia’s Strategic Communication in Euro-Atlantic Space Executive Summary; 2016. find
  8. [9] Carroll, J. J. Image and imitation. The visual rhetoric of Pro-Russian Propaganda. Ideology and Politics 2017, of Russia Today (RT). Journal of Communication 2020, 70, 623–645. find
  9. [16] Beskow, D.; Carley, K. Characterization and Comparison of Russian and Chinese Disinformation Campaigns; 2020; pp 63–81. find
  10. [17] Rosulek, P. The post-truth age, the fake news industry, the Russian Federation and the Central European area. Trendy v podnikánı́ 2019, 9, 46–53. find
  11. [18] Chudinov, A.; Koshkarova, N.; Ruzhent-seva, N. Linguistic Interpretation of Russian Political Agenda Through Fake, Deepfake, Post-Truth. Journal of Siberian Federal University. Humanities & Social Sciences 2019, 1840–1853. find
  12. [19] Wilbur, D. Propaganda or Not: Examining the Claims of Extensive Russian Information Operations within the United States. 2021, 20, 146–156. find
  13. [20] Snyder, T. The road to unfreedom : Russia, Europe, America / Timothy Snyder, first edition. ed.; Tim Duggan Books New York, 2018; p 359 pages ;. find
  14. [10] Meister, S. Isolation and propaganda, the roots and instrumnts of Russia’s disinformation campaign. Transatlantic Academy 2015-16 Paper Series 2016, [21] Sly, J. Timothy Snyder. On Tyranny: Twenty Lessons from the Twentieth Century . New York: Tim Duggan Books, 2017. 126p. Paper, $7.99 (ISBN 97… find
  15. College and Research Libraries 2017, 78, 868. find
  16. [11] Fortuin, E. “Ukraine commits genocide on Russians”: the term “genocide” in Russian propaganda. Russ Linguist 2022, 313–347. find
  17. [22] Alsmadi, I.; O’Brien, M. How Many Bots in Russian Troll Tweets? Information Processing & Management 2020, 57, 102303. find
  18. [12] Pillars of Russia’s Disinformation and Propaganda Ecosystem; 2020. find
  19. [23] Asr, F. T.; Taboada, M. Big Data and quality data for fake news and misinformation detection. Big Data & Society 2019, 6, 2053951719843310. find
  20. [13] Narayanan, V. P. Russian Involvement and Junk News during Brexit COMPROP DATA MEMO 2017.10 / December 19th 2017. 2017. find
  21. [14] Cosentino, G. Polarize and conquer: Russian influence operations in the united states. In Social Media and the Post-Truth World Order. Springer 2020a, 33–57. find
  22. [15] Elswah, M.; Howard, P. N. “Anything that Causes Chaos”: The Organizational Behavior [24] Antoun, W.; Baly, F.; Achour, R.; Hussein, A.; Hajj, H. State of the Art Models for Fake News Detection Tasks. 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT). 2020… find
  23. [25] Mahyoob, M.; Algaraady, J.; Alrahaili, M. Linguistic-Based Detection of Fake News in Social Media. International Journal of English Linguistics 2020, 11, 99. find
  24. [26] Rashkin, H.; Choi, E.; Jang, J. Y.; Volkova, S.; Choi, Y. Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
  25. Copenhagen, Denmark, 2017; pp 2931–2937. find
  26. [27] Li, X.; Xia, Y.; Long, X.; Li, Z.; Li, S. Exploring Text-transformers in AAAI 2021 Shared Task: COVID-19 Fake News Detection in English. 2021, [28] Yu, S.; Martino, G. D. S.; Mohtarami, M.; Glass, J. R.; Nakov, P. Interpretable Propaganda Detection in News Articles.
  27. [29] Tundis, A.; Mukherjee, G.; Mühlhäuser, M. Mixed-Code Text Analysis for the Detection of Online Hidden Propaganda. Proceedings of the 15th International Conference on Availability, Reliability and Security. New York, NY, USA, 2020. find
  28. [30] Oliinyk, V.-A.; Vysotska, V.; Burov, Y.; My-kich, K.; Fernandes, V. B. Propaganda Detection in Text Data Based on NLP and Machine Learning. MoMLeT+DS. 2020. find
  29. [31] Dadu, T.; Pant, K.; Mamidi, R. Towards Detection of Subjective Bias using Contextualized Word Embeddings. CoRR 2020, abs/2002.06644 . — [arXiv:2002.06644](https://arxiv.org/abs/2002.06644) find
  30. [32] Devlin, J.; Chang, M.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR 2018, abs/1810.04805 .
  31. [33] UkraineWorld.org, «Страна»: популярне i проросiйське медiа в Українi. Radio Svoboda 2022, [34] Bergengruen, V. How Telegram Became the Digital Battlefield in the Russia-Ukraine War. find
  32. The Time 2022, [35] O’Brien, P. How Telegram Became the Digital Battlefield in the Russia-Ukraine War. France 24 2022, [36] Valeriya Safronova, A. S., Neil MacFarquhar Where Russians Turn for Uncensored News on Ukraine. find
  33. The New Your Times 2022, [37] Sweney, M. Telegram: the app at the heart of Ukraine’s propaganda battle. find
  34. The Guardian 2022, [38] Mohammad, S. M.; Turney, P. D. Crowdsourcing a Word-Emotion Association Lexicon. find
  35. Computational Intelligence 2013, 29, 436–465. find
  36. [39] Gessen, M. The War That Russians Do Not See. The New Yorker 2022, [40] Cohen, J. A Coefficient of Agreement for Nominal Scales. find
  37. Educational and Psychological Measurement 1960, 37–46. find
  38. [41] Powers, D. Evaluation: From Precision, Recall and F-Factor to ROC, Informedness, Markedness & Correlation. Mach. Learn. Technol. 2008, 2 . find
  39. [42] Grandini, M.; Bagli, E.; Visani, G. Metrics for Multi-Class Classification: an Overview. ArXiv 2020, abs/2008.05756 . — [arXiv:2008.05756](https://arxiv.org/abs/2008.05756) find
  40. [43] Breiman, L. Random Forests. Machine Learning 2004, 45, 5–32. find
  41. [44] Marples, D. R. Stepan Bandera: The resurrection of a Ukrainian national hero, Europe-Asia Studies, 58:4, 555-566. DOI: 10 2006, 1080/09668130600652118. find
  42. [45] Gaufman, E. Security Threats and Public Perception: Digital Russia and the Ukraine Crisis; New Security Challenges, 2016; p 107. find
  43. [46] Daniel Boffey, D. S.; Borger, J. Mariupol theatre bombing killed 300, Ukrainian officials say. The Guardian 2022, [47] Harding, L. How Ukrainian defiance has derailed Putin’s plans. find
  44. The Guardian 2022, [48] Horton, J.; Sardarizadeh, S.; Schraer, R.; Robinson, O.; Coleman, A.; Palumbo, D.; Cheetham, J. Bucha killings: Satellite image of bodies site contradicts Russian claims. find
  45. Reality Check and BBC Monitoring, BBC News 2022, [49] De Cao, N.; Schlichtkrull, M.; Aziz, W.; Titov, I. How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking. 2020; pp 3243–3255. find
  46. [50] Jawahar, G.; Sagot, B.; Seddah, D. What Does BERT Learn about the Structure of Language? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy, 2019; pp 3651–3657.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/solopova-2023-automated-multilingual-detection-pro-kremlin/