Elamur Research⌕
☰
home / papers / 2025 / Hidden Persuasion: Detecting Manipulative Narratives on Social Media During the 2022 Russian Invasion of Ukraine

Hidden Persuasion: Detecting Manipulative Narratives on Social Media During the 2022 Russian Invasion of Ukraine

Summary

The authors present a solution for the UNLP 2025 Shared Task, which asks participants to detect manipulative rhetorical techniques in posts from Ukrainian Telegram channels. The task has two parts. The first is to identify which of ten manipulation types are present in a text (several can occur at once). The second is to locate the specific fragments that contain the manipulation. The dataset has over 9.5 thousand posts in Ukrainian and Russian, and its classes are highly imbalanced. For classification, the authors fine-tuned Gemma 2 2B IT in stages with LoRA. The model first learned to generate a list of techniques from a prompt and was then trained as a multi-label classifier. CatBoost runs on top of its probabilities, using meta-features and a separate threshold for each class. Fragments are detected by XLM-RoBERTa-Large with two jointly trained heads: one classifies techniques and the other tags tokens. The team placed second in classification and third in span detection. This shows that a compact LLM with lightweight fine-tuning can compete in multilingual propaganda detection. However, the model still recognizes rare techniques poorly.

Key points

  • Data: 3,822 posts for training (56% in Ukrainian, 44% in Russian) and 5,735 for testing; about 68% of posts contain manipulation.
  • Classification (macro F1): the CatBoost baseline scores 0.408, Gemma 0.450, and Gemma with post-processing 0.454. The MolodiAmbitni team scored 0.470 on the private leaderboard; the winner, GA, scored 0.494.
  • Span detection (span-level F1): the XLM-RoBERTa baseline scores 0.586 and the two-headed model 0.599; on the private leaderboard, 0.600 (third place).
  • Meta-features for CatBoost: distances to K-means centroids (K=10) built from trigger phrases; technique frequencies among the 10 nearest neighbors; simple text statistics. Embeddings were obtained with the mGTE model via SentenceTransformers.
  • Weaknesses: the best result is for loaded_language (F1 0.782, 2,959 examples). The whataboutism, straw_man and bandwagon classes are recognized much worse. The test set was split off without regard to post chronology, so the evaluation may be inflated. Code: github.com/akhynkokateryna/manipulative-narrative-detection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.unlp-1.19 · arXiv 2505.24028

@inproceedings{akhynko2025hidden,
  title = {{Hidden Persuasion: Detecting Manipulative Narratives on Social Media During the 2022 Russian Invasion of Ukraine}},
  author = {Akhynko, Kateryna and Kosovan, Oleksandr and Trokhymovych, Mykola},
  booktitle = {Proceedings of the Fourth Ukrainian Natural Language Processing Workshop (UNLP 2025)},
  year = {2025},
  doi = {10.18653/v1/2025.unlp-1.19},
  eprint = {2505.24028},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.unlp-1.19}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ola Altiti, Malak Abdullah, and Rasha Obiedat. 2020. JUST at SemEval-2020 task 11: Detecting propaganda techniques using BERT pre-trained model. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1749–1755, Barcelona (online). International Committee for Computational Linguistic… find
  2. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. Llm2vec: Large language models are secretly powerful text encoders. Preprint, arXiv:2404.05961. find our card
  3. Giovanni Da San Martino, Alberto Barrón-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 task 11: Detection of propaganda techniques in news articles. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1377–1414, Barcelona (online). Internationa… find
  4. Artur Kiulian, Anton Polishko, Mykola Khandoga, Oryna Chubych, Jack Connor, Raghav Ravishankar, and Adarsh Shirawalmath. 2024. From bytes to borsch: Fine-tuning gemma and mistral for the ukrainian language representation. Preprint, arXiv:2404.09138. find
  5. Rahul Mehta and Vasudeva Varma. 2023. LLM-RM at SemEval-2023 task 2: Multilingual complex NER using XLM-RoBERTa. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 453–456, Toronto, Canada. Association for Computational Linguistics. find
  6. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  7. Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir R… find
  8. Siva Sai, Alfred W. Jacob, Sakshi Kalra, and Yashvard-han Sharma. 2021. Stacked embeddings and multiple fine-tuned xlm-roberta models for enhanced hostility identification. In Combating Online Hostile Posts in Regional Languages during Emergency Situation, pages 224–235, Cham. Springer International… find
  9. Veronika Solopova, Christoph Benzmüller, and Tim Landgraf. 2023. The evolution of pro-kremlin propaganda from a machine learning and linguistics perspective. In Proceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP), pages 40–48, Dubrovnik, Croatia. Association for Computati… find
  10. Kilian Sprenkamp, Daniel Gordon Jones, and Liudmila Zavolokina. 2023. Large language models for propaganda detection. Preprint, arXiv:2310.06422. — [arXiv:2310.06422](https://arxiv.org/abs/2310.06422) find
  11. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton T… — [arXiv:2504.10663](https://arxiv.org/abs/2504.10663) find
  12. Mykola Trokhymovych and Diego Saez-Trumper. 2021. Wikicheck: An end-to-end open source automatic fact-checking api based on wikipedia. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management, CIKM ’21, page 4155–4164, New York, NY, USA. Association for Computi… find
  13. Mykola Trokhymovych, Indira Sen, and Martin Gerlach. 2024. An open multilingual system for scoring readability of Wikipedia. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6296– 6311, Bangkok, Thailand. Association for Comput… find
  14. Mykyta Vorobiov. 2024. Has ukraine become too dependent on telegram? Accessed: 12 April 2025. find
  15. Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, et al. 2024. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference on Empirical Methods in… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/akhynko-2025-hidden-persuasion-detecting-manipulative-narratives/