Elamur Research⌕
☰
home / papers / 2025 / Transforming Causal LLM into MLM Encoder for Detecting Social Media Manipulation in Telegram

Transforming Causal LLM into MLM Encoder for Detecting Social Media Manipulation in Telegram

Summary

The authors participated in the Fourth UNLP 2025 shared task, which involved detecting manipulation in Ukrainian- and Russian-language Telegram posts. The task had two tracks: determining which of 10 manipulation techniques are present in a post (multilabel classification), and marking at the character level exactly which text fragments are manipulative (span identification). For classification, they fine-tuned the decoder-only model Gemma2-27B, combined several models into an ensemble, and tuned a threshold separately for each class. This threshold is the mean of two values: one yields the highest F1 on validation, while the other makes the share of positive predictions equal to the true class frequency. For span identification, the authors converted the causal Gemma2 into a bidirectional encoder, biGemma2. To do so, they first further trained the model on a masked language modeling task using Ukrainian and Russian news corpora, and only then fine-tuned it for span labeling. Both solutions took first place on the leaderboards of their respective tracks. The work shows that large decoder-only models can be adapted to tasks traditionally solved with encoders, which is useful for languages with little annotated data.

Key points

  • Data: 9,500 posts annotated by media experts. Local training used 3,058 posts and validation used 764 (multi-label stratified K-Fold split). The official leaderboard test set contains 5,735 posts.
  • Span identification (Char-F1, private LB): an ensemble of biGemma2-27b, Aya-101, and mDeBERTa-v3 scored 0.642, and biGemma2-27b alone scored 0.640. For comparison: standard Gemma2-27b scored 0.611, mDeBERTa-v3 0.612, and the diffusion model LLaDA-8b 0.542.
  • Classification: the Gemma2-27b ensemble achieved a macro-F1 of about 0.49 on the private LB. GPT-4o in zero-shot mode reached only 0.32, and 0.36 with chain-of-thought. A model trained on an English translation performed noticeably worse.
  • Open resources: the model ABazdyrev/bigemma-2-27b-lora on Hugging Face, and the code in the repository AntonBazdyrev/unlp2025_shared_task. The corpora zeusfsx/ukrainian-news (200K documents) and AIR-Bench/qa_news_ru were used for MLM further training.
  • What did not work: joint training on both tasks with an auxiliary loss function and a two-stage "classifier first, then span identification" pipeline did not improve quality. Re-annotation of part of the data by three experts showed that the class definitions overlap. This is likely what limits the maximum achievable accuracy of the models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.unlp-1.13

@inproceedings{bazdyrev2025transforming,
  title = {{Transforming Causal LLM into MLM Encoder for Detecting Social Media Manipulation in Telegram}},
  author = {Bazdyrev, Anton and Bashtovyi, Ivan and Havlytskyi, Ivan and Kharytonov, Oleksandr and Khodakovskyi, Artur},
  booktitle = {Proceedings of the Fourth Ukrainian Natural Language Processing Workshop (UNLP 2025)},
  year = {2025},
  doi = {10.18653/v1/2025.unlp-1.13},
  url = {https://doi.org/10.18653/v1/2025.unlp-1.13}
}

References

29 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 oth… find our card
  2. AI@Meta. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  3. Anton Bazdyrev. 2025. Russo-ukrainian war disinformation detection in suspicious telegram channels. arXiv preprint arXiv:2503.05707. 2024. Token prepending: A training-free approach for eliciting better sentence embeddings from llms. arXiv preprint arXiv:2412.11556. find
  4. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961. find our card
  5. Gemma Team. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  6. Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, André Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, Emmanuel Malherbe, Étienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, João Alves, Kevin El-Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei… find
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. find
  8. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, and 1 others. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. find
  9. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news articles. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natura… find
  10. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac,… find our card
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  12. David Dukić and Jan Šnajder. 2024. Looking right is sometimes right: Investigating the capabilities of decoder-only llms for sequence labeling. arXiv preprint arXiv:2401.14556. find
  13. Yuchen Fu, Zifeng Cheng, Zhiwei Jiang, Zhonghui Wang, Yafeng Yin, Zhengliang Li, and Qing Gu. find
  14. Gemma Team. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  15. Daniil Gurgurov, Mareike Hartmann, and Simon Ostermann. 2024. Adapting multilingual llms to low-resource languages with knowledge graphs via adapters. arXiv preprint arXiv:2407.01406. find
  16. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. Preprint, arXiv:2111.09543. find our card
  17. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the International Conference on Machine Learning (ICML), pages 4411–4421. PMLR. find
  18. Raviraj Joshi, Kanishk Singla, Anusha Kamath, Raunak Kalani, Rakesh Paul, Utkarsh Vaidya, Sanjay Singh Chauhan, Niranjan Wartikar, and Eileen Long. 2024. Adapting multilingual llms to low-resource languages using continued pre-training and synthetic corpus. arXiv preprint arXiv:2410.14815. — [arXiv:2410.14815](https://arxiv.org/abs/2410.14815) find
  19. Roman Kyslyi, Nataliia Romanyshyn, and Volodymyr Sydorskyi. 2025. The unlp 2025 shared task on detecting social media manipulation. find
  20. ACL 2025, page to appear. find
  21. Xianming Li and Jing Li. 2023. Bellm: Backward dependency enhanced large language model for sentence embeddings. arXiv preprint arXiv:2311.05296. find
  22. Zongxi Li, Xianming Li, Yuzhang Liu, Haoran Xie, Jing Li, Fu-lee Wang, Qing Li, and Xiaoqin Zhong. 2023. Label supervised llama finetuning. arXiv preprint arXiv:2310.01208. find
  23. Zachary C. Lipton, Charles Elkan, and Balakrishnan Narayanaswamy. 2014. Thresholding classifiers to maximize f1 score. arXiv preprint arXiv:1402.1892. — [arXiv:1402.1892](https://arxiv.org/abs/1402.1892) find
  24. Mistral AI team. 2024. Large enough. https://mistral.ai/news/mistral-large-2407. Blog post; Accessed: 2025-04-17. ↗
  25. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. Large language diffusion models. arXiv preprint arXiv:2502.09992. find our card
  26. Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. 2024. Repetition improves language model embeddings. arXiv preprint arXiv:2402.15449. find
  27. Natalya Steblyna. 2022. Pro-russian propaganda detection in the most popular telegram channels of odesa region (frame analysis). Bulletin of Lviv Polytechnic National University: journalism, 1:80–88. find
  28. Paul Suganthan, Fedor Moiseev, Le Yan, Junru Wu, Jianmo Ni, Jay Han, Imed Zitouni, Enrique Alfonseca, Xuanhui Wang, and Zhe Dong. 2025. Adapting decoder-based language models for diverse encoder downstream tasks. arXiv preprint arXiv:2503.02656. — [arXiv:2503.02656](https://arxiv.org/abs/2503.02656) find
  29. Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction fine… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bazdyrev-2025-transforming-causal-llm-into-mlm/