Elamur Research⌕
☰
home / papers / 2025 / Hope Speech Detection in Social Media English Corpora: Performance of Traditional and Transformer Models

Hope Speech Detection in Social Media English Corpora: Performance of Traditional and Transformer Models

Paper · 2025 · Luis Ramos, Hiram Calvo and Olga Kolesnikova

Summary

The paper tackles binary hope speech detection in English social media posts. Here hope means goal-directed, agentic expression, which is more than general optimism or positive mood. The authors used the English data of the PolyHope-M shared task at RANLP 2025, where the organizers had already split the posts into train, development and test portions. They compared two families of models:

  • Classic scikit-learn classifiers (Logistic Regression, Naïve Bayes, and SVM with linear and RBF kernels), trained on cleaned, lemmatized text represented by TF-IDF or CountVectorizer word n-grams.
  • Fine-tuned Hugging Face encoders that had originally been trained for other purposes: hate speech detection, language identification and sentiment.

On the test set, the RoBERTa checkpoint from Dynabench hate-speech work scored best, but only by a small margin. A linear SVM and XLM-RoBERTa were close behind at about 0.79 on every metric. The transformer's advantage appears mainly in precision. The study is useful because it shows that cheap, well-tuned lexical baselines remain competitive on a small, balanced hope dataset.

Key points

  • Class counts per split:

    SplitHopeNot Hope
    Train22962245
    Development834816
    Test2065 texts in total

    The classes are essentially balanced.

  • Preprocessing lowercased the text, lemmatized it with spaCy and removed stop words with NLTK. It also dropped emojis, URLs, digits, special characters and placeholder tokens such as #USER#. The best vectorizer setting was word n-grams spanning lengths 1 to 8.

  • Macro-F1 on the development set: linear SVM 0.78, Logistic Regression (CountVectorizer) 0.78, RBF SVM 0.77, Naïve Bayes 0.75. All classifiers used default parameters.

  • Three transformer checkpoints were tried on the development set:

    • distilbert-base-uncased-finetuned-sst-2-english
    • papluca/xlm-roberta-base-language-detection
    • facebook/roberta-hate-speech-dynabench-r4-target

    Each was fine-tuned for 5 epochs with batch size 16, and the best checkpoint was kept according to F1.

  • On the test set, roberta-hate-speech-dynabench-r4-target reached weighted precision 0.82, weighted recall 0.80, weighted and macro F1 0.79, and accuracy 0.80. The SVM and XLM-RoBERTa each scored about 0.79.

Interesting ideas

  • I1. Instead of starting from a plain base model, the authors fine-tuned checkpoints already specialised for nearby tasks such as hate speech or language ID, and one of these won; inherited task knowledge can transfer to a related affective label.
  • I2. The paper adopts Snyder's definition of hope, which requires both a pathway to a goal and the agency to pursue it, instead of treating hope as a sentiment polarity; this makes the label structural rather than lexical.
  • I3. Two systems can tie on F1 yet differ in precision and recall, so reporting both macro and weighted versions of each exposes trade-offs that a single score would hide.
  • I4. Default-parameter linear models over word n-grams nearly matched fine-tuned transformers, which makes them a cheap reference point for judging whether a heavier model really adds value.
  • I5. The dataset contains a post that literally uses "hoping" for another pandemic and is labelled Hope, which shows how surface cues and actual intent can diverge.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): When training gparse on a new UD dialect or domain, try initialising from encoders already tuned on an adjacent task, and compare them against a generic base as a cheap ablation.
  • I2 (frames): Hope could become a frame recognised in prag by an experiencer, a desired future state and a plan or agent, which would separate it from mere positive tone in an explainable way.
  • I3 (learn): Evaluation reports for stance or speech-act classifiers should show the precision/recall split per class, so that a change that only raises precision is not mistaken for a broad gain.
  • I4 (learn): Keep a shallow n-gram classifier as a negative-control baseline, so that any gain claimed by the structured tree→frames pipeline can be measured against it.
  • I5 (frames): Such posts can serve as test items for the irony and implicature module, which should check whether the hoped-for state is plausibly desired before tagging the post as hope.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Instituto Politécnico NacionalMexico City · Mexico18 works · 4,307 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Luis Ramos, Hiram Calvo and Olga Kolesnikova. 2025. Hope Speech Detection in Social Media English Corpora: Performance of Traditional and Transformer Models. arXiv:2510.23585.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2510.23585 · arXiv 2510.23585

@misc{ramos2025hope,
  title = {{Hope Speech Detection in Social Media English Corpora: Performance of Traditional and Transformer Models}},
  author = {Luis Ramos, Hiram Calvo and Olga Kolesnikova},
  year = {2025},
  doi = {10.48550/arxiv.2510.23585},
  eprint = {2510.23585},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.23585}
}

References

24 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zahra Ahani, Moein Shahiki Tash, Fazlourrahman Balouchzahi, Luis Ramos, Grigori Sidorov, and Alexander Gelbukh. 2024a. Social support detection from social media texts. arXiv preprint arXiv:2411.02580. ↗
  2. Zahra Ahani, Moein Shahiki Tash, Majid Tash, Alexander Gelbukh, and Irna Gelbukh. 2024b. Multiclass hope speech detection through transformer methods. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2024), co-located with the 40th Conference of the Spanish Society for Natural Language Processing (SEPLN 2024), CEUR-WS. org. find
  3. Muhammad Arif, Moein Shahiki Tash, Ainaz Jamshidi, Iqra Ameer, Fida Ullah, Jugal Kalita, Alexander Gelbukh, and Fazlourrahman Balouchzahi. 2024. Exploring multidimensional aspects of hope speech computationally: A psycholinguistic and emotional perspective. Preprint. ↗
  4. Fazlourrahman Balouchzahi, Sabur Butt, Maaz Amjad, Luis Jose Gonzalez-Gomez, Abdul Gafar Manuel Meque, Helena Gomez-Adorno, Bharathi Raja Chakravarthi, Grigori Sidorov, Thomas Mandl, Ruba Priyadharshini, Hector Ceballos, and Saranya Rajiakodi. 2025a. Overview of polyhope-m at ranlp: Bridging hope speech detection across multiple languages. Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing. find
  5. Fazlourrahman Balouchzahi, Sabur Butt, Maaz Amjad, Grigori Sidorov, and Alexander Gelbukh. 2025b. Urduhope: Analysis of hope and hopelessness in urdu texts. Knowledge-Based Systems, 308:112746. ↗
  6. Fazlourrahman Balouchzahi, Sabur Butt, Grigori Sidorov, and Alexander Gelbukh. 2022. Cic@ lt-edi-acl2022: Are transformers the only hope? hope speech detection for spanish and english comments. In Proceedings of the second workshop on language technology for equality, diversity and inclusion, pages 206–211. ↗
  7. Fazlourrahman Balouchzahi, Grigori Sidorov, and Alexander Gelbukh. 2023. Polyhope: Two-level hope speech detection from tweets. Expert Systems with Applications, 225:120078. ↗
  8. Bharathi Raja Chakravarthi. 2020. HopeEDI: A multilingual hope speech detection dataset for equality, diversity, and inclusion. In Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media, pages 41–53, Barcelona, Spain (Online). Association for Computational Linguistics. Acc SVM-Linear 0.79 0.78 0.79 0.79 0.79 0.79 0.79 papluca/xlm-roberta-base-language-detection 0.79 0.79 0.79 0.79 0.79 0.79 0.79 facebook/roberta-hate-speech-dynabench-r4-target 0.82 0.80 0.79 0.82 0.80 0.79 0.80 *Table 5: Best model performances on test set* Anvi Alex Eponon, Moein Shahiki-Tash, Ildar Batyrshin, Christian E Maldonado-Sifuentes, Grigori Sidorov, and Alexander Gelbukh. 2025. Knowledge compression via question generation: Enhancing multihop document retrieval without fine-tuning. arXiv preprint arXiv:2506.13778. ↗
  9. Shahriyar Zaman Ridoy, Jannat Sultana, Zinnat Fowzia Ria, Mohammed Arif Uddin, Md Hasibur Rahman, and Rashedur M Rahman. 2024. An efficient text cleaning pipeline for clinical text for transformer encoder models. In 2024 IEEE 12th International Conference on Intelligent Systems (IS), pages 1–9. IEEE. find
  10. Daniel Garcı́a-Baena, Miguel Ángel Garcı́a-Cumbreras, Salud Marı́a Jiménez-Zafra, José Antonio Garcı́a-Dı́az, and Rafael Valencia-Garcı́a. 2023. Hope speech detection in spanish: The lgbt case. Language Resources and Evaluation, 57(4):1487–1514. ↗
  11. Ni Wayan Sumartini Saraswati, Christina Purnama Yanti, I Dewa Made Krishna Muku, and Dewa Ayu Putu Rasmika Dewi. 2025. Evaluation analysis of the necessity of stemming and lemmatization in text classification. MATRIK: Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer, 24(2):321–332. ↗
  12. Olga Kolesnikova, Moein Shahiki Tash, Zahra Ahani, Ameeta Agrawal, Raúl Monroy, and Grigori Sidorov. 2025. Advanced machine learning techniques for social support detection on social media. Heliyon, 11(10). ↗
  13. Mikhail Krasitskii, Olga Kolesnikova, Liliana Chanona Hernandez, Grigori Sidorov, and Alexander Gelbukh. 2024. Unveiling hope in social media: A multilingual approach using bert. ↗
  14. Denanir F Nasiri and Indra Budi. 2019. Aspect category detection on indonesian e-commerce mobile application review. In 2019 international conference on data and software engineering (icodse), pages 1–6. IEEE. find
  15. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  16. Amna Qasim, Gull Mehak, Nisar Hussain, Alexander Gelbukh, and Grigori Sidorov. 2025. Detection of depression severity in social media text using transformer-based models. Information, 16(2):114. ↗
  17. Luis Ramos, Carolina Palma-Preciado, Olga Kolesnikova, Magdalena Saldana-Perez, Grigori Sidorov, and Moein Shahiki-Tash. 2024. Intellileksika at homo-mex 2024: Detection of homophobic content in spanish lyrics with machine learning. In Journal Name or Conference Proceedings Here. find
  18. Luis Ramos, Moein Shahiki-Tash, Zahra Ahani, Alex Eponon, Olga Kolesnikova, and Hiram Calvo. 2025. Harnessing uncleaned data for stress detection in tamil and telugu code-mixed texts. Computación y Sistemas, 29(3). ↗
  19. Moein Shahiki Tash, Zahra Ahani, Mohim Tash, Olga Kolesnikova, and Grigori Sidorov. 2024. Analyzing emotional trends from x platform using senticnet: A comparative analysis with cryptocurrency price. Cognitive Computation, 16(6):3168–3185. ↗
  20. Grigori Sidorov, Fazlourrahman Balouchzahi, Luis Ramos, Helena Gómez-Adorno, and Alexander Gelbukh. 2024. Mind-hope: Multilingual identification of nuanced dimensions of hope. ↗
  21. C Richard Snyder. 2002. Hope theory: Rainbows in the mind. Psychological inquiry, 13(4):249–275. find
  22. Moein Shahiki Tash, Olga Kolesnikova, Zahra Ahani, and Grigori Sidorov. 2024. Psycholinguistic and emotion analysis of cryptocurrency discourse on x platform. Scientific Reports, 14(1):8585. ↗
  23. Moein Shahiki Tash, Luis Ramos, Zahra Ahani, Raul Monroy, Hiram Calvo, Grigori Sidorov, et al. 2025. Online social support detection in spanish social media texts. arXiv preprint arXiv:2502.09640. ↗
  24. Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2020. Learning from the worst: Dynamically generated datasets to improve online hate detection. arXiv preprint arXiv:2012.15761. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ramos-2025-hope-speech-detection-social-media/