Elamur Research⌕
☰
home / papers / 2025 / QiMP at SemEval-2025 Task 11: Optimizing Text-based Emotion Classification in English Beyond Traditional Methods

QiMP at SemEval-2025 Task 11: Optimizing Text-based Emotion Classification in English Beyond Traditional Methods

Paper · 2025 · Mariia Bogatyreva, Pascal Gaertner, Quim Ribas et al.

Summary

The paper describes a system for SemEval-2025 Task 11, Track A: multi-label detection of perceived emotions in short English snippets. The five labels are anger, fear, joy, sadness and surprise. The data is the English part of BRIGHTER, which comes mostly from Reddit and was labelled by crowdworkers, and it is strongly imbalanced. The authors compare about 30 approaches:

  • classic classifiers on tf-idf, Word2Vec or GloVe features;
  • lexicon methods (NRC Emotion Lexicon, NLTK sentiment);
  • BERT-tokenizer features fed to traditional classifiers;
  • fine-tuned transformers;
  • zero-shot OpenAI o1 and DeepSeek V3.

They also tried augmentation and reweighted losses to address the imbalance. The final system is an ensemble of DeBERTa-large models that differ only in random seed, tuned with Optuna and validated with StratifiedKFold. It reaches F1 0.7537 and ranks 27th of 96 teams. The work matters mainly as a broad benchmark of many methods on one dataset, and as evidence that, on this small subjective task, fine-tuning and ensembling beat both lexicons and zero-shot LLMs.

Key points

  • The final model was trained on train plus dev, 2,884 sentences in total, with no external data.
  • Five DeBERTa-large runs with seeds 42, 99, 123, 1337 and 2024 were combined. Ensembling gave the largest single gain of all the techniques tried.
  • Per-class behaviour was uneven:
    • Anger, the rarest class, had the lowest recall (61.04%).
    • Fear, the most frequent class, had high recall (91.20%) but the worst specificity (64.77%).
    • Exact-set (subset) accuracy was 47.04%.
  • Several attempts brought little or no benefit, and sometimes hurt:
    • synonym replacement and back-translation via French with OPUS-MT;
    • class-weighted BCE and Focal Loss;
    • stop-word removal and lemmatization.
  • Zero-shot o1 and DeepSeek V3 scored below a simple BERT-tokenizer plus logistic regression pipeline.

Interesting ideas

  • I1. A homogeneous ensemble, built from one architecture with several seeds, exploits variance from initialization alone and still beats single models
  • I2. Report recall and specificity per class, not only a global F1, so you can see which rare classes are missed and which frequent ones are over-predicted
  • I3. Do cheap iteration on a small proxy model (BERT-base) and carry only the winning changes to the expensive model
  • I4. Naive augmentation (synonyms, back-translation) tended to cause meaning drift and added noise instead of signal
  • I5. Word-to-emotion lexicons failed because context decides the emotion, not single words

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could be trained with several seeds and vote on heads and labels, and disagreement between seeds could flag uncertain arcs for explanation or repair.
  • I2 (learn): evaluation of dependency labels, frames or VerbNet roles could show per-label recall and specificity, exposing overused frequent labels as well as neglected rare ones.
  • I3 (learn): training experiments could first be screened on the perceptron parser or a small gparse config before committing GPU time to the full transformer parser.
  • I4 (learn): silver data from paraphrase or translation should go through the agreement of witness parsers and checks of meaning preservation before it is used, rather than being added blindly.
  • I5 (frames): affect or irony detection in prag should derive emotion from who did what to whom in the frame and the event outcome, not from word lists, which fits Mova's no-bag-of-words principle.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,379 worksSentiment & emotion · 1,035 works

Cite

Cite as Mariia Bogatyreva, Pascal Gaertner, Quim Ribas, Daryna Dementieva and Alexander Fraser. 2025. QiMP at SemEval-2025 Task 11: Optimizing Text-based Emotion Classification in English Beyond Traditional Methods. Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bogatyreva2025qimp,
  title = {{QiMP at SemEval-2025 Task 11: Optimizing Text-based Emotion Classification in English Beyond Traditional Methods}},
  author = {Mariia Bogatyreva, Pascal Gaertner, Quim Ribas, Daryna Dementieva and Alexander Fraser},
  booktitle = {Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025)},
  year = {2025},
  url = {https://aclanthology.org/2025.semeval-1.298/}
}

References

34 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. Preprint, arXiv:1907.10902. ↗
  2. Sabur Butt, Shakshi Sharma, Rajesh Sharma, Grigori Sidorov, and Alexander Gelbukh. 2021. What goes on inside rumour and non-rumour tweets and their reactions: A Psycholinguistic Analyses. arXiv preprint. ArXiv:2112.03003 [cs]. ↗
  3. Jose Camacho-Collados, Kiamehr Rezaee, Talayeh Riahi, Asahi Ushio, Daniel Loureiro, Dimosthenis Antypas, Joanne Boisson, Luis Espinosa-Anke, Fangyu Liu, Eugenio Martínez-Cámara, et al. 2022. TweetNLP: Cutting-Edge Natural Language Processing for Social Media. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Abu Dhabi, U.A.E. Association for Computational Linguistics. find
  4. Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. 2019. Learning imbalanced datasets with label-distribution-aware margin loss. Preprint, arXiv:1906.07413. ↗
  5. Francisco Charte, Antonio J. Rivera, María J. del Jesus, and Francisco Herrera. 2015. Mlsmote: Approaching imbalanced multilabel learning through synthetic instance generation. Knowledge-Based Systems, 89:385–397. find
  6. Zebang Cheng, Fuqiang Niu, Yuxiang Lin, Zhi-qi Cheng, Xiaojiang Peng, and Bowen Zhang. MIPS at SemEval-2024 task 3: Multimodal emotion-cause pair extraction in conversations with multimodal language models. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 667–674. Association for Computational Linguistics. find
  7. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, et al. 2025. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. ↗
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  9. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. Preprint, arXiv:1808.09381. ↗
  10. Santiago Garcia, Elizabeth Martinez, Juan Cuadrado, Juan Martinez-santos, and Edwin Puertas. 2024. VerbaNexAI lab at SemEval-2024 task 10: Emotion recognition and reasoning in mixed-coded conversations based on an NRC VAD approach. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1332–1338. Association for Computational Linguistics. find
  11. Eduardo C. Garrido-Merchan, Roberto Gozalo-Brizuela, and Santiago Gonzalez-Carvajal. 2023. Comparing bert against traditional machine learning models in text classification. Journal of Computational and Cognitive Engineering, 2(4):352–356. find
  12. R. Geethanjali and A. Valarmathi. 2024. A novel hybrid deep learning IChOA-CNN-LSTM model for modality-enriched and multilingual emotion recognition in social media. 14(1):22270. find
  13. Petr Hajek and Michal Munk. 2023. Speech emotion recognition and text sentiment analysis for financial distress prediction. 35(29):21463–21477. find
  14. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
  15. OpenAI: Aaron Jaech, Adam Kalai, et al. 2024. Openai o1 system card. Preprint, arXiv:2412.16720. ↗
  16. Ismail Hakki Karaman, Gulser Koksal, Levent Eriskin, and Salih Salihoglu. 2024. A similarity-based oversampling method for multi-label imbalanced text data. Preprint, arXiv:2411.01013. ↗
  17. Roman Kazakov, Kseniia Petukhova, and Ekaterina Kochmar. PetKaz at SemEval-2024 task 3: Advancing emotion classification with an LLM for emotion-cause pair extraction in conversations. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1127–1134. Association for Computational Linguistics. find
  18. Taewoon Kim and Piek Vossen. 2021. Emoberta: Speaker-aware emotion recognition in conversation with roberta. Preprint, arXiv:2108.12009. ↗
  19. Shivani Kumar, Md. Shad Akhtar, Erik Cambria, and Tanmoy Chakraborty. SemEval 2024 - task 10: Emotion discovery and reasoning its flip in conversation (EDiReF). In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1933–1946. Association for Computational Linguistics. Sheetal Kusal, Shruti Patil, Jyoti Choudrie, Ketan Kotecha, Deepali Vora, and Ilias Pappas. A systematic review of applications of natural language processing and future challenges with special emphasis in text-based emotion detection. 56(12):15129– 15215. find
  20. Shanglin Lei, Guanting Dong, Xiaoping Wang, Keheng Wang, Runqi Qiao, and Sirui Wang. 2024. Instructerc: Reforming emotion recognition in conversation with multi-task retrieval-augmented large language models. Preprint, arXiv:2309.11911. Xuan Liu, Tianyi Shi, Guohui Zhou, Mingzhe Liu, Zhengtong Yin, Lirong Yin, and Wenfeng Zheng. Emotion classification for short texts: an improved multi-label method. 10(1):306. ↗
  21. Ilya Loshchilov and Frank Hutter. 2016. Online batch selection for faster training of neural networks. Preprint, arXiv:1511.06343. ↗
  22. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. Preprint, arXiv:1310.4546. ↗
  23. Saif M. Mohammad and Peter D. Turney. 2013. Nrc emotion lexicon. Technical report. 234 p. find
  24. Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine de Kock, Nirmal Surange, Daniela Teodorescu, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino D. M. A. Ali, Ilseyar Alimova, Vladimir Araujo, Nikolay Babakov, Naomi Baes, Ana-Maria Bucur, Andiswa Bukula, Guanqun Cao, Rodrigo Tufino Cardenas, Rendi Chevi, Chiamaka Ijeoma Chukwuneke, Alexandra Ciobotaru, Daryna Dementieva, Murja Sani Gadanya, Robert Geislinger, Bela Gipp, Oumaima Hourrane, Oana Ignat, Falalu Ibrahim Lawan, Rooweither Mabuya, Rahmad Mahendra, Vukosi Marivate, Andrew Piper, Alexander Panchenko, Charles Henrique Porto Ferreira, Vitaly Protasov, Samuel Rutunda, Manish Shrivastava, Aura Cristina Udrea, Lilian Diana Awuor Wanzare, Sophie Wu, Florian Valentin Wunderlich, Hanif Muhammad Zhafran, Tianhui Zhang, Yi Zhou, and Saif M. Mohammad. 2025a. Brighter: Bridging the gap in human-annotated textual emotion recognition datasets for 28 languages. Preprint, arXiv:2502.11926. ↗
  25. Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Seid Muhie Yimam, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine De Kock, Tadesse Destaw Belay, Ibrahim Said Ahmad, Nirmal Surange, Daniela Teodorescu, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino Ali, Vladimir Araujo, Abinew Ali Ayele, Oana Ignat, Alexander Panchenko, Yi Zhou, and Saif M. Mohammad. 2025b. SemEval task 11: Bridging the gap in text-based emotion detection. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), Vienna, Austria. Association for Computational Linguistics. find
  26. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
  27. David Rozado, Ruth Hughes, and Jamin Halberstadt. 2022. Longitudinal analysis of sentiment and emotion in news media headlines using automated labelling with transformer language models. 17(10):e0276367. find
  28. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108. ↗
  29. Zuhair Hasan Shaik, Dhivya Prasanna, Enduri Jahnavi, Rishi Thippireddy, Vamsi Madhav, Sunil Saumya, and Shankar Biradar. FeedForward at SemEval-2024 task 10: Trigger and sentext-height enriched emotion analysis in multi-party conversations. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 745–756. Association for Computational Linguistics. find
  30. Jörg Tiedemann, Mikko Aulamo, Daria Bakshandaeva, Michele Boggia, Stig-Arne Grönroos, Tommi Nieminen, Alessandro Raganato Yves Scherrer, Raul Vazquez, and Sami Virpioja. 2023. Democratizing neural machine translation with OPUS-MT. Language Resources and Evaluation, (58):713–755. find
  31. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT — Building open translation services for the World. In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT), Lisbon, Portugal. find
  32. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention is all you need. Preprint, arXiv:1706.03762. ↗
  33. Liping Zhao, Waad Alhoshan, Alessio Ferrari, and Keletso J. Letsholo. 2022. Classification of natural language processing techniques for requirements engineering. Preprint, arXiv:2204.04282. ↗
  34. Ao Zhou, Bin Liu, Jin Wang, and Grigorios Tsoumakas. 2024. Multi-label adaptive batch selection by highlighting hard and imbalanced samples. Preprint, arXiv:2403.18192. *Figure 2: Emotion Distribution in training and development data* *Figure 3: Emotion Distribution* *Figure 4: Label Sentence Distribution* *Figure 5: Matthews Correlation Coefficient (MCC) per Emotion* *Figure 6: ROC-AUC Curves per Emotion* *Figure 7: Precision and Recall per Emotion* *Figure 8: Confusion Matrix for Anger* *Figure 11: Confusion Matrix for Sadness* *Figure 9: Confusion Matrix for Fear* *Figure 12: Confusion Matrix for Surprise* *Figure 10: Confusion Matrix for Joy* ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bogatyreva-2025-qimp-semeval-2025-task-11/