Elamur Research⌕
☰
home / papers / 2025 / Amado at SemEval-2025 Task 11: Multi-label Emotion Detection in Amharic and English Data

Amado at SemEval-2025 Task 11: Multi-label Emotion Detection in Amharic and English Data

Paper · 2025 · Girma Yohannis Bade, Olga Kolesnikova, José Luis Oropeza et al.

Summary

The paper describes team Amado's system for Track A of SemEval-2025 Task 11. In this track a text may carry several emotions at once, so each text receives a set of labels rather than a single class. The team worked on Amharic and English. They used the organizers' data from the BRIGHTER and EthioEmo collections. English is annotated for anger, fear, joy, sadness and surprise, and Amharic also includes disgust.

The pipeline is deliberately simple:

  • Preprocessing: lowercasing, stripping tags, punctuation and digits, then tokenizing and padding to 128 tokens.
  • Model: a BERT-base model fine-tuned with Hugging Face's Trainer, using early stopping, an adaptive learning rate and at most five epochs.

On the hidden test sets, the system's macro F1 came to 0.6300 on Amharic and 0.7025 on English. Both scores sit within about one point of the official RemBERT baseline.

The work matters mainly as a reference point. A standard fine-tuned encoder roughly matches the organizers' baseline even for a low-resource language like Amharic. The authors also note that multi-label emotion recognition remains hard for English too.

Key points

  • Data size: English has 2,768 training, 116 development and 2,767 test rows (5,651 total). Amharic has 3,549, 592 and 1,774 rows (5,915 total).

  • Labels per text: English training rows carry 4,334 label assignments in total, so many texts have more than one emotion.

  • Test scores:

    LanguageAccuracyMacro recallMacro precisionMacro F1
    Amharic0.46960.67810.59100.6300
    English0.44050.72720.68220.7025

  • Baseline comparison: RemBERT scores 0.6383 (Amharic) and 0.7083 (English). BERT-base trails by 0.0083 and 0.0058 respectively.

  • Error sources: the authors cite missing context about the writer's real feeling and overlap between classes. Their examples are anger co-occurring with disgust and joy resembling surprise.

Interesting ideas

  • I1. Treating emotion as a set of independent yes/no decisions per text, instead of picking one winner, reflects that mixed feelings are normal in real writing.
  • I2. Reporting both the number of rows and the number of label assignments per class exposes how often labels co-occur.
  • I3. Placing one's score next to the official baseline with an explicit difference column makes it easy to see whether a system adds anything at all.
  • I4. The gap between low accuracy (about 0.44–0.47) and much higher macro F1 shows how much the choice of metric shapes the story for multi-label outputs.
  • I5. The error analysis names specific confusable label pairs (anger/disgust, joy/surprise) instead of just citing overall difficulty.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): speech-act and irony detection in prag could output a set of compatible labels with separate support for each, rather than forcing one interpretation per utterance.
  • I2 (learn): silver datasets could log label cardinality per frame or role type, revealing when annotators or witness parsers routinely assign multiple readings.
  • I3 (learn): every evaluation report could include a fixed baseline row and a delta column, alongside the negative controls, so small gains are not mistaken for real progress.
  • I4 (answer): school-test and QA evaluations with several acceptable answers should report both strict all-correct scores and per-item scores, to separate near misses from failures.
  • I5 (frames): keeping an explicit list of confusable frame or emotion pairs would let prag explain why a reading was ambiguous and which contextual cue would resolve it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,379 worksSentiment & emotion · 1,035 worksSemitic languages · 3,066 works

Cite

Cite as Girma Yohannis Bade, Olga Kolesnikova, José Luis Oropeza, Grigori Sidorov and Mesay Gemeda Yigezu. 2025. Amado at SemEval-2025 Task 11: Multi-label Emotion Detection in Amharic and English Data. Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bade2025amado,
  title = {{Amado at SemEval-2025 Task 11: Multi-label Emotion Detection in Amharic and English Data}},
  author = {Girma Yohannis Bade, Olga Kolesnikova, José Luis Oropeza, Grigori Sidorov and Mesay Gemeda Yigezu},
  booktitle = {Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025)},
  year = {2025},
  url = {https://aclanthology.org/2025.semeval-1.185/}
}

References

16 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iqra Ameer, Necva Bölücü, Muhammad Hammad Fahim Siddiqui, Burcu Can, Grigori Sidorov, and Alexander Gelbukh. 2023. Multi-label emotion classification in texts using transfer learning. Expert Systems with Applications, 213:118534. find
  2. Girma Bade, Olga Kolesnikova, Grigori Sidorov, and José Oropeza. 2024a. Social media hate and offensive speech detection using machine learning method. In Proceedings of the Fourth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages, pages 240–244. find
  3. Girma Yohannis Bade, O Koleniskova, José Luis Oropeza, Grigori Sidorov, and Kidist Feleke Bergene. 2024b. Hope speech in social media texts using transformer. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2024), co-located with the 40th Conference of the Spanish Society for Natural Language Processing (SEPLN 2024), CEURWS. org. find
  4. Girma Yohannis Bade, Olga Kolesnikova, and Jose Luis Oropeza. 2024c. Evaluating the quality of data: Case of sarcasm dataset. find
  5. Girma Yohannis Bade, Olga Kolesnikova, José Luis Oropeza, and Grigori Sidorov. 2024d. Lexicon-based language relatedness analysis. Procedia Computer Science, 244:268–277. find
  6. Girma Yohannis Bade and Hussien Seid. 2018. Development of longest-match based stemmer for texts of wolaita language. International Journal on Data Science and Technology, 4(3):79–83. find
  7. Girma Yohannis Bade, Muhammad Tayyab Zamir, Olga Kolesnikova, José Luis Oropeza, Grigori Sidorov, and Alexander Gelbukh. 2025. Girma@ dravidianlangtech 2025: Detecting ai generated product reviews. In Proceedings of the Fifth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages, pages 133–138. find
  8. Tadesse Destaw Belay, Israel Abebe Azime, Abinew Ali Ayele, Grigori Sidorov, Dietrich Klakow, Philip Slusallek, Olga Kolesnikova, and Seid Muhie Yimam. 2025. Evaluating the capabilities of large language models for multi-label emotion understanding. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3523–3540, Abu Dhabi, UAE. Association for Computational Linguistics.
  9. Ilyes Bendjoudi, Frederic Vanderhaegen, Denis Hamad, and Fadi Dornaika. 2021. Multi-label, multi-task cnn approach for context-based emotion recognition. Information Fusion, 76:422–428. find
  10. Jiawen Deng and Fuji Ren. 2020. Multi-label emotion detection via emotion-specified feature extraction and emotion correlation learning. IEEE Transactions on Affective Computing, 14(1):475–486. find
  11. Melkamu Abay Mersha, Girma Yohannis Bade, Jugal Kalita, Olga Kolesnikova, Alexander Gelbukh, and 1 others. 2024. Ethio-fake: Cutting-edge approaches to combat fake news in under-resourced languages using explainable ai. Procedia Computer Science, 244:133–142. find
  12. Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine de Kock, Nirmal Surange, Daniela Teodorescu, Ibrahim Said Ahmad, and 1 others. 2025a. Brighter: Bridging the gap in human-annotated textual emotion recognition datasets for 28 languages. arXiv preprint arXiv:2502.11926. ↗
  13. Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Seid Muhie Yimam, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine De Kock, Tadesse Destaw Belay, Ibrahim Said Ahmad, Nirmal Surange, Daniela Teodorescu, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino Ali, Vladimir Araujo, Abinew Ali Ayele, Oana Ignat, Alexander Panchenko, and 2 others. 2025b. SemEval task 11: Bridging the gap in text-based emotion detection. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), Vienna, Austria. Association for Computational Linguistics. find
  14. Prabod Rathnayaka, Supun Abeysinghe, Chamod Samarajeewa, Isura Manchanayake, Malaka J Walpola, Rashmika Nawaratne, Tharindu Bandaragoda, and Damminda Alahakoon. 2019. Gated recurrent neural network approach for multilabel emotion detection in microblogs. arXiv preprint arXiv:1907.07653. ↗
  15. Jesus Serrano-Guerrero, Mohammad Bani-Doumi, Francisco P Romero, and Jose A Olivas. 2022. Understanding what patients think about hospitals: A deep learning approach for detecting emotions in patient opinions. Artificial Intelligence in Medicine, 128:102298. find
  16. Dong Zhang, Xincheng Ju, Junhui Li, Shoushan Li, Qiaoming Zhu, and Guodong Zhou. 2020. Multimodal multi-label emotion detection with modality and label dependence. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 3584–3593. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bade-2025-amado-semeval-2025-task-11/