Elamur Research⌕
☰
home / papers / 2025 / SmolLab_SEU at MAHED Shared Task: Do Arabic-Native Encoders Surpass Multilingual Models in Detecting the Nuances of Hope, Hate, and Emotion?

SmolLab_SEU at MAHED Shared Task: Do Arabic-Native Encoders Surpass Multilingual Models in Detecting the Nuances of Hope, Hate, and Emotion?

Paper · 2025 · Md Abdur Rahman, Md Sabbir Dewan, Md. Tofael Ahmed Bhuiyan et al.

Summary

The paper asks whether encoders pre-trained only on Arabic beat large multilingual encoders at detecting hope, hate and emotion in Arabic social media text. The authors entered the two text tracks of the MAHED 2025 shared task. Subtask 1 is a three-way label (hate, hope, not_applicable). Subtask 2 is a hierarchical problem: one of 12 emotions, whether the text is offensive, and, if so, whether it is targeted hate. They fine-tuned seven Transformer encoders, four Arabic-native and three multilingual, after a shared cleaning step. For Subtask 2 they built a cascade of three separately trained classifiers instead of one joint model. Arabic-native models generally came out ahead, and ARBERTv2 was best on both subtasks, placing 11th and 5th on the leaderboards. Error analysis points to implicit wording and heavy label skew as the main obstacles, especially when separating directed hate from plain rudeness. The work gives a clear benchmark suggesting that language- and domain-matched pre-training still matters for subtle affect tasks.

Key points

  • The model suite has two groups:
    • Arabic-native: MARBERTv2, ARBERTv2, AraBERTv2 large and QARiB.
    • Multilingual: XLM-RoBERTa large, mDeBERTaV3 base and multilingual DistilBERT base.
  • Data splits (train/validation/test):
    • Subtask 1: 6,890 / 1,476 / 1,477 instances.
    • Subtask 2: 5,960 / 1,277 / 1,278 instances.
    • Both mix Modern Standard and dialectal Arabic, and macro F1 is the official metric.
  • ARBERTv2 reached macro F1 0.6824 on Subtask 1 (11th place) and 0.5142 on Subtask 2 (5th place). AraBERTv2 large (0.5802) and DistilBERT (0.6110) trailed on Subtask 1.
  • On Subtask 2, QARiB had the top accuracy (0.7415) but only 0.4915 macro F1. This gap suggests it mostly predicted the majority classes.
  • Training setup differed by subtask:
    • Subtask 1 used the HF Trainer with early stopping on validation macro F1 (patience 3).
    • Subtask 2 used a custom loop with class-weighted cross-entropy, AdamW and FP16.
    • The hate model in the cascade was trained only on examples labelled offensive.

Interesting ideas

  • I1. When one label only makes sense under another, train the child classifier solely on parent-positive examples and route inputs to it at inference, so each stage specializes without joint-loss interference.
  • I2. A large gap between accuracy and macro F1 is a cheap signal that a model has collapsed onto frequent classes, so reporting both exposes hidden failures on rare labels.
  • I3. Here a pre-training corpus matched to the target genre helped more than raw size: a large general Arabic model underperformed smaller encoders trained on social and web text.
  • I4. Early stopping and checkpoint selection were driven by the task's actual metric (macro F1) rather than by loss, which aligns model choice with what is really evaluated.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Irony or implicature typing could run only on utterances first flagged as non-literal, with the second-stage model trained just on that subset.
  • I2 (learn): Evaluation reports for taggers and relation labellers could flag any model whose accuracy is high but macro F1 lags, as a guard against rare UD relations being ignored.
  • I3 (tree): When choosing the encoder behind gparse, compare candidates by how closely their pre-training text matches fables and school-style prose, not just by parameter count.
  • I4 (learn): Parser and tagger training could select checkpoints by LAS or per-label macro F1 on held-out silver data instead of by training loss.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Md Abdur Rahman, Md Sabbir Dewan, Md. Tofael Ahmed Bhuiyan and Md Ashiqur Rahman. 2025. SmolLab_SEU at MAHED Shared Task: Do Arabic-Native Encoders Surpass Multilingual Models in Detecting the Nuances of Hope, Hate, and Emotion?. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.90

@inproceedings{rahman2025smollab,
  title = {{SmolLab\_SEU at MAHED Shared Task: Do Arabic-Native Encoders Surpass Multilingual Models in Detecting the Nuances of Hope, Hate, and Emotion?}},
  author = {Md Abdur Rahman, Md Sabbir Dewan, Md. Tofael Ahmed Bhuiyan and Md Ashiqur Rahman},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.90},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.90}
}

References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. ↗
  2. Sufian Kedir Abdella and Worku Jifara Sori. 2024. Detection of emotions in afan oromo social media texts using deep learning method. Ethiopian Journal of Science and Sustainable Development, 11(1):70–84. ↗
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  4. Badriyya B Al-Onazi, Hassan Alshamrani, Fatimah Ok-leh Aldaajeh, Amira Sayed A Aziz, and Mohammed Rizwanullah. 2023. Modified seagull optimization with deep learning for affect classification in arabic tweets. IEEE Access, 11:98958–98968. ↗
  5. Hussein Farooq Tayeb Al-Saadawi and Resul Das. 2024. Ter-ca-wgnn: trimodel emotion recognition using cumulative attribute-weighted graph neural network. Applied Sciences, 14(6):2252. ↗
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. Md. ↗
  7. Rafiul Biswas and Wajdi Zaghouani. 2025a. An annotated corpus of arabic tweets for hate speech analysis. CoRR, abs/2505.11969. Md. ↗
  8. Rafiul Biswas and Wajdi Zaghouani. 2025b. Emohopespeech: An annotated dataset of emotions and hope speech in english and arabic. CoRR, abs/2505.11959. ↗
  9. El Mehdi Cherrat, Hassan Ouahi, Abdellatif BEKKAR, and 1 others. 2024. Sentiment analysis from texts written in standard arabic and moroccan dialect based on deep learning approaches. International Journal of Computing and Digital Systems, 16(1):447–458. find
  10. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186. ↗
  12. Hanane Elfaik and 1 others. 2023. Leveraging feature-level fusion representations and attentional bidirectional rnn-cnn deep models for arabic affect analysis on twitter. Journal of King Saud University-Computer and Information Sciences, 35(1):462–482. find
  13. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. Preprint, arXiv:2111.09543. ↗
  14. D Sami Khafaga, Maheshwari Auvdaiappan, K Deepa, Mohamed Abouhawwash, and F Khalid Karim. 2023. Deep learning for depression detection using twitter data. Intelligent Automation & Soft Computing, 36(2):1301–1313. ↗
  15. Muhammad Shahid Khan, Muhammad Shahid Iqbal Malik, and Aamer Nadeem. 2024. Detection of violence incitation expressions in urdu tweets using convolutional neural network. Expert Systems with Applications, 245:123174. ↗
  16. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Wajdi Zaghouani, Hamdy Mubarak, and Md. ↗
  17. Rafiul Biswas. 2024. So hateful! building a multi-label hate speech annotated Arabic dataset. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 15044–15055, Torino, Italia. ELRA and ICCL.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rahman-2025-smollab-seu-mahed-shared-task/