Elamur Research⌕
☰
home / papers / 2025 / Evaluation of Pretrained and Instruction-Based Pretrained Models for Emotion Detection in Arabic Social Media Text

Evaluation of Pretrained and Instruction-Based Pretrained Models for Emotion Detection in Arabic Social Media Text

Paper · 2025 · Md. Rafiul Biswas, Shimaa Ibrahim, Mabrouka Bessghaier et al.

Summary

The paper asks which kind of model best detects emotions in Arabic tweets when one tweet can carry several emotions at once. The authors took a random sample of 10,000 emotion-annotated tweets from the multi-perspective Twitter corpus of Zaghouani et al. (2024), which covers 12 labels and is strongly skewed toward Disgust. After heavy Arabic-specific preprocessing, they compared three strategies on this sample:

  • fine-tuning encoder transformers (AraBERTv2, CamelBERT, XLM-RoBERTa) as multi-label classifiers;
  • prompting large models (GPT-4o, Gemini, Deepseek, Fanar) with generated instruction templates;
  • LoRA instruction fine-tuning of Llama 3.1, Mistral and Phi 4.

The encoder models won clearly, with AraBERTv2 reaching a micro F1 of 0.74. The best prompted model, GPT-4o, scored 0.42. Among the fine-tuned LLMs only Llama 3.1 was competitive, at 0.66. The authors conclude that general-purpose LLMs, whether prompted or lightly tuned, still trail smaller domain-specific encoders on nuanced multi-label classification in a lower-resource language.

Key points

  • Data: 10,000 tweets collected August–October 2020 with 12 labels: neutral plus 11 emotions such as anger, trust and pessimism.
    • Disgust appears 5,883 times and "no emotions" 1,767 times.
    • Fear (10) and Pessimism (13) are extremely rare.
    • Frequent co-occurrences include Disgust+Trust (312) and Sadness+Disgust (225).
  • Preprocessing:
    • unify Alef/Ya variants and strip tatweel, while keeping diacritics;
    • remove Arabic stopwords (Alrefaie list) and English stopwords (NLTK);
    • remove or replace URLs, usernames and hashtags with placeholders;
    • map common dialect words to MSA equivalents.
  • Encoder results: AraBERTv2 scored micro F1 0.74, precision 0.82, recall 0.68 and accuracy 0.65. CamelBERT and XLM-RoBERTa each reached 0.72 micro F1.
  • Prompting results (micro F1): GPT-4o 0.42, Gemini 0.35, Fanar 0.34, Deepseek 0.11. Prompts were built with the CreateInstructionSetForLLM tool, and GPT-4o was accessed through Azure OpenAI.
  • Instruction fine-tuning: done with Unsloth, LoRA and 4-bit quantization. Micro F1 was Llama 3.1 0.66, Mistral 0.24 and Phi 4 0.11, even though training losses were similar across models (about 0.16).

Interesting ideas

  • I1. Instead of a fixed 0.5 cut-off, sweep the sigmoid threshold for multi-label decisions and pick the range that maximizes F1 (0.4–0.5 here), since this balances precision and recall when labels co-occur.
  • I2. Run all three regimes (cheap prompting, adapter fine-tuning, a small task-tuned encoder) on the same data and split, so the cost/accuracy trade-off is measured rather than assumed.
  • I3. Comparable training losses hid huge differences in task metrics (0.11 vs 0.66 F1), a reminder that loss curves alone do not show whether a model has learned the task.
  • I4. Normalize spelling variants and map dialectal words to a standard form before modeling, which reduces sparsity in informal text.
  • I5. With a long tail of rare classes, judge models with micro and macro F1 together, so that a dominant class like Disgust does not mask failure on rare ones.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Speech-act or irony detectors that can assign several labels at once could tune per-label thresholds on held-out data instead of using a default cut-off.
  • I2 (learn): Before promoting an LLM teacher tier, benchmark it against a small trained component on the same gold set to see whether the teacher is actually better than the student.
  • I3 (learn): Training reports for gparse or taggers should always pair loss with task scores such as LAS or accuracy, plus negative controls, before a model is accepted.
  • I4 (tokens): An explicit, explainable normalization layer for informal English (spelling variants, slang to standard lemmas) could feed the tagger cleaner input while keeping the original token.
  • I5 (answer): Evaluations of readers on tests should report per-category scores alongside the overall average, so that weak, rare question types become visible.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northwestern University in QatarDoha · Qatar39 works · 148 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksSocial Psychology · 330 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksSemitic languages · 3,066 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Md. Rafiul Biswas, Shimaa Ibrahim, Mabrouka Bessghaier and Wajdi Zaghouani. 2025. Evaluation of Pretrained and Instruction-Based Pretrained Models for Emotion Detection in Arabic Social Media Text. Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI Era.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{biswas2025evaluation,
  title = {{Evaluation of Pretrained and Instruction-Based Pretrained Models for Emotion Detection in Arabic Social Media Text}},
  author = {Md. Rafiul Biswas, Shimaa Ibrahim, Mabrouka Bessghaier and Wajdi Zaghouani},
  booktitle = {Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI Era},
  year = {2025},
  doi = {10.26615/978-954-452-098-4-019},
  url = {https://doi.org/10.26615/978-954-452-098-4-019}
}

References

36 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammed M Abdelgwad, Taysir Hassan A Soliman, Ahmed I Taloba, and Mohamed Fawzy Farghaly. 2022. Arabic aspect based sentiment analysis using bidirectional gru based models. Journal of King Saud University-Computer and Information Sciences, 34(9):6652–6662. for Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12878–12901, Bangkok, Thailand. Association for Computational Linguistics.
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  4. Manal Abdullah, Muna AlMasawa, Ibtihal Makki, Maha Alsolmi, and Samar Mahrous. 2020. Emotions extraction from arabic tweets. International Journal of Computers and Applications, 42(7):661–675. find
  5. Ahmad Al-Qerem, Mohammed Raja, Sameh Taqatqa, and Mutaz Rsmi Abu Sara. 2024. Utilizing deep learning models (rnn, lstm, cnn-lstm, and bi-lstm) for arabic text classification. In Artificial Intelligence-Augmented Digital Twins: Transforming Industrial Operations for Innovation and Sustainability, pages 287–301. Springer. find
  6. Fatimah Aljwari. 2022. Emotion detection in arabic text using machine learning methods. IJISCS (International Journal of Information System and Computer Science), 6(3):175–185. find
  7. Amer J Almahdawi and William J Teahan. 2019. A new arabic dataset for emotion recognition. In Intelligent Computing: Proceedings of the 2019 Computing Conference, Volume 2, pages 200–216. Springer. find
  8. Ghadah Alqahtani and Abdulrahman Alothaim. 2022a. Emotion analysis of arabic tweets: Language models and available resources. Frontiers in Artificial Intelligence, 5:843038. find
  9. Ghadah Alqahtani and Abdulrahman Alothaim. 2022b. Emotion analysis of arabic tweets: Language models and available resources. Frontiers in Artificial Intelligence, 5. find
  10. Mohamed Taher Alrefaie. 2019. Arabic stop words. find
  11. Maha Jarallah Althobaiti. 2023a. An open-source dataset for arabic fine-grained emotion recognition of online content amid covid-19 pandemic. Data in Brief, 51:109745. find
  12. Muhammad Azeem Aslam, Wang Jun, Nisar Ahmed, Muhammad Imran Zaman, Li Yanan, Hu Hongfei, Wang Shiyu, and Xin Liu. 2024. Improving arabic multi-label emotion classification using stacked embeddings and hybrid loss function. arXiv preprint arXiv:2410.03979. ↗
  13. Santosh Kumar Bharti, S Varadhaganapathy, Rajeev Kumar Gupta, Prashant Kumar Shukla, Mohamed Bouye, Simon Karanja Hingaa, and Amena Mahmoud. 2022. Text-based emotion recognition using deep learning approach. Computational Intelligence and Neuroscience, 2022(1):2645381. find
  14. Md Rafiul Biswas. 2024. Createinstructionsetforllm: A pipeline to generate instruction-response datasets from multi-label corpora. https://github.com/rafiulbiswas/ CreateInstructionSetForLLM . Accessed: July 2025. ↗
  15. Hasna Chouikhi, Manel Aloui, Cyrine Ben Hammou, Ghaith Chaabane, Haithem Kchaou, and Chehir Dhaouadi. 2024. Gemmar: Enhancing llms through arabic instruction-tuning.
  16. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  17. Michael Han Daniel Han and Unsloth team. 2023. Unsloth. find
  18. Eman Yaser Daraghmi, Sajida Qadan, Yousef Daraghmi, Rami Yussuf, Omar Cheikhrouhou, and Mohammed Baz. 2024. From text to insight: An integrated cnn-bilstm-gru model for arabic cyberbullying detection. IEEE Access. find
  19. Xin Hong, Yuan Gong, Vidhyasaharan Sethu, and Ting Dang. 2025. Aer-llm: Ambiguity-aware emotion recognition leveraging large language models. find
  20. Maha Jarallah Althobaiti. 2023b. An open-source dataset for arabic fine-grained emotion recognition of online content amid covid-19 pandemic. Data in Brief, 51:109745. find
  21. Chenyang Huang, Amine Trabelsi, and Osmar R Zaı̈ane. 2019. Ana at semeval-2019 task 3: Contextual emotion detection in conversations through hierarchical lstms and bert. arXiv preprint arXiv:1904.00132. ↗
  22. Zaid Alyafeai, Khalid Almubarak, Ahmed Ashraf, Deema Alnuhait, Saied Alshahrani, Gubran Abdulrahman, Gamil Ahmed, Qais Gawah, Zead Saleh, Mustafa Ghaleb, Yousef Ali, and Maged Al-shaibani. 2024. CIDAR: Culturally relevant instruction dataset Doaa Mohey El-Din Mohamed Hussein. 2018. A survey on sentiment analysis challenges. Journal of King Saud University - Engineering Sciences, 30(4):330– 338. find
  23. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678. ↗
  24. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages 22631–22648. PMLR.
  25. Mounika Marreddy and Radhika Mamidi. 2023. Chapter 6 - learning sentiment analysis with word embeddings. In Dipankar Das, Anup Kumar Kolya, Abhishek Basu, and Soham Sarkar, editors, Computational Intelligence Applications for Text and Sentiment Data Analysis, Hybrid Computational Intelligence for Pattern Analysis and Understanding, pages 141–161. Academic Press. find
  26. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. find
  27. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1–17.
  28. El Habib Nfaoui and Hanane Elfaik. 2024. Evaluating arabic emotion recognition task using chatgpt models: A comparative analysis between emotional stimuli prompt, fine-tuning, and in-context learning. Journal of Theoretical and Applied Electronic Commerce Research, 19(2):1118–1141. find
  29. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.
  30. Abrar K Samara and Gheith A Abandah. 2021. Investigating fast bilstm neural networks for arabic language applications. In 2021 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT), pages 250–255. IEEE. find
  31. Jennifer Santoso, Kenkichi Ishizuka, and Taiichi Hashimoto. 2024. Large language model-based emotional speech annotation using context and acoustic feature for speech emotion recognition. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11026–11030. find
  32. Zhengyan Shi, Adam X. Yang, Bin Wu, Laurence Aitchison, Emine Yilmaz, and Aldo Lipani. 2024. Instruction tuning with loss over instructions. In Advances in Neural Information Processing Systems, volume 37, pages 69176–69205. Curran Associates, Inc. find
  33. V. K. Singh, R. Piryani, A. Uddin, and P. Waila. 2013. Sentiment analysis of movie reviews: A new feature-based heuristic for aspect-level sentiment classification. In 2013 International Mutli-Conference on Automation, Computing, Communication, Control and Compressed Sensing (iMac4s), pages 712–717. find
  34. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca . ↗
  35. Wajdi Zaghouani, Hamdy Mubarak, and Md Rafiul Biswas. 2024. So hateful! building a multi-label hate speech annotated arabic dataset. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 15044–15055.
  36. Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/biswas-2025-evaluation-pretrained-instruction-based-pretrained/