Elamur Research⌕
☰
home / papers / 2024 / Improving Arabic Multi-Label Emotion Classification using Stacked Embeddings and Hybrid Loss Function

Improving Arabic Multi-Label Emotion Classification using Stacked Embeddings and Hybrid Loss Function

Summary

The paper addresses multi-label emotion classification for Arabic tweets. Arabic is relatively low-resource, its emotion datasets are small, a few emotions dominate the label distribution, and labels tend to co-occur. The authors fine-tune three Arabic encoders, ArabicBERT, MarBERT and AraBERT, and concatenate their contextual embeddings into one richer representation. A second-stage meta-learner reads these stacked vectors: a Bi-LSTM models the sequence, and a fully connected head outputs one probability per emotion. Training uses a hybrid objective with three parts: per-class weights to boost rare emotions, a label correlation matrix to capture which emotions appear together, and a contrastive term to pull similar examples together and push dissimilar ones apart. The system reaches Jaccard accuracy of 0.81, F1 of 0.67 and Hamming loss of 0.15. Per-class analysis shows a smaller performance gap between frequent and rare emotions, and ablations credit each component. The work matters because it offers a reusable recipe for imbalanced, correlated multi-label problems in languages with little data.

Key points

  • The backbones are ArabicBERT, MarBERT and AraBERT. Each is fine-tuned, and their embeddings are stacked as input to a Bi-LSTM plus fully connected meta-learner.
  • The hybrid loss combines class weighting, a label correlation matrix and contrastive learning. It is positioned against BCE, Focal Loss and MCFL.
  • Reported scores are Jaccard accuracy 0.81, F1 0.67 and Hamming loss 0.15. The authors claim these beat earlier Arabic MLEC systems, which are often benchmarked on SemEval-2018 Task 1 E-c Arabic.
  • Anger, Joy and Sadness are the majority classes. Anticipation, Surprise, Trust and Neutral are the minority classes, and these improved noticeably under the hybrid loss.
  • Code and data are released at github.com/nisarahmedrana/ArabicMLEC.

Interesting ideas

  • I1. Stacking outputs from several independently fine-tuned encoders and training a small second-stage learner on top lets the combiner learn when each source is reliable, rather than averaging blindly. — For Mova (learn): Instead of simple voting among witness parsers, a lightweight meta-learner could be trained on their agreement patterns to decide which silver arcs to trust.
  • I2. Adding a label co-occurrence matrix to the loss rewards predictions whose label sets look like real combinations, which helps whenever outputs are structured sets rather than independent choices. — For Mova (frames): Speech-act, irony and implicature tags in crate prag often co-occur, and a correlation prior learned from annotated data could flag implausible tag combinations as explainable warnings.
  • I3. Weighting classes by rarity during training keeps rare categories from being ignored, a cheap fix for any skewed label distribution. — For Mova (tree): Rare dependency relations in UD treebanks, such as vocative, dislocated or reparandum, could get higher weight when training the perceptron parser or gparse.
  • I4. Reporting results separately for majority and minority classes exposes whether a gain is real or just driven by frequent labels. — For Mova (learn): Evaluations could list per-deprel and per-frame scores next to overall LAS, so a model that improves the average while degrading rare relations is caught.
  • I5. A contrastive term that separates dissimilar examples sharpens the boundaries between categories that are easily confused. — For Mova (frames): A similar objective could help separate near-neighbour VerbNet classes when assigning semantic roles, with the confusable class pairs logged for explanation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.03979 · arXiv 2410.03979

@misc{aslam2024improving,
  title = {{Improving Arabic Multi-Label Emotion Classification using Stacked Embeddings and Hybrid Loss Function}},
  author = {Aslam, Muhammad Azeem and Jun, Wang and Ahmed, Nisar and Zaman, Muhammad Imran and Yanan, Li and Hongfei, Hu and Shiyu, Wang and Liu, Xin},
  year = {2024},
  doi = {10.48550/arxiv.2410.03979},
  eprint = {2410.03979},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.03979}
}

References

72 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alqahtani, G., Alothaim, A.: Emotion analysis of arabic tweets: Language models and available resources. Frontiers in Artificial Intelligence 5, 843038 (2022) find
  2. Wikarsa, L., Thahir, S.N.: A text mining application of emotion classifications of twitter’s users using naive bayes method. In: 2015 1st International Conference on Wireless and Telematics (ICWT), pp. 1–6 (2015). IEEE find
  3. Goldenberg, A., Gross, J., Garcia, D.: Emotional sharing on social media: How twitter replies contribute to increased emotional intensity (2018) find
  4. Milazzo, F., Augello, A., Pilato, G., Gentile, V., Gentile, A., Sorce, S.: Exploiting correlation between body gestures and spoken sentences for real-time emotion recognition. In: Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter, pp. 1–6 (2017) find
  5. Wankhade, M., Rao, A.C.S., Kulkarni, C.: A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review 55(7), 5731–5780 (2022) find
  6. Acheampong, F.A., Wenyu, C., Nunoo-Mensah, H.: Text-based emotion detection: Advances, challenges, and opportunities. Engineering Reports 2(7), 12189 (2020) find
  7. Plutchik, R.: A general psychoevolutionary theory of emotion. In: Theories of Emotion, pp. 3–33. Elsevier, ??? (1980) find
  8. Ekman, P.: An argument for basic emotions. Cognition & emotion 6(3-4), 169–200 (1992) find our card
  9. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013) ↗
  10. Boudad, N., Ezzahid, S., Faizi, R., Thami, R.O.H.: Exploring the use of word embedding and deep learning in arabic sentiment analysis. In: Advanced Intelligent Systems for Sustainable Development (AI2SD’2019) Volume 4-Advanced Intelligent Systems for Applied Computing Sciences, pp. 243–253 (2020). Springer find
  11. Sabri, T., Bahassine, S., El Beggar, O., Kissi, M.: An improved arabic text classification method using word embedding. International Journal of Electrical & Computer Engineering (2088-8708) 14(1) (2024) find
  12. Gao, T., Yao, X., Chen, D.: Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821 (2021) ↗ our card
  13. Zhou, Y., Kang, X., Ren, F.: Employing contrastive strategies for multi-label textual emotion recognition. In: International Conference on Intelligent Information Processing, pp. 299–310 (2022). Springer find
  14. Tursunov, A., Kwon, S., Pang, H.-S.: Discriminating emotions in the valence dimension from speech using timbre features. Applied Sciences 9(12), 2470 (2019) find
  15. Bengio, Y., Courville, A., Vincent, P.: Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35(8), 1798–1828 (2013) find our card
  16. Lin, N., Qin, G., Wang, G., Zhou, D., Yang, A.: An effective deployment of contrastive learning in multi-label text classification. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 8730–8744 (2023) find
  17. Zheng, X., Yu, Z., Chen, L., Zhu, F., Wang, S.: Multi-label contrastive focal loss for pedestrian attribute recognition. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 7349–7356 (2021). IEEE find
  18. Chen, Z., Yang, C., Zhu, M., Peng, Z., Yuan, Y.: Personalized retrogress-resilient federated learning toward imbalanced medical data. IEEE Transactions on Medical Imaging 41(12), 3663–3674 (2022) find
  19. Haralabopoulos, G., Anagnostopoulos, I., McAuley, D.: Ensemble deep learning for multilabel binary classification of user-generated content. Algorithms 13(4), 83 (2020) find
  20. Nandwani, P., Verma, R.: A review on sentiment analysis and emotion detection from text. Social network analysis and mining 11(1), 81 (2021) find
  21. Mohammad, S., Bravo-Marquez, F., Salameh, M., Kiritchenko, S.: Semeval-2018 task 1: Affect in tweets. In: Proceedings of the 12th International Workshop on Semantic Evaluation, pp. 1–17 (2018) find our card
  22. Mansy, A., Rady, S., Gharib, T.: An ensemble deep learning approach for emotion detection in arabic tweets. International Journal of Advanced Computer Science and Applications 13(4) (2022) find
  23. Khalil, E.A.H., Houby, E.M.E., Mohamed, H.K.: Deep learning for emotion analysis in arabic tweets. Journal of Big Data 8(1), 136 (2021) find
  24. Elfaik, H., et al.: Combining context-aware embeddings and an attentional deep learning model for arabic affect analysis on twitter. IEEE Access 9, 111214– 111230 (2021) find
  25. Abdelali, A., Mubarak, H., Chowdhury, S.A., Hasanain, M., Mousi, B., Boughorbel, S., Kheir, Y.E., Izham, D., Dalvi, F., Hawasly, M., et al.: Benchmarking arabic ai with large language models. arXiv preprint arXiv:2305.14982 (2023) ↗
  26. Alswaidan, N., Menai, M.E.B.: Hybrid feature model for emotion recognition in arabic text. IEEE Access 8, 37843–37854 (2020) find
  27. Samy, A.E., El-Beltagy, S.R., Hassanien, E.: A context integrated model for multi-label emotion detection. Procedia computer science 142, 61–71 (2018) find
  28. Asudani, D.S., Nagwani, N.K., Singh, P.: Impact of word embedding models on text analytics in deep learning environment: a review. Artificial intelligence review 56(9), 10345–10425 (2023) find
  29. Mao, X., Chang, S., Shi, J., Li, F., Shi, R.: Sentiment-aware word embedding for emotion classification. Applied Sciences 9(7), 1334 (2019) find
  30. Fang, L., Luo, Y., Feng, K., Zhao, K., Hu, A.: A knowledge-enriched ensemble method for word embedding and multi-sense embedding. IEEE Transactions on Knowledge and Data Engineering (2022) find
  31. Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014) find
  32. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems 26 (2013) find
  33. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., Mikolov, T.: Fasttext. zip: Compressing text classification models. arXiv preprint arXiv:1612.03651 (2016) ↗ our card
  34. Li, X., Sun, X., Meng, Y., Liang, J., Wu, F., Li, J.: Dice loss for data-imbalanced nlp tasks. arXiv preprint arXiv:1911.02855 (2019) ↗ our card
  35. Al-Twairesh, N.: The evolution of language models applied to emotion analysis of arabic tweets. Information 12(2), 84 (2021) find
  36. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) ↗
  37. David, M.S., Renjith, S.: Comparison of word embeddings in text classification based on rnn and cnn. In: IOP Conference Series: Materials Science and Engineering, vol. 1187, p. 012029 (2021). IOP Publishing find
  38. Muromägi, A., Sirts, K., Laur, S.: Linear ensembles of word embedding models. arXiv preprint arXiv:1704.01419 (2017) ↗
  39. Speer, R., Chin, J.: An ensemble method to produce high-quality word embeddings (2016). arXiv preprint arXiv:1604.01692 (2016) ↗
  40. Antoun, W., Baly, F., Hajj, H.: Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104 (2020) ↗ our card
  41. Safaya, A., Abdullatif, M., Yuret, D.: Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media. arXiv preprint arXiv:2007.13184 (2020) ↗ our card
  42. Abdul-Mageed, M., Elmadany, A., Nagoudi, E.M.B.: Arbert & marbert: Deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785 (2020) ↗ our card
  43. Aoumeur, N.E., Li, Z., Alshari, E.M.: Improving the polarity of text through word2vec embedding for primary classical arabic sentiment analysis. Neural processing letters 55(3), 2249–2264 (2023) find
  44. Fadel, A., Tuffaha, I., Al-Ayyoub, M.: Pretrained ensemble learning for fine-grained propaganda detection. In: Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pp. 139–142 (2019) find
  45. Alammary, A.S.: Bert models for arabic text classification: a systematic review. Applied Sciences 12(11), 5720 (2022) find
  46. Berrimi, M., Oussalah, M., Moussaoui, A., Saidi, M.: Attention mechanism architecture for arabic sentiment analysis. ACM Transactions on Asian and Low-Resource Language Information Processing 22(4), 1–26 (2023) find
  47. Elbedwehy, S., Medhat, T.: Improved arabic image captioning model using feature concatenation with pre-trained word embedding. Neural Computing and Applications 35(26), 19051–19067 (2023) find
  48. Kamr, A.M., Mohamed, E.: akabert at semeval-2022 task 6: an ensemble transformer-based model for arabic sarcasm detection. In: Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp. 885–890 (2022) find our card
  49. Chouikhi, H., Chniter, H., Jarray, F.: Stacking bert based models for arabic sentiment analysis. In: KEOD, pp. 144–150 (2021) find
  50. Padma, S., Devi, B.K.: Classification of real value and complex value data using machine learning concepts. International Journal of Computer Science and Information Technologies 5(2), 2361–2364 (2014) find
  51. Kumar, P., Bhatnagar, R., Gaur, K., Bhatnagar, A.: Classification of imbalanced data: review of methods and applications. In: IOP Conference Series: Materials Science and Engineering, vol. 1099, p. 012077 (2021). IOP Publishing find
  52. Durand, T., Mehrasa, N., Mori, G.: Learning a deep convnet for multi-label classification with partial labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 647–657 (2019) find
  53. Dong, J.: Focal loss improves the model performance on multi-label image classifications with imbalanced data. In: Proceedings of the 2nd International Conference on Industrial Control Network And System Engineering Research, pp. 18–21 (2020) find
  54. Jiang, D., He, J., et al.: Text semantic classification of long discourses based on neural networks with improved focal loss. Computational Intelligence and Neuroscience 2021 (2021) find
  55. Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017) find our card
  56. Sun, X., Dong, K., Ma, L., Sutcliffe, R., He, F., Chen, S., Feng, J.: Drug-drug interaction extraction via recurrent hybrid convolutional neural networks with an improved focal loss. Entropy 21(1), 37 (2019) find
  57. Zhu, Z., Dai, W., Hu, Y., Li, J.: Speech emotion recognition model based on bi-gru and focal loss. Pattern Recognition Letters 140, 358–365 (2020) find
  58. Ridnik, T., Ben-Baruch, E., Zamir, N., Noy, A., Friedman, I., Protter, M., Zelnik-Manor, L.: Asymmetric loss for multi-label classification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 82–91 (2021) find
  59. Deng, J., Ren, F.: Multi-label emotion detection via emotion-specified feature extraction and emotion correlation learning. IEEE Transactions on Affective Computing 14(1), 475–486 (2020) find
  60. Bendjoudi, I., Hamad, D., Vanderhaegen, F., Dornaika, F.: Audio-visual and heart signals for attention and emotion analysis. In: ESREL 2020 PSAM 15-The 30th European Safety and Reliability Conference and the 15th Probabilistic Safety Assessment and Management Conference, pp. 2795–2801 (2020). Research Publishing Services find
  61. Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. Advances in neural information processing systems 33, 18661–18673 (2020) find
  62. Hu, X., Lin, L., Liu, A., Wen, L., Philip, S.Y., et al.: A multi-level supervised contrastive learning framework for low-resource natural language inference. IEEE/ACM Transactions on Audio, Speech, and Language Processing (2023) find
  63. Eden, A.H.: Three paradigms of computer science. Minds and machines 17, 135– 167 (2007) find
  64. Alrasheedy, M.N., Muniyandi, R.C., Fauzi, F.: Text-based emotion detection and applications: A literature review. In: 2022 International Conference on Cyber Resilience (ICCR), pp. 1–9 (2022). IEEE find
  65. Unnithan, N.A., Shalini, K., HB, B.G., Soman, K., et al.: Amrita student at semeval-2018 task 1: distributed representation of social media text for affects in tweets. In: Proceedings of The 12th International Workshop on Semantic Evaluation, pp. 319–323 (2018) find
  66. Abdelali, A., Hassan, S., Mubarak, H., Darwish, K., Samih, Y.: Pre-training bert on arabic tweets: Practical considerations. arXiv preprint arXiv:2102.10684 (2021) ↗ our card
  67. George, A., HB, B.G., Soman, K., et al.: Teamcen at semeval-2018 task 1: global vectors representation in emotion detection. In: Proceedings of the 12th International Workshop on Semantic Evaluation, pp. 334–338 (2018) find
  68. Abdullah, M., Shaikh, S.: Teamuncc at semeval-2018 task 1: Emotion detection in english and arabic tweets using deep learning. In: Proceedings of the 12th International Workshop on Semantic Evaluation, pp. 350–357 (2018) find
  69. Mulki, H., Ali, C.B., Haddad, H., Babaoğlu, I.: Tw-star at semeval-2018 task 1: Preprocessing impact on multi-label emotion classification. In: Proceedings of The 12th International Workshop on Semantic Evaluation, pp. 167–171 (2018) find
  70. Badaro, G., El Jundi, O., Khaddaj, A., Maarouf, A., Kain, R., Hajj, H., El-Hajj, W.: Ema at semeval-2018 task 1: Emotion mining for arabic. In: Proceedings of The 12th International Workshop on Semantic Evaluation, pp. 236–244 (2018) find
  71. Xu, P., Liu, Z., Winata, G.I., Lin, Z., Fung, P.: Emograph: Capturing emotion correlations using graph networks. arXiv preprint arXiv:2008.09378 (2020) ↗
  72. Alhuzali, H., Ananiadou, S.: Spanemo: Casting multi-label emotion classification as span-prediction. arXiv preprint arXiv:2101.10038 (2021) ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aslam-2024-improving-arabic-multi-label-emotion/