Elamur Research⌕
☰
home / papers / 2024 / Advancing Arabic Sentiment Analysis: ArSen Benchmark and the Improved Fuzzy Deep Hybrid Network

Advancing Arabic Sentiment Analysis: ArSen Benchmark and the Improved Fuzzy Deep Hybrid Network

Summary

Arabic sentiment analysis has few resources, and the common benchmarks (Gold Standard, SemEval) are old and contain fewer than 10,000 items. The authors build ArSen, a COVID-19 tweet benchmark. They drew roughly 10k tweets at random from the AROT-COV23 collection, stripped fields that could identify users, and kept engagement counts and author statistics as context. Trained advanced Arabic speakers labelled each tweet as positive, neutral or negative, with three annotators per tweet. They also adapt FDHN, a fuzzy-logic multi-channel network first built for fake-news detection, into IFDHN. In IFDHN, a TextCNN reads the tweet and a tuned CNN-BiLSTM branch handles the numeric context, which passes through a fuzzy layer. Textual context fields turned out to hurt performance and were dropped. IFDHN beats RoBERTa, AraT5-Tweet-Base, FNet and the original FDHN, and slightly exceeds LLaMA-3-8B while using a small fraction of its compute. The work shows that a small, context-aware model can compete with an LLM on a modest classification benchmark.

Key points

  • ArSen has about 10.2k labelled tweets split 80/10/10 (8153 / 1020 / 1020). The classes are heavily skewed: neutral 69.4%, positive 15.3%, negative 15.3%.
  • A label is kept when at least 2 of 3 annotators agree, and the tweet is discarded otherwise. Guidelines count sarcasm, warnings and obituaries as negative, and news, ads, advice and questions as neutral.
  • On the test set, IFDHN reaches 78.12% accuracy, 55.83% F1-Macro and 78.12% F1-Micro, which is +2.17 accuracy points over LLaMA-3-8B.
  • IFDHN uses about 0.23% of LLaMA-3-8B's compute and runs in 0.44 s, roughly 422× faster.
  • In the ablation, tweet text matters most, interaction metrics (like/quote/reply/retweet counts) come second, and user meta-data adds a further gain. The created_at and description fields were removed as harmful.

Interesting ideas

  • I1. The 2-of-3 annotator vote throws out items with no majority instead of forcing a label, which trades corpus size for cleaner gold data.
  • I2. The ablation runs three ways: each feature group alone, leave-one-out, and all together. This separates what a component does on its own from what it adds in combination.
  • I3. With 69% neutral labels, the non-text channels alone hit about 0.72 test accuracy yet only about 0.28 F1-Macro. The two metrics together expose a model that has learned little beyond the majority class.
  • I4. The guidelines tie sentiment to what the author felt when posting, not to the annotator's reaction, and they treat sarcasm as negative.
  • I5. Fuzzy membership scores between 0 and 1 replace hard class decisions for meanings that are inherently ambiguous.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver treebanks built from several witness parsers could keep only arcs or labels that a majority agrees on and drop the contested ones instead of picking a winner.
  • I2 (learn): The same three-way scheme could measure each knowledge source (VerbNet roles, WordNet, coreference) in the pipeline, both alone and by removal.
  • I3 (learn): Imbalanced evaluations should always report macro scores next to a majority-class baseline as a negative control, so trivial predictors cannot pass as progress.
  • I4 (frames): The irony and speech-act modules could record whose stance a frame expresses (author versus reader) and map sarcastic surface praise to the opposite polarity.
  • I5 (world): Entity states such as "happy" or "worried" could carry graded degrees instead of booleans, so later answering steps can explain borderline cases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Sentiment & emotion Evaluation & benchmarks Semitic languages

Institutions

🎓Georgia Institute of TechnologyAtlanta · United States55 works · 21,012 citations🎓University College DublinDublin · Ireland7 works · 1,146 citations

Also on the byline: Huaibei Normal University · Wuhu Institute of Technology.

Cite

Cite as Yang Fang, Cheng Xu, Shuhao Guan, Nan Yan and Yuke Mei. 2024. Advancing Arabic Sentiment Analysis: ArSen Benchmark and the Improved Fuzzy Deep Hybrid Network. Proceedings of the 28th Conference on Computational Natural Language Learning.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.conll-1.39

@inproceedings{fang2024advancing,
  title = {{Advancing Arabic Sentiment Analysis: ArSen Benchmark and the Improved Fuzzy Deep Hybrid Network}},
  author = {Yang Fang, Cheng Xu, Shuhao Guan, Nan Yan and Yuke Mei},
  booktitle = {Proceedings of the 28th Conference on Computational Natural Language Learning},
  year = {2024},
  doi = {10.18653/v1/2024.conll-1.39},
  url = {https://doi.org/10.18653/v1/2024.conll-1.39}
}

References

45 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammad Al-Smadi, Omar Qawasmeh, Mahmoud Al-Ayyoub, Yaser Jararweh, and Brij Gupta. 2018. Deep recurrent neural network vs. support vector machine for aspect-based sentiment analysis of arabic hotels’ reviews. Journal of Computational Science, 27:386– 393. find
  2. Basma Alharbi, Hind Alamro, Manal Alshehri, Zuhair Khayyat, Manal Kalkatawi, Inji Ibrahim Jaber, and Xiangliang Zhang. 2021. Asad: A twitter-based benchmark arabic sentiment analysis dataset. Preprint, arXiv:2011.00578. ↗ our card
  3. Manal Mostafa Ali. 2021. Arabic sentiment analysis about online learning to mitigate covid-19. Journal of Intelligent Systems, 30(1):524–540. find
  4. Mohamed Aly and Amir Atiya. 2013. LABR: A large scale Arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498, Sofia, Bulgaria. Association for Computational Linguistics. find
  5. Maryam Alzaid and Fethi Fkih. 2023. Sentiment analysis of students’ feedback on e-learning using a hybrid fuzzy model. Applied Sciences, 13(23). find
  6. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. Preprint, arXiv:2302.04023. ↗ our card
  7. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. LLM2vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling. our card
  8. Ranganathan Chandrasekaran, Vikalp Mehta, Tejali Valkunde, and Evangelos Moustakas. 2020. Topics, trends, and sentiments of tweets about the covid-19 pandemic: Temporal infoveillance study. J Med Internet Res, 22(10):e22624. find
  9. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol., 15(3). our card
  10. Rangan Das, Sagnik Sen, and Ujjwal Maulik. 2020. A survey on fuzzy deep neural networks. ACM Comput. Surv., 53(3). find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗ our card
  12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  13. Mazen El-Masri, Nabeela Berardinelli, and Hanady Ahmed. 2017. Successes and challenges of arabic sentiment analysis research: a literature review. Social Network Analysis and Mining, 7:22. find
  14. Yang Fang and Cheng Xu. 2024. Arsen-20: A new benchmark for arabic sentiment detection. In 5th Workshop on African Natural Language Processing. find
  15. Gabriel Elías Chanchí Golondrino, Manuel Alejandro Ospina Alarcón, and Luz Marina Sierra Martínez. 2023. Determination of the satisfaction attribute in usability tests using sentiment analysis and fuzzy logic. Int. J. Comput. Commun. Control, 18. find
  16. Shuhao Guan and Derek Greene. 2024a. Advancing post-OCR correction: A comparative study of synthetic data. In Findings of the Association for Computational Linguistics ACL 2024, pages 6036–6047, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics. find
  17. Shuhao Guan and Derek Greene. 2024b. Synthetically augmented self-supervised fine-tuning for diverse text ocr correction. In 27th European Conference on Artificial Intelligence, ECAI 2024, Santiago de Compostela. find
  18. Shuhao Guan, Cheng Xu, Moule Lin, and Derek Greene. 2024. Effective synthetic data and test-time adaptation for OCR correction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA. Association for Computational Linguistics. find
  19. Maha Heikal, Marwan Torki, and Nagwa El-Makky. 2018. Sentiment analysis of arabic tweets using deep learning. Procedia Computer Science, 142:114–122. Arabic Computational Linguistics. find
  20. Doaa Mohey El-Din Mohamed Hussein. 2018. A survey on sentiment analysis challenges. Journal of King Saud University - Engineering Sciences, 30(4):330– 338. find
  21. Dinh Tai Pham Huyen Trang Phan and Ngoc Thanh Nguyen. 2023. Fedn2: Fuzzy-enhanced deep neural networks for improvement of sentence-level sentiment analysis. Cybernetics and Systems, 0(0):1–17. find
  22. James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. 2022. FNet: Mixing tokens with Fourier transforms. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4296–4313, Seattle, United States. Association for Computational Linguistics. our card
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692. ↗
  24. May Oo Lwin, Jiahui Lu, Anita Sheldenkar, Peter Johannes Schulz, Wonsun Shin, Raj Gupta, and Yinping Yang. 2020. Global sentiments surrounding the covid-19 pandemic on twitter: Analysis of twitter trends. JMIR Public Health Surveill, 6(2):e19447. find
  25. Mounika Marreddy and Radhika Mamidi. 2023. Chapter 6 - learning sentiment analysis with word embeddings. In Dipankar Das, Anup Kumar Kolya, Abhishek Basu, and Soham Sarkar, editors, Computational Intelligence Applications for Text and Sentiment Data Analysis, Hybrid Computational Intelligence for Pattern Analysis and Understanding, pages 141–161. Academic Press. find
  26. Mahmoud Nabil, Mohamed Aly, and Amir Atiya. 2015. ASTD: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515–2519, Lisbon, Portugal. Association for Computational Linguistics. our card
  27. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics. our card
  28. OpenAI. 2024. Gpt-4 technical report. arXiv:2303.08774. ↗ our card
  29. Preprint, Kellin Pelrine, Anne Imouza, Camille Thibault, Meilina Reksoprodjo, Caleb Alexander Gupta, Joel Christoph, Jean-François Godbout, and Reihaneh Rabbany. 2023. Towards reliable misinformation mitigation: Generalization, uncertainty, and GPT-4. In The 2023 Conference on Empirical Methods in Natural Language Processing. find
  30. Eshrag Refaee and Verena Rieser. 2014. An Arabic Twitter corpus for subjectivity and sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 2268–2273, Reykjavik, Iceland. European Language Resources Association (ELRA). find
  31. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502– 518, Vancouver, Canada. Association for Computational Linguistics. our card
  32. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are emergent abilities of large language models a mirage? In Thirty-seventh Conference on Neural Information Processing Systems. our card
  33. Bingli Sun, Xiao Song, Wenxin Li, Lu Liu, Guanghong Gong, and Yan Zhao. 2024. A user review data-driven supplier ranking model using aspect-based sentiment analysis and fuzzy theory. Engineering Applications of Artificial Intelligence, 127:107224. find
  34. Sayyida Tabinda Kokab, Sohail Asghar, and Shehneela Naz. 2022. Transformer-based deep learning models for the sentiment analysis of social media data. Array, 14:100157. find
  35. Kian Long Tan, Chin Poo Lee, Kalaiarasi Sonai Muthu Anbananthen, and Kian Ming Lim. 2022. Roberta-lstm: A hybrid model for sentiment analysis with transformer and recurrent neural network. IEEE Access, 10:21517–21525. find
  36. Dimple Tiwari and Bharti Nagpal. 2022. Keaht: A knowledge-enriched attention-based hybrid transformer model for social sentiment analysis. New Gen. Comput., 40(4):1165–1202. find
  37. Srishti Vashishtha, Vedika Gupta, and Mamta Mittal. 2023. Sentiment analysis using fuzzy logic: A comprehensive literature review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 13. find
  38. William Yang Wang. 2017. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426, Vancouver, Canada. Association for Computational Linguistics. our card
  39. Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. 2024. Benchmark data contamination of large language models: A survey. Preprint, arXiv:2406.04244. ↗ our card
  40. Cheng Xu and M-Tahar Kechadi. 2023. Fuzzy deep hybrid network for fake news detection. In Proceedings of the 12th International Symposium on Information and Communication Technology, SOICT ’23, page 118–125, New York, NY, USA. Association for Computing Machinery. find
  41. Cheng Xu and M-Tahar Kechadi. 2024. An enhanced fake news detection system with fuzzy deep learning. IEEE Access, 12:88006–88021. find
  42. Cheng Xu, Jing Wang, Tianlong Zheng, Yue Cao, and Fan Ye. 2022. Prediction of prognosis and survival of patients with gastric cancer by a weighted improved random forest model: an application of machine learning in medicine. Archives of Medical Science, 18(5):1208–1220. find
  43. Cheng Xu and Nan Yan. 2023. AROT-COV23: A dataset of 500k original arabic tweets on COVID-19. In 4th Workshop on African Natural Language Processing. find
  44. Nan Yan and Cheng Xu. 2024. Decolonizing african NLP: A survey on power dynamics and data colonialism in tech development. In 5th Workshop on African Natural Language Processing. find
  45. L.A. Zadeh. 1996. Fuzzy logic = computing with words. IEEE Transactions on Fuzzy Systems, 4(2):103–111. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fang-2024-advancing-arabic-sentiment-analysis-arsen/