Elamur Research⌕
☰
home / papers / 2021 / Empirical evaluation of shallow and deep learning classifiers for Arabic sentiment analysis

Empirical evaluation of shallow and deep learning classifiers for Arabic sentiment analysis

Summary

The paper asks whether deep learning really loses to classical "shallow" classifiers on Arabic sentiment analysis, as some earlier work reported. The authors argue that the earlier result came from training sets that were too small. To test this, they ran a broad benchmark on two large multi-dialect review corpora, HARD (hotels) and BRAD (books), which mix MSA and Dialectal Arabic. They compared eight common shallow classifiers with CNN, LSTM, GRU and every hybrid of the three, plus a transformer model with several embedding options, including araBERT. They evaluated binary and 5-label rating prediction using accuracy and F1. They also plotted learning curves over dataset size and examined misclassified reviews. Deep models won in almost every setting, and the margin depended on how much training data was available. The transformer combined with araBERT was the strongest model except on very small data. The work gives practical guidance on when neural models pay off for a morphologically rich, dialect-heavy, lower-resource language.

Key points

  • Corpora: HARD has 373,750 reviews from booking.com (unbalanced variant, about 90K unique tokens). BRAD has 508,538 book reviews (about 220K unique tokens).
  • Shallow side: Random Forest was the top classifier, with Decision Tree in second place and AdaBoost after it. Random Forest also beat the deep models on 5-label classification.
  • The deep architectures (CNN, LSTM, GRU and hybrids) scored roughly the same with a default embedding layer. The transformer gained much more from richer embeddings, with araBERT giving the largest gain.
  • Rough rule from the learning curves:
    • Shallow methods are preferable below about 10,000 sentences.
    • Deep methods are the better choice at tens of thousands of sentences.
    • Transformer+araBERT leads everywhere except below about 3,000 sentences.
  • Both families scored higher on HARD than on BRAD, even though BRAD is 33% larger. The authors attribute this to BRAD's vocabulary being more than twice the size.

Interesting ideas

  • I1. The authors use a learning curve over training-set size to find where a complex model starts beating a simple one, instead of reporting a single comparison at one data size. — For Mova (learn): Train the perceptron parser and gparse on growing slices of a treebank and plot both. This shows the data size at which the GPU parser actually earns its cost for each UD dialect or domain.
  • I2. They explain a contradiction in the literature by naming a specific confound, dataset size, and then testing it directly rather than simply asserting that their own result is right. — For Mova (learn): When a new result clashes with an earlier evaluation, write down the suspected confound (size, domain, annotation dialect) and design a targeted check, similar to a negative control.
  • I3. They treat vocabulary spread as a hidden difficulty factor: the bigger corpus was harder because its unique-token space was much larger. — For Mova (tokens): Report type/token ratio and out-of-vocabulary rates per corpus next to tagger scores. Use lemma and morphology normalization to shrink the effective vocabulary and measure how much that helps.
  • I4. Their error analysis shows that failures cluster in reviews with mixed praise and complaints, while reviews with explicit cue words were classified correctly. — For Mova (frames): Assign stance per clause or per aspect frame (who praised what, who complained about what) instead of per document. This makes mixed reviews explainable rather than forcing a single label.
  • I5. They run an ablation that keeps the architecture fixed and swaps only the embedding source (dataset-derived, custom web-derived, pre-trained). — For Mova (tree): Hold the gparse architecture constant and vary only the encoder or embeddings. This isolates how much parsing accuracy comes from representation versus parser design.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Sentiment & emotion · Evaluation & benchmarks · Semitic languages

Institutions

🎓University of SharjahSharjah · United Arab Emirates5 works · 152 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3466171 · arXiv 2112.00534

@misc{nassif2021empirical,
  title = {{Empirical evaluation of shallow and deep learning classifiers for Arabic sentiment analysis}},
  author = {Nassif, Ali Bou and Darya, Abdollah Masoud and Elnagar, Ashraf},
  year = {2021},
  doi = {10.1145/3466171},
  eprint = {2112.00534},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3466171}
}

References

63 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mahmoud Al-Ayyoub, Abed Allah Khamaiseh, Yaser Jararweh, and Mohammed N Al-Kabi. 2019. A comprehensive survey of arabic sentiment analysis. Information Processing & Management 56, 2 (2019), 320–342. find
  2. Mahmoud Al-Ayyoub, Aya Nuseir, Kholoud Alsmearat, Yaser Jararweh, and Brij Gupta. 2018. Deep learning for Arabic NLP: A survey. Journal of computational science 26 (2018), 522–531. find
  3. Sadam Al-Azani and El-Sayed El-Alfy. 2018. Emojis-based sentiment classification of Arabic microblogs using deep recurrent neural networks. In 2018 International Conference on Computing Sciences and Engineering (ICCSE). IEEE, 1–6. find
  4. Sadam Al-Azani and El-Sayed M El-Alfy. 2017. Hybrid deep learning for sentiment polarity determination of arabic microblogs. In International Conference on Neural Information Processing. Springer, 491–500. find
  5. Ahmad Al-Sallab, Ramy Baly, Hazem Hajj, Khaled Bashir Shaban, Wassim El-Hajj, and Gilbert Badaro. 2017. Aroma: A recursive deep learning model for opinion mining in arabic as a low resource language. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 16, 4 (2017), 1–20. find
  6. Ahmad Al Sallab, Hazem Hajj, Gilbert Badaro, Ramy Baly, Wassim El Hajj, and Khaled Bashir Shaban. 2015. Deep learning models for sentiment analysis in Arabic. In Proceedings of the second workshop on Arabic natural language processing. 9–17. find
  7. Mohammad Al-Smadi, Omar Qawasmeh, Mahmoud Al-Ayyoub, Yaser Jararweh, and Brij Gupta. 2018. Deep Recurrent neural network vs. support vector machine for aspect-based sentiment analysis of Arabic hotels’ reviews. Journal of computational science 27 (2018), 386–393. find
  8. Abdulaziz M Alayba, Vasile Palade, Matthew England, and Rahat Iqbal. 2018. A combined CNN and LSTM model for arabic sentiment analysis. In International Cross-Domain Conference for Machine Learning and Knowledge Extraction. Springer, 179–191. find
  9. Huda Abdulrahman Almuzaini and Aqil M Azmi. 2020. Impact of stemming and word embedding on deep learning-based arabic text categorization. IEEE Access 8 (2020), 127913–127928. Manuscript submitted to ACM find
  10. Anwar Alnawas and Nursal Arici. 2019. Sentiment Analysis of Iraqi Arabic Dialect on Facebook Based on Distributed Representations of Documents. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 18, 3 (2019), 20. find
  11. Naomi S Altman. 1992. An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician 46, 3 (1992), 175–185. find
  12. A Aziz Altowayan and Ashraf Elnagar. 2017. Improving Arabic sentiment analysis with sentiment-specific embeddings. In 2017 IEEE International Conference on Big Data (Big Data). IEEE, 4314–4320. find
  13. Gilbert Badaro, Ramy Baly, Rana Akel, Linda Fayad, Jeffrey Khairallah, Hazem Hajj, Khaled Shaban, and Wassim El-Hajj. 2015. A light lexicon-based mobile application for sentiment mining of arabic tweets. In Proceedings of the second workshop on arabic natural language processing. 18–25. find
  14. Gilbert Badaro, Ramy Baly, Hazem Hajj, Wassim El-Hajj, Khaled Bashir Shaban, Nizar Habash, Ahmad Al-Sallab, and Ali Hamdi. 2019. A survey of opinion mining in Arabic: A comprehensive system perspective covering challenges and advances in tools, resources, models, applications, and visualizations. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 18, 3 (2019), 27. find
  15. Gilbert Badaro, Ramy Baly, Hazem Hajj, Nizar Habash, and Wassim El-Hajj. 2014. A large scale Arabic sentiment lexicon for Arabic opinion mining. In Proceedings of the EMNLP 2014 workshop on arabic natural language processing (ANLP). 165–173. find
  16. Gilbert Badaro, Obeida El Jundi, Alaa Khaddaj, Alaa Maarouf, Raslan Kain, Hazem Hajj, and Wassim El-Hajj. 2018. Ema at semeval-2018 task 1: Emotion mining for arabic. In Proceedings of The 12th International Workshop on Semantic Evaluation. 236–244. find
  17. Gilbert Badaro, Hazem Hajj, and Nizar Habash. 2020. A Link Prediction Approach for Accurately Mapping a Large-scale Arabic Lexical Resource to English WordNet. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 19, 6 (2020), 1–38. find
  18. Fady Baly, Hazem Hajj, et al. 2020. AraBERT: Transformer-based Model for Arabic Language Understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection. 9–15. find our card
  19. Ramy Baly, Gilbert Badaro, Georges El-Khoury, Rawan Moukalled, Rita Aoun, Hazem Hajj, Wassim El-Hajj, Nizar Habash, and Khaled Shaban. 2017. A characterization study of arabic twitter data with a benchmarking for state-of-the-art opinion mining models. In Proceedings of the third Arabic natural language processing workshop. 110–118. find
  20. Ramy Baly, Gilbert Badaro, Ali Hamdi, Rawan Moukalled, Rita Aoun, Georges El-Khoury, Ahmad Al Sallab, Hazem Hajj, Nizar Habash, Khaled Shaban, et al. 2017. Omam at semeval-2017 task 4: Evaluation of english state-of-the-art sentiment analysis models for arabic and a new topic-based model. In Proceedings of the 11th international workshop on semantic evaluation (SEMEVAL-2017). 603–610. find
  21. Ramy Baly, Hazem Hajj, Nizar Habash, Khaled Bashir Shaban, and Wassim El-Hajj. 2017. A sentiment treebank and morphologically enriched recursive deep models for effective sentiment analysis in arabic. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 16, 4 (2017), 23. find
  22. Majdi Beseiso and Haytham Elmousalami. 2020. Subword attentive model for Arabic sentiment analysis: A deep learning approach. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 19, 2 (2020), 1–17. find
  23. Naaima Boudad, Rdouan Faizi, Rachid Oulad Haj Thami, and Raddouane Chiheb. 2017. Sentiment analysis in Arabic: A review of the literature. Ain Shams Engineering Journal (2017). find
  24. Leo Breiman. 2001. Random forests. Machine learning 45, 1 (2001), 5–32. find
  25. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 (2014). ↗ our card
  26. Jesse Davis and Mark Goadrich. 2006. The relationship between Precision-Recall and ROC curves. In Proceedings of the 23rd international conference on Machine learning. ACM, 233–240. find
  27. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). ↗ our card
  28. Hai Ha Dohaiha, PWC Prasad, Angelika Maag, and Abeer Alsadoon. 2018. Deep learning for aspect-based sentiment analysis: a comparative review. Expert Systems With Applications (2018). find
  29. Omar Einea, Ashraf Elnagar, and Ridhwan Al Debsi. 2019. Sanad: Single-label arabic news articles dataset for automatic text categorization. Data in brief 25 (2019), 104076. find
  30. Obeida ElJundi, Wissam Antoun, Nour El Droubi, Hazem Hajj, Wassim El-Hajj, and Khaled Shaban. 2019. hulmona: The universal language model in arabic. In Proceedings of the Fourth Arabic Natural Language Processing Workshop. 68–77. find
  31. Ashraf Elnagar. 2016. Investigation on sentiment analysis for Arabic reviews. In 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA). IEEE, 1–7. find
  32. Ashraf Elnagar, Ridhwan Al-Debsi, and Omar Einea. 2020. Arabic text classification using deep learning models. Information Processing & Management 57, 1 (2020), 102121. find
  33. Ashraf Elnagar and Omar Einea. 2016. BRAD 1.0: Book reviews in Arabic dataset. In 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA). 1–8. find
  34. Ashraf Elnagar, Yasmin S Khalifa, and Anas Einea. 2018. Hotel Arabic-reviews dataset construction for sentiment analysis applications. In Intelligent Natural Language Processing: Trends and Applications. Springer, 35–52. find
  35. Ashraf Elnagar, Leena Lulu, and Omar Einea. 2018. An Annotated Huge Dataset for Standard and Colloquial Arabic Reviews for Subjective Sentiment Analysis. Procedia computer science 142 (2018), 182–189. find
  36. Ashraf Elnagar, Sane M Yagi, Ali Bou Nassif, Ismail Shahin, and Said A Salloum. 2021. Systematic Literature Review of Dialectal Arabic: Identification and Detection. IEEE Access 9 (2021), 31010–31042. Manuscript submitted to ACM find
  37. Yoav Freund and Robert E Schapire. 1997. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences 55, 1 (1997), 119–139. find
  38. Yoav Freund, Robert E Schapire, et al. 1996. Experiments with a new boosting algorithm. In icml, Vol. 96. Citeseer, 148–156. find
  39. Edmund A Gehan. 1965. A generalized Wilcoxon test for comparing arbitrarily singly-censored samples. Biometrika 52, 1-2 (1965), 203–224. find
  40. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning Word Vectors for 157 Languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). find our card
  41. Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. 2016. LSTM: A search space odyssey. IEEE transactions on neural networks and learning systems 28, 10 (2016), 2222–2232. find
  42. Ali Hamdi, Khaled Shaban, and Anazida Zainal. 2018. Clasenti: A class-specific sentiment analysis framework. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 17, 4 (2018), 1–28. find
  43. Sepp Hochreiter and Jurgen Schmidhuber. 1996. Bridging long time lags by weight guessing and “Long Short-Term Memory”. Spatiotemporal models in biological and artificial systems 37, 65-72 (1996), 11. find
  44. Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In International Conference on Machine Learning. 2342–2350. find
  45. Jawad Khan, Aftab Alam, Jamil Hussain, and Young-Koo Lee. 2019. EnSWF: effective features extraction and selection in conjunction with ensemble learning methods for document sentiment classification. Applied Intelligence (2019), 1–23. find
  46. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014). ↗ our card
  47. Yann LeCun, D Touresky, G Hinton, and T Sejnowski. 1988. A theoretical framework for back-propagation. In Proceedings of the 1988 connectionist models summer school, Vol. 1. CMU, Pittsburgh, Pa: Morgan Kaufmann, 21–28. find
  48. Lin Li, Tiong-Thye Goh, and Dawei Jin. 2018. How textual quality of online reviews affect classification performance: a case of deep learning sentiment analysis. Neural Computing and Applications (2018), 1–29. find
  49. Andy Liaw, Matthew Wiener, et al. 2002. Classification and regression by randomForest. R news 2, 3 (2002), 18–22. find
  50. Weibo Liu, Zidong Wang, Xiaohui Liu, Nianyin Zeng, Yurong Liu, and Fuad E Alsaadi. 2017. A survey of deep neural network architectures and their applications. Neurocomputing 234 (2017), 11–26. find
  51. Leena Lulu and Ashraf Elnagar. 2018. Automatic Arabic Dialect Classification Using Deep Learning Models. Procedia Computer Science 142 (2018), 262 – 269. find
  52. Gjorgji Madjarov, Dragi Kocev, Dejan Gjorgjevikj, and Sašo Džeroski. 2012. An extensive experimental comparison of methods for multi-label learning. Pattern recognition 45, 9 (2012), 3084–3104. find
  53. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013). ↗ our card
  54. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation. 1–17. find our card
  55. Ali Bou Nassif, Ashraf Elnagar, Ismail Shahin, and Safaa Henno. 2020. Deep learning for Arabic subjective sentiment analysis: Challenges and research opportunities. Applied Soft Computing (2020), 106836. find
  56. Andrew Ng. 2015. What data scientists should know about deep learning. URL https://www.slideshare. net/ExtractConf 44 (2015). ↗
  57. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532–1543. find our card
  58. F Rosenblatt. 1960. Perceptrons and the Theory of Brain Mechanisms: Cornell Aeronautical Laboratory. Technical Report. Report No. VG-1196-G-8. find
  59. S Rasoul Safavian and David Landgrebe. 1991. A survey of decision tree classifier methodology. IEEE transactions on systems, man, and cybernetics 21, 3 (1991), 660–674. find
  60. Jürgen Schmidhuber. 2015. Deep learning in neural networks: An overview. Neural networks 61 (2015), 85–117. find
  61. Amira Shoukry and Ahmed Rafea. 2012. Sentence-level Arabic sentiment analysis. In 2012 International Conference on Collaboration Technologies and Systems (CTS). IEEE, 546–550. find
  62. Mohammad Tubishat, Mohammad AM Abushariah, Norisma Idris, and Ibrahim Aljarah. 2019. Improved whale optimization algorithm for feature selection in Arabic sentiment analysis. Applied Intelligence 49, 5 (2019), 1688–1707. find
  63. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998–6008. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nassif-2021-empirical-evaluation-shallow-deep-learning/