Elamur Research⌕
☰
home / papers / 2022 / Arabic Sentiment Analysis by Pretrained Ensemble

Arabic Sentiment Analysis by Pretrained Ensemble

Paper · 2022 · Abdelrahim Qaddoumi

Summary

The paper describes team 259's entry to Subtask 2 of NADI 2022, which is three-way sentiment classification (positive, negative, neutral) of Arabic tweets that mix Modern Standard Arabic with regional dialects. The author did no fine-tuning and no preprocessing. Instead, he took five sentiment classifiers already published on HuggingFace: four CAMeLBERT variants, each pretrained on a different kind of Arabic (MSA, DA, CA, Mix), and an Arabic-MARBERT-Sentiment model. To combine them, he added their output logits and took the highest class. He tried every combination of models, choosing the best one on the 1,500-tweet training split. The pair CAMeLBERT Mix plus CAMeLBERT MSA won. It scored 72.49% macro F1 over the positive and negative classes on the development set and about 69% on the test set, which placed the team 3rd. The work shows that ready-made models with complementary pretraining data can be ensembled into a competitive system with almost no compute and no labelled training.

Key points

  • The NADI 2022 Subtask 2 data has 5,000 tweets: 1,500 train, 500 dev and 3,000 test. The training split is 581 pos, 579 neg and 340 neut.
  • The scoring metric is Macro-F1-PN, a macro F1 computed only over the positive and negative classes.
  • Single-model dev F1-PN was 70.18% for CAMeLBERT MSA, 69.70% for Mix, 67.07% for DA and 61.92% for CA.
  • The ensembles reached 72.49% (Mix_MSA, the best), 71.94% (Mix_CA_MSA), 71.63% (DA_MSA) and 71.36% (all four CAMeLBERT models). Mix_MSA scored 69.86% on test.
  • The CAMeLBERT pretraining corpora differ greatly in size: CA has about 0.85B words, while Mix has 17.3B words. Even so, the CA model still produced a non-trivial score.

Interesting ideas

  • I1. Several frozen, off-the-shelf classifiers are combined by summing their raw logits, and an exhaustive search over model subsets picks the best ensemble on held-out data; this gives a strong system with zero training.
  • I2. The winning pair joins a model trained on the standard written variety with one trained on mixed varieties, so the ensemble covers the mixture actually present in the target text.
  • I3. Adding more models did not help: the four-model ensemble scored below the two-model one, so careful selection beats piling up components.
  • I4. The labelled training split is used only to choose a configuration, not to update any weights, which keeps the pipeline cheap and transparent.
  • I5. The metric ignores the neutral class and averages F1 over the polar labels only, so it focuses on the decisions that matter most.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when merging witness parsers into silver data, we could score every subset of witnesses on a gold slice and keep the best-agreeing combination rather than always using all of them.
  • I2 (tree): when training or ensembling gparse, we could pair a model trained on formal-register treebanks with one trained on web/informal treebanks to cover mixed-register input.
  • I3 (learn): this supports keeping evaluation-driven pruning of witnesses and teacher tiers rather than assuming more voters always improve silver labels.
  • I4 (learn): small gold sets could be reserved purely for selecting rules or model combinations, which makes it easy to explain why a configuration was chosen.
  • I5 (frames): for irony or speech-act detection, we could report F1 only over the marked categories so that a large "plain" class does not inflate scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States219 works · 103,487 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksSemitic languages · 3,066 works

Cite

Cite as Abdelrahim Qaddoumi. 2022. Arabic Sentiment Analysis by Pretrained Ensemble. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.47

@inproceedings{qaddoumi2022arabic,
  title = {{Arabic Sentiment Analysis by Pretrained Ensemble}},
  author = {Abdelrahim Qaddoumi},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.47},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.47}
}

References

19 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Mona Diab, and Mohammed Korayem. 2011. Subjectivity and sentiment analysis of modern standard arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 587–591.
  2. Mahmoud Al-Ayyoub, Abed Allah Khamaiseh, Yaser Jararweh, and Mohammed N Al-Kabi. 2019. A comprehensive survey of arabic sentiment analysis. Information processing & management, 56(2):320–342. find
  3. Emad Al-Saidat and Islam Al-Momani. 2010. Future markers in modern standard arabic and jordanian arabic: a contrastive study. European journal of social sciences, 12(3):397–408. find
  4. Basma Alharbi, Hind Alamro, Manal Alshehri, Zuhair Khayyat, Manal Kalkatawi, Inji Ibrahim Jaber, and Xiangliang Zhang. 2020. Asad: A twitter-based benchmark arabic sentiment analysis dataset. arXiv preprint arXiv:2011.00578. ↗
  5. Chien Chin Chen and You-De Tseng. 2011. Quality evaluation of product reviews using an information quality framework. Decision Support Systems, 50(4):755–768. find
  6. Ibrahim Abu Farha and Walid Magdy. 2021. A comparative study of effective approaches for arabic sentiment analysis. Information Processing & Management, 58(2):102438. ↗
  7. Muhammad Abdul-Mageed, Mona Diab, and Sandra Kübler. 2014. Samar: Subjectivity and sentiment analysis for arabic social media. Computer Speech & Language, 28(1):20–37. find
  8. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678. ↗
  9. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020a. Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  10. Jamal Laoudi, Claire Bonial, Lucia Donatelli, Stephen Tratz, and Clare Voss. 2018. Towards a computational lexicon for moroccan darija: Words, idioms, and constructions. In Proceedings of the Joint Workshop on Linguistic Annotation, Multiword Expressions and Constructions (LAW-MWE-CxG-2018), pages 74–85.
  11. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020b. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics.
  12. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. NADI 2021: The second nuanced Arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  13. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The Third Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Seven Arabic Natural Language Processing Workshop (WANLP 2022).
  14. Nawaf A Abdulla, Nizar A Ahmed, Mohammed A Shehab, and Mahmoud Al-Ayyoub. 2013. Arabic sentiment analysis: Lexicon-based and corpus-based. In Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1–167. find
  15. Alejandro Moreo, M Romero, JL Castro, and Jose Manuel Zurita. 2012. Lexicon-based comments-oriented news sentiment analyzer system. Expert Systems with Applications, 39(10):9166–9180. find
  16. Manish Munikar, Sushil Shakya, and Aakash Shrestha. 2019. Fine-grained sentiment classification using bert. In 2019 Artificial Intelligence for Transforming Business and Society (AITB), volume 1, pages 1–5. IEEE. find
  17. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  18. Alena Neviarouskaya, Helmut Prendinger, and Mitsuru Ishizuka. 2010. Recognition of affect, judgment, and appreciation in text. In Proceedings of the 23rd international conference on computational linguistics (Coling 2010), pages 806–814. find
  19. Anshul Wadhawan. 2021. Arabert and farasa segmentation based approach for sarcasm and sentiment detection in arabic tweets. arXiv preprint arXiv:2103.01679. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qaddoumi-2022-arabic-sentiment-analysis-pretrained-ensemble/