Elamur Research⌕
☰
home / papers / 2023 / Foul at SemEval-2023 Task 12: MARBERT Language model and lexical filtering for sentiments analysis of tweets in Algerian Arabic

Foul at SemEval-2023 Task 12: MARBERT Language model and lexical filtering for sentiments analysis of tweets in Algerian Arabic

Paper · 2023 · Faiza Belbachir

Summary

This paper presents a system for three-way sentiment classification (positive, negative, neutral) of Algerian Arabic tweets, built for Task A of the AfriSenti shared task (SemEval-2023 Task 12). Algerian dialect is under-resourced. It mixes Arabic, Berber, French, Turkish and Spanish vocabulary, its spelling is inconsistent, and tweets use many emojis. The pipeline has three stages. Light preprocessing removes @user mentions and RT, merges letter variants and deliberately keeps stopwords. A fine-tuned MARBERT model then classifies each tweet. Finally, two small hand-built lexicons, one of dialect opinion words and one of emojis, can relabel tweets that the model called neutral. When evaluated on the train and dev splits, MARBERT achieved better results than both CAMeLBERT and Arabic-BERT. The lexicon step added a small gain of about 0.4 points in F1. The submitted system reached a weighted F1 of 0.70 on the test set and placed 9th of 30 teams. The work shows that a dialect-aware pretrained model combined with cheap, targeted lexical rules is a workable recipe for low-resource varieties.

Key points

  • The data has 1651 training, 414 development and 958 test tweets, mostly positive. The author re-split train and dev into 1858/207 because official validation submissions were limited.
  • On the test set the system scored precision 0.704, recall 0.703 and weighted F1 0.703, ranking 9th of 30.
  • The neutral class is the weakest for every model: MARBERT reaches about 0.51 F1 on it versus roughly 0.78–0.79 on the polar classes. Arabic-BERT and CAMeLBERT trail MARBERT.
  • Weaker baselines included a TF-IDF + SVM classifier, about 2 F1 points lower, and an LSTM with roughly 60% accuracy.
  • Hyperparameters followed Alamro et al.: 5–10 epochs, batch size 16 and a learning rate of about 1.2e-5 to 1.8e-5.

Interesting ideas

  • I1. The lexicon override fires only when the neural model outputs neutral, so the rules correct the model's least certain class without overriding its confident decisions.
  • I2. Lexicon candidates are chosen as rare but strongly opinionated words, on the assumption that low-frequency items are poorly represented by the pretrained model, and are then filtered by hand.
  • I3. Letter variants that dialect writers treat as interchangeable are collapsed into one arbitrary representative per equivalence class before modelling.
  • I4. Stopwords are kept because function words such as negators can flip a tweet's polarity.
  • I5. Emojis are given their own polarity lexicon, separate from words, as a distinct signal channel.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): In irony or speech-act detection, rule-based cues could be applied only when the statistical component returns a default or "no-signal" label, keeping each override explainable.
  • I2 (learn): Silver-data repair could focus on low-frequency lemmas where witness parsers are least reliable, and send only those to manual or LLM-teacher review.
  • I3 (tokens): A similar normalization layer for noisy English spellings, such as informal variants and character repeats, could map them to canonical forms before lemmatization while keeping the original surface form.
  • I4 (tree): This supports never discarding function words before parsing, since negation and modality dependents must stay attached in the UD tree for frame polarity to be correct.
  • I5 (world): Emoji and emoticon tokens could be mapped to speaker affect states in the text's world model, rather than being dropped as noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Institut Polytechnique des Sciences Avancées.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSentiment & emotion · 1,035 worksSemitic languages · 3,066 works

Cite

Cite as Faiza Belbachir. 2023. Foul at SemEval-2023 Task 12: MARBERT Language model and lexical filtering for sentiments analysis of tweets in Algerian Arabic. Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.semeval-1.52

@inproceedings{belbachir2023foul,
  title = {{Foul at SemEval-2023 Task 12: MARBERT Language model and lexical filtering for sentiments analysis of tweets in Algerian Arabic}},
  author = {Faiza Belbachir},
  booktitle = {Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)},
  year = {2023},
  doi = {10.18653/v1/2023.semeval-1.52},
  url = {https://doi.org/10.18653/v1/2023.semeval-1.52}
}

References

32 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed and Mona Diab. 2012. Toward building a large-scale arabic sentiment lexicon. GWC 2012: 6th International Global Wordnet Conference, Proceedings, pages 18–22. find
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Hind Alamro, Manal Alshehri, Basma Alharbi, Zuhair Khayyat, Manal M. Kalkatawi, Inji Ibrahim Jaber, and Xiangliang Zhang. 2021. Overview of the arabic sentiment analysis 2021 competition at kaust. This paper provides an overview of the Arabic Sentiment Analysis Challenge organized by King Abdullah University of Science and Technology (KAUST). The task in this challenge is to develop machine learning models to classify a given tweet into one of the three categories Positive, Negative, or Neutral. From our recently released ASAD dataset, we provide the competitors with 55K tweets for training, and 20K tweets for validation, based on which the performance of participating teams are ranked on a leaderboard, https://www.kaggle.com/c/arabic-sentiment-analysis-2021-kaust. The competition received in total 1247 submissions from 74 teams (99 team members). The final winners are determined by another private set of 20K tweets that have the same distribution as the training and validation set. In this paper, we present the main findings in the competition and summarize the methods and tools used by the top ranked teams. The full dataset of 100K labeled tweets is also released for public usage, at https://www.kaggle.com/c/arabic-sentiment-analysis-2021-kaust/data. ↗
  4. Randah Alharbi, Walid Magdy, Kareem Darwish, Ahmed AbdelAli, and Hamdy Mubarak. 2018. Part-of-speech tagging for Arabic Gulf dialect using Bi-LSTM. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  5. Badr AlKhamissi, Mohamed Gabr, Muhammad El-Nokrashy, and Khaled Essam. 2021. Adapting MAR-BERT for improved Arabic dialect identification: Submission to the NADI 2021 shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 260–264, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find
  6. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. SentiWordNet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA).
  7. Faiza Belbachir and Mohand Boughanem. 2018. Using language models to improve opinion detection. Information Processing and Management, 54:958–968. ↗
  8. Chia-Ping Chen, Tzu-Hsuan Tseng, and Tzu-Hsuan Yang. 2018. Sentiment analysis on social network: Using emoticon characteristics for Twitter polarity classification. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 23, Number 1, June 2018, Taipei, Taiwan. Association for Computational Linguistics and Chinese Language Processing. find
  9. Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 551–561, Austin, Texas. Association for Computational Linguistics.
  10. Nurendra Choudhary, Rajat Singh, Vijjini Anvesh Rao, and Manish Shrivastava. 2018. Twitter corpus of resource-scarce languages for sentiment analysis and multilingual emoji prediction. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1570–1577, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  12. Abdelkader El Mahdaouy, Eric Gaussier, and Saı̈d Ouatik El Alaoui. 2017. Arabic text classification based on word and document embeddings. In Proceedings of the International Conference on Advanced Intelligent Systems and Informatics 2016, pages 32–41, Cham. Springer International Publishing. find
  13. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1615–1625, Copenhagen, Denmark. Association for Computational Linguistics.
  14. Emna Fsih, Sameh Kchaou, Rahma Boujelbane, and Lamia Hadrich-Belguith. 2022. Benchmarking transfer learning approaches for sentiment analysis of Arabic dialect. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 431–435, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. ↗
  15. Imene Guellil and Faiçal Azouaou. 2016. Arabic dialect identification with an unsupervised learning (based on a lexicon). application case: ALGERIAN dialect. In 2016 IEEE Intl Conference on Computational Science and Engineering, CSE 2016, and IEEE Intl Conference on Embedded and Ubiquitous Computing, EUC 2016, and 15th Intl Symposium on Distributed Computing and Applications for Business Engineering, DCABES 2016, Paris, France, August 24-26, 2016, pages 724–731. IEEE Computer Society. find
  16. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’04), pages 168–177, New York, NY, USA. Association for Computing Machinery. find
  17. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  18. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 655–665, Baltimore, Maryland. Association for Computational Linguistics. ↗
  19. M’hamed Mataoui, Omar Zelmati, and Madiha Boumechache. 2016. A proposed lexicon-based sentiment analysis approach for the vernacular algerian arabic. Res. Comput. Sci., 110:55–70. ↗
  20. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. ↗
  21. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 task 1: Affect in tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 1–17, New Orleans, Louisiana. Association for Computational Linguistics.
  22. Saif Mohammad, Mohammad Salameh, and Svetlana Kiritchenko. 2016. Sentiment lexicons for arabic social media. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. European Language Resources Association (ELRA).
  23. Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, Nedjma Ousidhoum, David Ifeoluwa Adelani, Seid Muhie Yimam, Ibrahim Sa’id Ahmad, Meriem Beloucif, Saif M. Mohammad, Sebastian Ruder, Oumaima Hourrane, Pavel Brazdil, Felermino Dário Mário António Ali, Davis David, Salomey Osei, Bello Shehu Bello, Falalu Ibrahim, Tajuddeen Gwadabe, Samuel Rutunda, Tadesse Belay, Wendimu Baye Messelle, Hailu Beshada Balcha, Sisay Adugna Chala, Hagos Tesfahun Gebremichael, Bernard Opoku, and Steven Arthur. 2023a. AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages. ↗
  24. Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Seid Muhie Yimam, David Ifeoluwa Adelani, Ibrahim Sa’id Ahmad, Nedjma Ousidhoum, Abinew Ali Ayele, Saif M. Mohammad, Meriem Beloucif, and Sebastian Ruder. 2023b. SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval). In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023). Association for Computational Linguistics. ↗
  25. Alexander Pak. 2012. Automatic, adaptive, and applicative sentiment analysis. Theses, Université Paris Sud - Paris XI. ↗
  26. Alexander Pak and Patrick Paroubek. 2010. Twitter based system: Using twitter for disambiguating sentiment ambiguous adjectives. In Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval@ACL 2010, Uppsala University, Uppsala, Sweden, July 15-16, 2010, pages 436–439. The Association for Computer Linguistics. find
  27. Eshrag Refaee and Verena Rieser. 2014. Evaluating distant supervision for subjectivity and sentiment analysis on arabic twitter feeds. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Langauge Processing (ANLP), page 174–179. Association for Computational Linguistics.
  28. Houda Saâdane, Hosni Seffih, Christian Fluhr, Khalid Choukri, and Nasredine Semmar. 2018. Automatic identification of maghreb dialects using a dictionary-based approach. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  29. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.
  30. Yudianto Sujana, Jiawen Li, and Hung-Yu Kao. 2020. Rumor detection on Twitter using multiloss hierarchical BiLSTM with an attenuation factor. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 18–26, Suzhou, China. Association for Computational Linguistics. ↗
  31. Samia Touileb and Jeremy Barnes. 2021. The interplay between language similarity and script on a novel multi-layer Algerian dialect corpus. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3700–3712, Online. Association for Computational Linguistics. ↗
  32. Benjamin Weissman. 2022. Emoji semantics/pragmatics: investigating commitment and lying. In Proceedings of the Fifth International Workshop on Emoji Understanding and Applications in Social Media, pages 21–28, Seattle, Washington, USA. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/belbachir-2023-foul-semeval-2023-task-12/