Elamur Research⌕
☰
home / papers / 2021 / Overview of the Arabic Sentiment Analysis 2021 Competition at KAUST

Overview of the Arabic Sentiment Analysis 2021 Competition at KAUST

Summary

This paper reports on a Kaggle-hosted shared task run by KAUST. Participants had to sort Arabic tweets into positive, negative or neutral. The data came from ASAD, a corpus of 100K sentiment-labelled tweets from 2012–2020. It covers Khaleeji, Hijazi, Egyptian and Modern Standard Arabic and was annotated by the company Lucidya.

Teams trained on 55K tweets and were ranked publicly on a 20K split (TEST1). Finalists then had to hand over their code. The organisers re-checked TEST1 with it and scored a further hidden 20K split (TEST2) by Macro-F1. In total, 74 teams sent 1,247 submissions.

Every leading system fine-tuned MARBERT, an Arabic language model that includes dialectal text, after standard tweet cleaning. The winner reached 0.79249 Macro-F1 on TEST2, well above the 0.68 AraBERT baseline. Two teams tied for second at about 0.7904.

The work matters for two reasons. It releases what the authors call the largest Arabic tweet sentiment set so far. It also records which practical tricks improved scores on imbalanced, dialect-heavy social media text.

Key points

  • Splits: TRAINING has 55,000 tweets, TEST1 (public leaderboard) 20,000, and TEST2 (private) 20,000. All splits share a skewed class balance of about 16% positive, 16% negative and 68% neutral.
  • Participation:
    • 764 registered users came from 45 countries, led by Saudi Arabia, Algeria and Egypt.
    • 81% of the 74 teams were solo.
    • Only six of the top ten teams submitted code for final checking.
  • Final TEST2 results (Macro-F1):
    • Wissam Antoun placed first with 0.79249.
    • Abdullah I. Alharbi and Ali Salhi shared second with about 0.79039.
    • CS-UM6P placed third with 0.78961.
  • All top systems relied on MARBERT plus preprocessing, for example stripping out links, hashtags and user identifiers, along with characters from other scripts and runs of duplicated letters. The prize pool was 17,000 USD.
  • ASAD is roughly ten times the size of the SemEval 2017 Task 4 Arabic set (about 9.5K tweets). It is about six times the size of ArcSarcasm-v2 from WANLP 2021 (about 15.5K tweets). The tweet IDs and labels are public, and a portal returns tweet text from the IDs.

Interesting ideas

  • I1. Ali Salhi grew the training set with self-training. He retrained the model N times with different learning rates and adopted a test tweet's label only when all N runs gave the same answer, so instability across runs served as a cheap filter for noisy pseudo-labels. — For Mova (learn): The same rule can gate silver treebank data: keep a sentence's tree only if several runs or witness parsers trained with different settings produce identical arcs and labels.
  • I2. The same team appended each word's stem to the tweet as extra input, giving the model both the surface form and a normalised version to cope with rich morphology and spelling variation. — For Mova (tree): gparse could receive lemmas from crate en next to word forms as an extra input channel, which may help on inflected or misspelled tokens in noisy text.
  • I3. CS-UM6P added a one-versus-all binary head for each class, each with its own attention layer over the encoder tokens, and summed those binary logits with the multi-class logits at prediction time. — For Mova (frames): Irony or speech-act classification in crate prag could pair a multi-way decision with dedicated per-label detectors, and the separate attention weights would show which tokens drove each label.
  • I4. The organisers ranked teams publicly on one split, then re-ran the finalists' submitted code on a second split nobody had seen, which guards against tuning to the leaderboard and catches results that cannot be reproduced. — For Mova (learn): Keep a sealed test set that is used only for final reporting, and require every reported score to be regenerated from committed code rather than copied from logs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Sentiment & emotion · Evaluation & benchmarks · Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2109.14456 · arXiv 2109.14456

@misc{alamro2021overview,
  title = {{Overview of the Arabic Sentiment Analysis 2021 Competition at KAUST}},
  author = {Alamro, Hind and Alshehri, Manal and Alharbi, Basma and Khayyat, Zuhair and Kalkatawi, Manal and Jaber, Inji Ibrahim and Zhang, Xiangliang},
  year = {2021},
  doi = {10.48550/arxiv.2109.14456},
  eprint = {2109.14456},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2109.14456}
}

References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohd Zeeshan Ansari, M.B. Aziz, M.O. Siddiqui, H. Mehra, and K.P. Singh. Analysis of political sentiment orientations on twitter. Procedia Computer Science, 167:1821–1828, 2020. International Conference on Computational Intelligence and Data Science. find
  2. Nur Vidya, Mohamad Ivan Fanany, and Indra Budi. Twitter sentiment to analyze net brand reputation of mobile phone providers. Procedia Computer Science, 72:519–526, 2015. The Third Information Systems International Conference 2015. find
  3. Sonia Anastasia and Indra Budi. Twitter sentiment analysis of online transportation service providers. In 2016 International Conference on Advanced Computer Science and Information Systems (ICACSIS), pages 359–365, 2016. find
  4. Akshat Bakliwal, Jennifer Foster, Jennifer van der Puil, Ron O’Brien, Lamia Tounsi, and Mark Hughes. Sentiment analysis of political tweets: Towards an accurate classifier. In Proceedings of the Workshop on Language Analysis in Social Media, pages 49–58, Atlanta, Georgia, June 2013. Association for Computational Linguistics. find
  5. Jyoti Ramteke, Samarth Shah, Darshan Godhia, and Aadil Shaikh. Election result prediction using twitter sentiment analysis. In 2016 International Conference on Inventive Computation Technologies (ICICT), volume 1, pages 1–5, 2016. find
  6. Ravikumar Patel and Kalpdrum Passi. Sentiment analysis on twitter data of world cup soccer tournament using machine learning. IoT, 1(2):218–239, 2020. find
  7. Qiang Yang, Hind Alamro, Somayah Albaradei, Adil Salhi, Xiaoting Lv, Changsheng Ma, Manal Alshehri, Inji Jaber, Faroug Tifratene, Wei Wang, Takashi Gojobori, Carlos M. Duarte, Xin Gao, and Xiangliang Zhang. WANLP 2021 (subtask 2) Training Testing Total 2,180 (17%) 575 (19%) 2,755 (18%) 4,621 (37%) 1,677 (56%) 6,298 (41%) 5,747 (46%) 748 (25%) 6,495 (41%) 12,548 (100%) 3,000 (100%) 15,548 (100%) SenWave: Monitoring the global sentiments under the covid-19 pandemic. arXiv preprint arXiv:2006.10842, 2020. ↗
  8. Xiangliang Zhang, Qiang Yang, Somayah Albaradei, Xiaoting Lyu, Hind Alamro, Adil Salhi, Changsheng Ma, Manal Alshehri, I. Jaber, Faroug Tifratene, Wei Wang, T. Gojobori, C. Duarte, and Xin Gao. Rise and fall of the global conversation and shifting sentiments during the covid-19 pandemic. Humanities and Social Sciences Communications, 8:1–10, 2021. find
  9. Ayah Zirikly and Mona Diab. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 176–185, 01 2015. find
  10. Oumaima Oueslati, Erik Cambria, Moez Ben HajHmida, and Habib Ounelli. A review of sentiment analysis research in arabic language. Future Generation Computer Systems, 112:408–430, 2020. find our card
  11. Basma Alharbi, Hind Alamro, Manal Alshehri, Zuhair Khayyat, Manal Kalkatawi, Inji Ibrahim Jaber, and Xiangliang Zhang. ASAD: A twitter-based benchmark arabic sentiment analysis dataset. arXiv preprint arXiv:2011.00578, 2021. ↗ our card
  12. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online, August 2021. Association for Computational Linguistics. find our card
  13. Wissam Antoun, Fady Baly, and Hazem Hajj. AraBERT: Transformer-based model for arabic language understanding, 2020. find our card
  14. Ibrahim Abu Farha and Walid Magdy. From arabic sentiment analysis to sarcasm detection: The arsarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39, 2020. find our card
  15. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online), December 2020. International Committee for Computational Linguistics. find our card
  16. Abdullah I Alharbi, Phillip Smith, and Mark Lee. Enhancing contextualised language models with static character and word embeddings for emotional intensity and sentiment strength detection in arabic tweets. Procedia Computer Science, 189:258–265, 2021. find
  17. Abdullah I Alharbi and Mark Lee. Combining character and word embeddings for affect in arabic informal social media microblogs. In International Conference on Applications of Natural Language to Information Systems, pages 213–224. Springer, 2020. find
  18. Xingyou Wang, Weijie Jiang, and Zhiyong Luo. Combination of convolutional and recurrent neural network for sentiment analysis of short texts. In Proceedings of COLING 2016, the 26th international conference on computational linguistics: Technical papers, pages 2428–2437, 2016. find
  19. Abdelkader El Mahdaouy, Abdellah El Mekki, Kabil Essefar, Nabil El Mamoun, Ismail Berrada, and Ahmed Khoumsi. Deep multi-task model for sarcasm detection and sentiment analysis in arabic language. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 334–339, 2021. find our card
  20. Abdellah El Mekki, Abdelkader El Mahdaouy, Kabil Essefar, Nabil El Mamoun, Ismail Berrada, and Ahmed Khoumsi. Bert-based multi-task model for country and province level msa and dialectal arabic identification. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 271–275, 2021. find
  21. Kabil Essefar, Abdellah El Mekki, Abdelkader El Mahdaouy, Nabil El Mamoun, and Ismail Berrada. Cs-um6p at semeval-2021 task 7: Deep multi-task learning model for detecting and rating humor and offense. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021. find
  22. Sara Rosenthal, Noura Farra, and Preslav Nakov. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pages 502–518, 2017. find our card
  23. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. Overview of the wanlp 2021 shared task on sarcasm and sentiment detection in arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 296–305, 2021. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alamro-2021-overview-arabic-sentiment-analysis-2021/