Elamur Research
☰
home / papers / 2021 / Overview of the WANLP 2021 Shared Task on Sarcasm and Sentiment Detection in Arabic

Overview of the WANLP 2021 Shared Task on Sarcasm and Sentiment Detection in Arabic

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.wanlp-1.36.pdf

Summary

This overview reports on a WANLP 2021 shared task built to draw research attention to Arabic sarcasm. Sarcasm has received little study in Arabic, yet it routinely misleads sentiment classifiers. The organisers released ArSarcasm-v2, which extends the earlier ArSarcasm corpus with ironic tweets from DAICT and with randomly streamed Arabic tweets from late 2020. The new portion was labelled through crowdsourcing on appen, following the original guidelines. Every tweet carries three labels: sarcasm, sentiment and dialect. Teams competed in two subtasks. The first was binary sarcasm detection, scored by F1 on the sarcastic class. The second was three-way sentiment classification, scored by macro F1 averaged over the positive and negative classes only. Nearly all teams fine-tuned pre-trained Arabic transformers such as AraBERT and MARBERT, often in ensembles or multi-task setups. The best sarcasm score was only about 0.62 F1, far below the sentiment results. The authors conclude that Arabic sarcasm detection is still largely unsolved and needs more data. The corpus is public and gives a benchmark annotated for dialect.

Key points

  • ArSarcasm-v2 has 15,548 tweets, split into 12,548 for training and 3,000 for testing. About 19.2% of the tweets (2,989) are sarcastic. MSA and Egyptian dominate, while Maghrebi appears in just 45 tweets.
  • Around 5,000 newly added tweets were labelled by at least three annotators of Arab origin, with hidden test questions used for quality control. Agreement was 87.3% for sarcasm, 78.9% for sentiment and 77.0% for dialect.
  • The sarcasm subtask drew 27 submissions and the sentiment subtask 22. Seventeen system description papers followed.
  • BhamNLP won sarcasm detection with an F1-sarcastic of 0.6225. SPPU-AASM placed second (0.614) and DeepBlueAI third (0.6127), the latter using an AraBERT + XLM-R ensemble.
  • CS-UM6P won sentiment analysis with an F1PN of 0.748. Their system was a MARBERT encoder trained with multi-task learning and task-specific attention layers. DeepBlueAI (0.7392) and rematchka (0.7321) placed second and third.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

49 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 48 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{farha2021overview,
  title = {{Overview of the WANLP 2021 Shared Task on Sarcasm and Sentiment Detection in Arabic}},
  author = {Farha, Ibrahim Abu and Zaghouani, Wajdi and Magdy, Walid},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.36/}
}

References

53 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abbasi, Hsinchun Chen, and Arab Salem. 2008. Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. ACM Transactions on Information Systems (TOIS), 26(3):1–34. find
  2. Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. 2020. DAICT: A dialectal Arabic irony corpus extracted from Twitter. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6265–6271, Marseille, France. European Language Resources Association. find
  3. Reem Abdel-Salam. 2021. Wanlp 2021 shared-task: Towards irony and sentiment detection in arabic tweets using multi-headed-lstm-cnn-gru and marbert. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  4. Muhammad Abdul-Mageed. 2019. Modeling arabic subjectivity and sentiment in lexical space. Information Processing & Management, 56(2):291–307. find
  5. Muhammad Abdul-Mageed, Mona T. Diab, and Mohammed Korayem. 2011. Subjectivity and sentiment analysis of modern standard arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2, HLT ’11, pages 587–591, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  6. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. Arbert & marbert: Deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗ our card
  7. N. A. Abdulla, N. A. Ahmed, M. A. Shehab, and M. Al-Ayyoub. 2013. Arabic sentiment analysis: Lexicon-based and corpus-based. In 2013 IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies (AEECT), pages 1–6. find
  8. Gavin Abercrombie and Dirk Hovy. 2016. Putting sarcasm detection into context: The effects of class imbalance and manual labelling on supervised machine classification of twitter conversations. In Proceedings of the ACL 2016 Student Research Workshop, pages 107–113. find
  9. Ibrahim Abu Farha and Walid Magdy. 2019. Mazajak: An online Arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy. Association for Computational Linguistics. find
  10. Ibrahim Abu Farha and Walid Magdy. 2020. From Arabic sentiment analysis to sarcasm detection: The Ar-Sarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39, Marseille, France. European Language Resource Association. find
  11. Ibrahim Abu Farha and Walid Magdy. 2021. Benchmarking transformer-based language models for ara-bicsentiment and sarcasm detection. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  12. Ibrahim Abu Farha and Walid Magdy. 2021. A comparative study of effective approaches for arabic sentiment analysis. Information Processing Management, 58(2):102438. find
  13. Abeer Abuzayed and Hend Al-Khalifa. 2021. Sarcasm and sentiment detection in arabic tweets using bert-based models and data augmentation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  14. Mohammad Al-Smadi, Bashar Talafha, Mahmoud Al-Ayyoub, and Yaser Jararweh. 2019. Using long short-term memory deep neural networks for aspect-based sentiment analysis of Arabic reviews. International Journal of Machine Learning and Cybernetics, 10(8):2163–2175. find
  15. Abdulaziz M. Alayba, Vasile Palade, Matthew England, and Rahat Iqbal. 2018. A combined cnn and lstm model for arabic sentiment analysis. In Machine Learning and Knowledge Extraction, pages 179–191, Cham. Springer International Publishing. find
  16. Laila Bashmal and Daliyah H. Alzeer. 2021. Arsarcasm shared task: An ensemble bert model for sarcasm detection in arabic tweets. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  17. Kareem Darwish, Walid Magdy, et al. 2014. Arabic information retrieval. Foundations and Trends® in Information Retrieval, 7(4):239–342. find
  18. Samhaa R. El-Beltagy, Mona El Kalamawy, and Abu Bakr Soliman. 2017. NileTMRG at SemEval-2017 task 4: Arabic sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 790–795, Vancouver, Canada. Association for Computational Linguistics. find
  19. Abdelkader El Mahdaouy, Abdellah El Mekki, Nabil El Mamoun, Kabil Essefar, Ismail Berrada, and Ahmed Khoumsi. 2021. Deep multi-task model for sarcasm detection and sentiment analysis in arabic language. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  20. Hazem E. Elagbry, Shimaa Attia, Ahmed Abdel-Rahman, Ahmed Abdel-Ate, and Sandra Girgis. 2021. A contextual word embedding for arabic sarcasm detection with random forests. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  21. Abdullah I. Alharbi and Mark Lee. 2021. Multi-task learning using a combination of contextualised and static word embeddings for arabic sarcasm detection and sentiment analysis. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  22. AbdelRahim A Elmadany, Hamdy Mubarak, and Walid Magdy. 2018. Arsas: An arabic speech-act and sentiment corpus of tweets. In OSACT 3: The 3rd Workshop on Open-Source Arabic Corpora and Processing Tools, page 20. find
  23. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mário J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 167–177, Berlin, Germany. Association for Computational Linguistics. find our card
  24. Dalya Faraj and Malak Abdullah. 2021. Sarcasmdet at sarcasm detection task 2021 in arabic using arabert pretrained model. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  25. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. find our card
  26. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Lrec, pages 392–398. Citeseer. find our card
  27. Kamel Gaanoun and Imade Benelallam. 2021. Sarcasm and sentiment detection in arabic language: A hybrid approach combining embeddings and rule-based features. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  28. Francesco Barbieri, Francesco Ronzano, and Horacio Saggion. 2014a. Italian irony detection in twitter: a first approach. In The First Italian Conference on Computational Linguistics CLiC-it, page 28. find
  29. Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. Idat at fire2019: Overview of the track on irony detection in arabic tweets. In Proceedings of the 11th Forum for Information Retrieval Evaluation, pages 10–13. find
  30. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014b. Modelling sarcasm in twitter, a novel approach. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 50–58. find our card
  31. Debanjan Ghosh, Weiwei Guo, and Smaranda Muresan. 2015. Sarcastic or not: Word embeddings to predict the literal or sarcastic meaning of words. In proceedings of the 2015 conference on empirical methods in natural language processing, pages 1003–1012. find
  32. Dhaou Ghoul and Gaël Lejeune. 2021. Sarcasm and sentiment detection in arabic: investigating the interest of character-level features. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  33. Nizar Y. Habash. 2010. Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. find
  34. Ahmed Mourad and Kareem Darwish. 2013. Subjectivity and sentiment analysis of modern standard arabic and arabic microblogs. In Proceedings of the 4th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 55–64. find
  35. Malek Naski, Abir Messaoudi, Hatem Haddad, Moez Ben Haj Hmida, Chayma Fourati, and Aymen Ben Elhaj Mabrouk. 2021. icompass at shared task on sarcasm and sentiment detection in arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  36. Amey Hengle, Atharva Kshirsagar, Shaily Desai, and Manisha Marathe. 2021. Combining context-free and contextualized word representations for arabic sarcasm detection and sentiment identification. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  37. Fatmeh Husain and Ozlem Uzuner. 2021. Leveraging offensive language for sarcasm and sentiment detection in arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  38. Doaa Mohey El-Din Mohamed Hussein. 2018. A survey on sentiment analysis challenges. Journal of King Saud University - Engineering Sciences, 30(4):330 – 338. find
  39. Abraham Israeli, Yotam Nahum, Shai Fine, and Kfir Bar. 2021. The idc system for the sentiment classification and sarcasm detection in arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  40. Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys (CSUR), 50(5):73. find our card
  41. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757–762. find our card
  42. Hamada Nayel, Eslam Amer, Aya Allam, and Hanya Abdallah. 2021. Machine learning-based model for sentiment and sarcasm detection. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  43. Silviu Oprea and Walid Magdy. 2020. isarcasm: A dataset of intended sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1279–1289. find our card
  44. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pages 79–86. Association for Computational Linguistics. find
  45. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pages 97–106. ACM. find
  46. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval ’17, Vancouver, Canada. Association for Computational Linguistics. find our card
  47. Aditya Joshi, Vaibhav Tripathi, Pushpak Bhattacharyya, and Mark Carman. 2016. Harnessing sequence labeling for sarcasm detection in dialogue from tv series ‘friends’. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 146–155. find
  48. Bingyan Song, Chunguang Pan, Wang Shengguang, and Luo Zhipeng. 2021. Deepblueai at wanlp-eacl2021 task 2: A deep ensemble-based method for sarcasm and sentiment detection in arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  49. M. Khalifa and Noura Hussein. 2019. Ensemble learning for irony detection in arabic tweets. In FIRE. find
  50. Anshul Wadhawan. 2021. Arabert and farasa segmentation based approach for sarcasm and sentiment detection in arabic tweets. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  51. Svetlana Kiritchenko, Saif Mohammad, and Mohammad Salameh. 2016. Semeval-2016 task 7: Determining sentiment intensity of english and arabic phrases. In Proceedings of the 10th international workshop on semantic evaluation (SEMEVAL-2016), pages 42–51. find
  52. Mohamed Lichouri, Mourad Abbas, Besma Benaziz, Aicha Zitouni, and Khaled Lounnas. 2021. Preprocessing solutions for detection of sarcasm and sentiment for arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find
  53. Deirdre Wilson. 2006. The pragmatics of verbal irony: Echo or pretence? Lingua, 116(10):1722. Annotation Guidelines find