Elamur Research
☰
home / papers / 2020 / From Arabic Sentiment Analysis to Sarcasm Detection: The ArSarcasm Dataset

From Arabic Sentiment Analysis to Sarcasm Detection: The ArSarcasm Dataset

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.osact-1.5.pdf

Summary

Sarcasm undermines sentiment analysis because a speaker's real, usually negative, attitude is hidden behind positive-sounding wording. Arabic had no usable public sarcasm corpus: earlier collections were never released or have largely vanished from Twitter. The authors fill this gap with ArSarcasm, built by relabelling tweets from existing Arabic sentiment datasets on the Figure-Eight crowdsourcing platform. Each tweet received three labels: sarcastic or not, a three-way sentiment label following SemEval guidelines, and a dialect tag. Comparing the new sentiment labels with the original ones shows strong annotator subjectivity, with labels shifting according to individual biases. The state-of-the-art analyser Mazajak performs clearly worse on sarcastic tweets, which suggests that such tools rely on surface polarity. A BiLSTM baseline reaches only 0.46 F1 on the sarcastic class. ArSarcasm matters as a freely downloadable Arabic sarcasm resource with dialect annotation. It also shows that detecting sarcasm requires context, world knowledge and cultural background beyond the text itself.

Key points

  • ArSarcasm has 10,547 tweets, of which 1,682 (about 16%) are sarcastic. It is released for research at github.com/iabufarha/ArSarcasm.
  • The labels cover sarcasm (binary), sentiment (positive/negative/neutral) and dialect (Egyptian, Gulf, Levantine, Maghrebi, MSA).
  • Mazajak scored F1 0.43/0.44 on sarcastic tweets (new/original sentiment labels). On non-sarcastic tweets it scored 0.64/0.61.
  • The baseline is a BiLSTM (128 cells) with a dense layer, using Mazajak embeddings, the Adam optimiser and an 80/20 train/test split.
  • Baseline results on the sarcastic class were precision 0.62, recall 0.38 and F1 0.46. The low recall shows that much sarcasm goes undetected.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

45 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 42 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{farha2020arabic,
  title = {{From Arabic Sentiment Analysis to Sarcasm Detection: The ArSarcasm Dataset}},
  author = {Farha, Ibrahim Abu and Magdy, Walid},
  booktitle = {Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection},
  year = {2020},
  url = {https://aclanthology.org/2020.osact-1.5/}
}

References

45 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abbasi, A., Chen, H., and Salem, A. (2008). Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. ACM Transactions on Information Systems (TOIS), 26(3):1–34. find
  2. Abbes, I., Zaghouani, W., and El-Hardlo, O. (2020). Daict: A dialectal arabic irony corpus extracted from twitter. LREC 2020. find
  3. Abdul-Mageed, M., Diab, M. T., and Korayem, M. (2011). Subjectivity and sentiment analysis of modern standard arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2, HLT ’11, pages 587–591, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  4. Abercrombie, G. and Hovy, D. (2016). Putting sarcasm detection into context: The effects of class imbalance and manual labelling on supervised machine classification of twitter conversations. In Proceedings of the ACL 2016 Student Research Workshop, pages 107–113. find
  5. Abu Farha, I. and Magdy, W. (2019). Mazajak: An online Arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy, August. Association for Computational Linguistics. find
  6. Aly, M. and Atiya, A. (2013). Labr: A large scale arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498. find
  7. Amir, S., Wallace, B. C., Lyu, H., Carvalho, P., and Silva, M. J. (2016). Modelling context with user embeddings for sarcasm detection in social media. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 167–177, Berlin, Germany, August. Association for Computational Linguistics. find our card
  8. Badaro, G., Baly, R., Hajj, H., Habash, N., and El-Hajj, W. (2014). A large scale arabic sentiment lexicon for arabic opinion mining. In Proceedings of the EMNLP 2014 workshop on arabic natural language processing (ANLP), pages 165–173. find
  9. Bamman, D. and Smith, N. A. (2015). Contextualized sarcasm detection on twitter. In Ninth International AAAI Conference on Web and Social Media. find
  10. Barbieri, F., Ronzano, F., and Saggion, H. (2014a). Italian irony detection in twitter: a first approach. In The First Italian Conference on Computational Linguistics CLiC-it, page 28. find
  11. Barbieri, F., Saggion, H., and Ronzano, F. (2014b). Modelling sarcasm in twitter, a novel approach. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 50–58. find our card
  12. Bouazizi, M. and Ohtsuki, T. O. (2016). A pattern-based approach for sarcasm detection on twitter. IEEE Access, 4:5477–5488. find
  13. Darwish, K., Magdy, W., et al. (2014). Arabic information retrieval. Foundations and Trends R in Information Retrieval, 7(4):239–342. find
  14. Davidov, D., Tsur, O., and Rappoport, A. (2010). Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the fourteenth conference on computational natural language learning, pages 107– 116. Association for Computational Linguistics. find our card
  15. El-Beltagy, S. R. (2016). Nileulex: A phrase and word level sentiment lexicon for egyptian and modern standard arabic. In LREC. find
  16. Elmadany, A. A., Mubarak, H., and Magdy, W. (2018). Arsas: An arabic speech-act and sentiment corpus of tweets. In OSACT 3: The 3rd Workshop on Open-Source Arabic Corpora and Processing Tools, page 20. find
  17. Filatova, E. (2012). Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Lrec, pages 392– 398. Citeseer. find our card
  18. Ghanem, B., Karoui, J., Benamara, F., Moriceau, V., and Rosso, P. (2019). Idat at fire2019: Overview of the track on irony detection in arabic tweets. In Proceedings of the 11th Forum for Information Retrieval Evaluation, pages 10–13. find
  19. Ghosh, D., Guo, W., and Muresan, S. (2015). Sarcastic or not: Word embeddings to predict the literal or sarcastic meaning of words. In proceedings of the 2015 conference on empirical methods in natural language processing, pages 1003–1012. find
  20. Gibbs Jr, R. W., Gibbs, R. W., and Gibbs, J. (1994). The poetics of mind: Figurative thought, language, and understanding. Cambridge University Press. find
  21. Grice, H. P., Cole, P., and Morgan, J. L. (1975). Syntax and semantics. find
  22. Habash, N. Y. (2010). Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. find
  23. Hussein, D. M. E.-D. M. (2018). A survey on sentiment analysis challenges. Journal of King Saud University - Engineering Sciences, 30(4):330 – 338. find
  24. Ibrahim, H. S., Abdou, S. M., and Gheith, M. (2015). Mika: A tagged corpus for modern standard arabic and colloquial sentiment analysis. In 2015 IEEE 2nd International Conference on Recent Trends in Information Systems (ReTIS), pages 353–358. IEEE. find
  25. Joshi, A., Sharma, V., and Bhattacharyya, P. (2015). Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757–762. find our card
  26. Joshi, A., Tripathi, V., Bhattacharyya, P., and Carman, M. (2016). Harnessing sequence labeling for sarcasm detection in dialogue from tv series ‘friends’. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 146–155. find
  27. Joshi, A., Bhattacharyya, P., and Carman, M. J. (2017). Automatic sarcasm detection: A survey. ACM Computing Surveys (CSUR), 50(5):73. find our card
  28. Justo, R., Corcoran, T., Lukin, S. M., Walker, M., and Torres, M. I. (2014). Extracting relevant knowledge for the detection of sarcasm and nastiness in the social web. Knowledge-Based Systems, 69:124–133. find
  29. Karoui, J., Zitoune, F. B., and Moriceau, V. (2017). Soukhria: Towards an irony detection system for arabic in social media. Procedia Computer Science, 117:161– 168. find
  30. Khodak, M., Saunshi, N., and Vodrahalli, K. (2018). A large self-annotated corpus for sarcasm. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA). find our card
  31. Kiritchenko, S., Mohammad, S., and Salameh, M. (2016). Semeval-2016 task 7: Determining sentiment intensity of english and arabic phrases. In Proceedings of the 10th international workshop on semantic evaluation (SEMEVAL-2016), pages 42–51. find
  32. Liu, B. (2012). Sentiment Analysis and Opinion Mining. Synthesis Lectures on Human Language Technologies, 5(1):1–167. find
  33. Mahyoub, F. H., Siddiqui, M. A., and Dahab, M. Y. (2014). Building an arabic sentiment lexicon using semi-supervised learning. Journal of King Saud University - Computer and Information Sciences, 26(4):417 – 424. Special Issue on Arabic NLP. find
  34. Mourad, A. and Darwish, K. (2013). Subjectivity and sentiment analysis of modern standard arabic and arabic microblogs. In Proceedings of the 4th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 55–64. find
  35. Nabil, M., Aly, M., and Atiya, A. (2015). Astd: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515–2519. find
  36. Oprea, S. and Magdy, W. (2019a). Exploring author context for detecting intended vs perceived sarcasm. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2854–2859, Florence, Italy, July. Association for Computational Linguistics. find
  37. Oprea, S. and Magdy, W. (2019b). isarcasm: A dataset of intended sarcasm. arXiv preprint arXiv:1911.03123. ↗ our card
  38. Pang, B., Lee, L., and Vaithyanathan, S. (2002). Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pages 79–86. Association for Computational Linguistics. find
  39. Ptáček, T., Habernal, I., and Hong, J. (2014). Sarcasm detection on czech and english twitter. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 213–223. find our card
  40. Rajadesingan, A., Zafarani, R., and Liu, H. (2015). Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pages 97– 106. ACM. find
  41. Refaee, E. and Rieser, V. (2014). An arabic twitter corpus for subjectivity and sentiment analysis. In LREC, pages 2268–2273. find
  42. Riloff, E., Qadir, A., Surve, P., De Silva, L., Gilbert, N., and Huang, R. (2013). Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714. find our card
  43. Rosenthal, S., Farra, N., and Nakov, P. (2017). SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval ’17, Vancouver, Canada, August. Association for Computational Linguistics. find our card
  44. Van Hee, C., Lefever, E., and Hoste, V. (2018). Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39–50. find our card
  45. Wilson, D. (2006). The pragmatics of verbal irony: Echo or pretence? Lingua, 116(10):1722. find