Elamur Research⌕
☰
home / papers / 2022 / JCT at SemEval-2022 Task 6-A: Sarcasm Detection in Tweets Written in English and Arabic using Preprocessing Methods and Word N-grams

JCT at SemEval-2022 Task 6-A: Sarcasm Detection in Tweets Written in English and Arabic using Preprocessing Methods and Word N-grams

Summary

The paper reports the JCT team's systems for SemEval-2022 Task 6-A (iSarcasmEval), a binary task that labels tweets in English and Arabic as sarcastic or not. The authors use a classic, shallow pipeline: Tf-idf features over word n-grams (mostly unigrams) fed to four scikit-learn classifiers: Random Forest, SVC, Logistic Regression and Decision Tree. They combine these with several text-cleaning options, six for English and three for Arabic. Because sarcastic tweets are the minority class, they also try three oversampling techniques: SMOTE, BorderlineSMOTE and ADASYN.

Model selection relied on internal 70-30 and 80-20 splits. The best English submission was an SVC with SMOTE and stop-word removal. The best Arabic submission was an SVC with no balancing and with letter elongation removed.

Internal F1 on the sarcastic class (0.418 for English, 0.679 for Arabic) dropped sharply on the official test sets (0.215 and 0.295). The authors attribute this to a different class ratio and different content in the test data. The paper is mainly useful as a cautionary baseline: surface lexical features generalize poorly for intended sarcasm.

Key points

  • Data sizes:
    • English training set: 3,467 tweets, about 25% sarcastic.
    • Arabic training set: 3,847 tweets, about 39% sarcastic.
    • Both test sets: 1,400 tweets each, only 200 sarcastic (about 14%). The test sets are more skewed than the training data.
  • English preprocessing: lowercasing, and removal of stop words, numbers, emojis, HTML tags and punctuation.
  • Arabic preprocessing: removal of punctuation, Tashkeel (diacritics) and longation (elongation characters).
  • Arabic findings: removing longation helped consistently. The best internal score was 0.698 (ADASYN + LR). For English, no single cleaning step clearly dominated.
  • Submitted results: English fell from 0.418 internal to 0.215 test, and Arabic from 0.679 to 0.295. The ranking is reported roughly as places 33–35. The result tables label the metric "PCL class", which appears to be a leftover from another task.

Interesting ideas

  • I1. Treat preprocessing as a hyperparameter and search over combinations of cleaning steps, rather than fixing a pipeline in advance, since which step helps depends on language and data. — For Mova (tokens): Normalization switches in crate en (case, elongated words like "soooo", emoji handling) could be ablated per corpus to confirm which ones actually help tagging and lemmas.
  • I2. Script-specific normalization, such as stripping Arabic elongation characters, removes stylistic stretching that fragments the vocabulary. — For Mova (tokens): Mapping expressive lengthening and repeated letters in English to a canonical lemma, while keeping a "stretched" flag, would preserve the emphasis signal without inflating sparsity.
  • I3. The large drop from internal to official scores is explained by a class-ratio shift between training and test data. This is a reminder that validation splits should mimic the deployment prior, not just the training prior. — For Mova (learn): Evaluations could include test slices with deliberately shifted label priors as a negative control, exposing gains that only reflect the training distribution.
  • I4. Bag-of-n-grams models with oversampling stay weak on intended sarcasm, which suggests the needed signal lies in context and in the gap between what is said and what is meant. — For Mova (frames): Irony detection in crate prag could compare the literal frame's evaluative polarity against the expected state of the world, rather than relying on lexical cues.
  • I5. As future work, the authors propose acronym disambiguation and character skip-grams to cope with noisy, sparse tweet text. — For Mova (tokens): An acronym-expansion step before tagging would give the parser and frame builder real words to work with on social-media input.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.145

@inproceedings{hacohenkerner2022jct,
  title = {{JCT at SemEval-2022 Task 6-A: Sarcasm Detection in Tweets Written in English and Arabic using Preprocessing Methods and Word N-grams}},
  author = {HaCohen-Kerner, Yaakov and Fchima, Matan and Meyrowitsch, Ilan},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.145},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.145}
}

References

41 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In LREC. ELRA. ↗ our card
  2. Leo Breiman. 2001. learning 45(1), 5-32. Random forests. find
  3. Machine Jason Brownlee. 2020. Imbalanced classification with python: Better metrics, balance skewed classes, cost-sensitive learning. Machine Learning Mastery. find
  4. Chih-Chung Chang, and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM transactions on intelligent systems and technology (TIST) 2(3), 1-27. find
  5. Nitesh V. Chawla, Kevin W Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2002. SMOTE: synthetic minority oversampling technique. Journal of artificial intelligence research, 16, 321-357. find
  6. John D. Campbell and Albert N. Katz. 2012. Are there necessary conditions for inducing a sense of sarcastic irony? Discourse Processes, 49(6), 459– 480. ↗
  7. Corinna Cortes and Vladimir Vapnik. 1995.Support-vector networks. Machine learning 20.3 : 273-297. find
  8. David R. Cox. 1958. The regression analysis of binary sequences, Journal of the Royal Statistical Society: Series B (Methodological), 20, 215–232. find
  9. Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. 2020. DAICT: A dialectal Arabic irony corpus extracted from Twitter. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6265–6271, Marseille, France. European Language Resources Association. find our card
  10. Bilal Ghanem, Jihen Karoui, F. Benamara, Veronique Moriceau, and P. Rosso. 2019. Idat at fire2019: Overview of the track on irony detection in Arabic tweets. Proceedings of the 11th Forum for Information Retrieval Evaluation. find
  11. Gavin Abercrombie and Dirk Hovy. 2016. Putting Sarcasm Detection into Context: The Effects of Class Imbalance and Manual Labeling on Supervised Machine Classification of Twitter Conversations. In Proceedings of the ACL 2016 Student Research Workshop, pages 107–113. ACL. find
  12. Yaakov HaCohen-Kerner, Ariel Kass, and Ariel Peretz. 2008. Combined one sense disambiguation of abbreviations. In Proceedings of ACL-08: HLT, Short Papers, Association for Computational Linguistics, pages 61-64, Columbus, Ohio, Association for Computational Linguistics. URL: https://aclanthology.org/P08-2. ↗
  13. Ibrahim Abu Farha, Silviu Oprea, Steve Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. In the 16th International Workshop on Semantic Evaluation (SemEval-2022), Association for Computational Linguistics. find our card
  14. Yaakov HaCohen-Kerner, Ariel Kass, and Ariel Peretz. 2010A. HAADS: A Hebrew Aramaic abbreviation disambiguation system. Journal of the American Society for Information Science and Technology, 61(9), 1923-1932. find
  15. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mario J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In CoNLL, pages 167–177. ACL. Yaakov HaCohen-Kerner, Hananya Beck, Elchai Yehudai, and Dror Mughaz. 2010B. Stylistic feature sets as classifiers of documents according to their historical period and ethnic origin. Applied Artificial Intelligence, 24(9), 847-862. find our card
  16. David Bamman and Noah A. Smith. 2015a. Contextualized sarcasm detection on twitter. In ICWSM, pages 574–577. AAAI Press. find
  17. Yaakov HaCohen-Kerner, Ziv Ido, and Ronen Ya’akobov. 2017. Stance classification of tweets using skip char Ngrams. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 266-278). Springer, Cham. imbalanced text dataset using deep neural language models. Applied Sciences, 11(2), 869. find
  18. Samer Muthana Sarsam, Hosam Al-Samarraie, Ahmed Ibrahim Alzahrani, and Bianca Wright. 2020. Sarcasm detection using machine learning algorithms in Twitter: A systematic review. International Journal of Market Research, 62(5), 578-598. find
  19. Yaakov HaCohen-Kerner, Yair Yigal, and Daniel Miller. 2019. The impact of Preprocessing on Classification of Mental Disorders, in Proc. of the 19th Industrial Conference on Data Mining, (ICDM 2019), New York. find
  20. Yaakov HaCohen-Kerner, Daniel Miller, and Yair Yigal. 2020. The influence of preprocessing on text classification using a bag-of-words representation, PloS one, vol. 15, p. e0232525. find
  21. Silviu Oprea and Walid Magdy, 2019. isarcasm: A dataset of intended sarcasm. arXiv preprint arXiv:1911.03123. ↗ our card
  22. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. Cascade: Contextual sarcasm detection in online discussion forums. In COLING, pages 1837–1848. ACL. find our card
  23. Haibo He and Yunqian Ma (Eds.). 2013. Imbalanced learning: foundations, algorithms, and applications. find
  24. David W. Hosmer, Stanley Lemeshow, and Rodney X. Sturdivant. 2013. Applied logistic regression, Vol. 398, John Wiley & Sons. find
  25. Silviu Oprea and Walid Magdy. 2020. The effect of sociocultural variables on sarcasm communication online. Proceedings of The 23rd ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW). find
  26. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In COLING, pages 213–223. ACL. find our card
  27. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the eighth ACM international conference on web search and data mining WSDM, pages 97–106. ACM. find
  28. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In IJCNLP, pages 757–762. ACL. find our card
  29. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys (CSUR), 50(5), 1-22. find our card
  30. Jihen Karoui, Farah Banamara Zitoune, and Veronique Moriceau. 2017. SOUKHRIA: Towards an irony detection system for Arabic in social media. Procedia Computer Science, 117:161–168. ↗
  31. Arabic Computational Linguistics Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2018. A large self-annotated corpus for sarcasm. In LREC. ELRA. find our card
  32. Byron C. Wallace, Choe, Do Kook Choe, and Eugene Charniak. 2015. Sparse, contextually informed models for irony detection: Exploiting user communities, entities and sentiment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1035–1044. ACL. find
  33. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcas as contrast between a positive sentiment and negative situation. In EMNLP, pages 704–714. ACL. find
  34. Bartosz Krawczyk. 2016. Learning from imbalanced data: open challenges and future directions. Progress in Artificial Intelligence, 5(4), 221-232. find
  35. Sara Rosenthal, Preslav Nakov, Alan Ritter, and Veselin Stoyanova. 2014. Semeval-2014 task 9: Sentiment analysis in twitter. In Proceedings of the International Workshop on Semantic Evaluation (SemEval-2014), Dublin, Ireland, August. find our card
  36. Diana G. Maynard and Mark A. Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Lrec 2014 proceedings. ELRA. find our card
  37. Yan-yan Song and Ying Lu. 2015. Decision tree methods: applications for classification and prediction. Shanghai archives of psychiatry, 27(2), 130. find
  38. Bleau Moores and Vijay Mago. 2022. A Survey on Automated Sarcasm Detection on Twitter. arXiv preprint arXiv:2202.02516. ↗
  39. Tin Kam Ho. 1995. Random decision forests. Proceedings of 3rd international conference on document analysis and recognition. Vol. 1. IEEE. find
  40. Sarang Shaikh, Sher Muhammad Daudpota, Ali Shariq Imran, and Zenun Kastrati. 2021. Towards improved classification accuracy on highly Palak Verma, Neha Shukla, and A. P. Shukla. 2021. Techniques of Sarcasm Detection: A Review. find
  41. In 2021 International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE) (pp. 968-972). IEEE. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kerner-2022-jct-semeval-2022-task-6/