Elamur Research⌕
☰
home / papers / 2022 / Amrita_CEN at SemEval-2022 Task 6: A Machine Learning Approach for Detecting Intended Sarcasm using Oversampling

Amrita_CEN at SemEval-2022 Task 6: A Machine Learning Approach for Detecting Intended Sarcasm using Oversampling

Paper · 2022 · Aparna K Ajayan, Krishna Mohanan, Anugraha S et al.

Summary

This paper presents the Amrita_CEN entry to SemEval-2022 Task 6 (iSarcasmEval). The task asks systems to detect sarcasm that the tweet's author intended, in English and Arabic (Subtask A). In English only, systems must also assign tweets to six ironic speech categories (Subtask B). The team used a deliberately classical pipeline:

  • NLTK preprocessing: tokenization, lemmatization and stopword removal, with a public stopword list for Arabic.
  • TF-IDF vectors built with scikit-learn.
  • SMOTE oversampling to counter the shortage of sarcastic examples.
  • A range of standard classifiers (KNN, SVM, Naïve Bayes, Logistic Regression, Decision Tree, Random Forest), with hyperparameters searched for SVM and Random Forest.

Subtask B was handled with one classifier per label. The best models were Random Forest for English, SVM for Arabic and Logistic Regression for the category task. Results were modest, especially on the sarcastic class and on the rare irony categories. The main value of the paper is as a reproducible lexical baseline. It shows how far surface features plus rebalancing get on author-labelled sarcasm, which is not very far.

Key points

  • Data. The English set has 3,467 tweets (vocabulary 5,509, maximum length 72 tokens). The Arabic set has 2,601 tweets (vocabulary 16,226). Each tweet carries an author-provided non-sarcastic "rephrase" field, and Arabic tweets also have a dialect field.
  • English Subtask A. Random Forest reached F1 0.4966 and accuracy 56.71%, ranking 23rd. F1 on the sarcastic class alone was only 0.3052.
  • Arabic Subtask A. SVM reached F1 0.6127 and accuracy 79.21%, ranking 20th. Sarcastic-class F1 was just 0.3490.
  • Subtask B. Logistic Regression in a multi-output, one-model-per-label setup scored macro F1 0.0567 and ranked 14th. Per-category F1 ranged from 0.2180 for sarcasm down to 0.0074 for understatement.
  • Tuning grid. For SVM, gamma and C were each tried at 0.1, 1, 10 and 100, with RBF, linear and polynomial kernels. For Random Forest, n_estimators was tried at 10, 100 and 1000, with max_features set to sqrt or log2.

Interesting ideas

  • I1. The dataset pairs each sarcastic tweet with its author's own literal rewording, which gives a ground-truth "what was really meant" next to "what was said."
  • I2. High accuracy (79%) next to weak sarcastic-class F1 (0.35) shows how a majority-leaning model can look good overall. Reporting the minority-class score separately exposes this.
  • I3. The authors argue that labels from hashtags like #sarcasm are noisy, and that third-party annotators record their own perception rather than the writer's intent. This distinction matters for any label source.
  • I4. Treating the irony categories as independent yes/no decisions allows one text to be both overstatement and rhetorical question.
  • I5. SMOTE creates new minority examples by interpolating between a sample and its nearest same-class neighbours. This can rebalance rare classes without simply duplicating examples.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): These pairs could test the irony module in crate prag, checking whether the frame inferred from the sarcastic text matches the frame parsed from the literal rewording.
  • I2 (learn): Evaluation of rare phenomena like irony or understatement should always report per-class scores, with a majority-class baseline as a negative control.
  • I3 (learn): When building silver data, Mova could tag each label with its source (author, annotator, heuristic, LLM teacher) and treat heuristic tags as lower-trust witnesses.
  • I4 (frames): Speech-act and irony flags in crate prag could be non-exclusive attributes on an utterance rather than a single forced category.
  • I5 (learn): A structured analogue could synthesize extra training cases for rare constructions by combining features of similar attested trees, though each synthetic case would need to stay explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Amrita Vishwa VidyapeethamCoimbatore · India7 works · 31 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Aparna K Ajayan, Krishna Mohanan, Anugraha S, Premjith B and Soman Kp. 2022. Amrita_CEN at SemEval-2022 Task 6: A Machine Learning Approach for Detecting Intended Sarcasm using Oversampling. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.115

@inproceedings{ajayan2022amrita,
  title = {{Amrita\_CEN at SemEval-2022 Task 6: A Machine Learning Approach for Detecting Intended Sarcasm using Oversampling}},
  author = {Aparna K Ajayan, Krishna Mohanan, Anugraha S, Premjith B and Soman Kp},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.115},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.115}
}

References

26 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. Idat at fire2019: Overview of the track on irony detection in arabic tweets. In Proceedings of the 11th Forum for Information Retrieval Evaluation, pages 10–13. ↗
  2. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Lrec, pages 392–398. Citeseer.
  3. Intissar Abbes, Yousra Hallem, and Nadia Taga. 2020. Second-hand shopping and brand loyalty: The role of online collaborative redistribution platforms. Journal of Retailing and consumer Services, 52:101885. ↗
  4. Gavin Abercrombie and Dirk Hovy. 2016. Putting sarcasm detection into context: The effects of class imbalance and manual labelling on supervised machine classification of twitter conversations. In Proceedings of the ACL 2016 student research workshop, pages 107–113.
  5. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSar-Gongde Guo, Hui Wang, David Bell, Yaxin Bi, and Kieran Greer. 2003. Knn model-based approach in classification. In OTM Confederated International Conferences" On the Move to Meaningful Internet Systems", pages 986–996. Springer. find
  6. Barathi Ganesh HB, M Anand Kumar, and KP Soman. 2016. Distributional semantic representation in health care text classification. In FIRE (Working Notes), pages 201–204. find
  7. Yuguang Huang and Lei Li. 2011. Naive bayes classification algorithm based on small sample set. In 2011 IEEE International conference on cloud computing and intelligence systems, pages 34–39. IEEE. find
  8. B Premjith and Soman Kp. 2020. Amrita_cen_nlp@ wosp 3c citation context classification task. In Proceedings of the 8th International Workshop on Mining Scientific Publications, pages 71–74. find
  9. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016. Are word embedding-based features useful for sarcasm detection? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1006–1011, Austin, Texas. Association for Computational Linguistics. ↗
  10. B Premjith, KP Soman, M Anand Kumar, and D Jyothi Ratnam. 2019. Embedding linguistic features in word embedding for preposition sense disambiguation in english—malayalam machine translation context. In Recent Advances in Computational Intelligence, pages 341–370. Springer. ↗
  11. Jihen Karoui, Farah Benamara, Véronique Moriceau, Viviana Patti, Cristina Bosco, and Nathalie Aussenac-Gilles. 2017. Exploring the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study. In 15th Conference of the European Chapter of the Association for Computational Linguistics, volume 1, pages 262–272.
  12. Arpita Khare, Amisha Gangwar, Sudhakar Singh, and Shiv Prakash. 2022. Sentiment analysis and sarcasm detection of indian general election tweets. ↗
  13. Mikhail Khodak, Nikunj Saunshi, Yingyu Liang, Tengyu Ma, Brandon Stewart, and Sanjeev Arora. 2018. A la carte embedding: Cheap but effective induction of semantic feature vectors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12–22, Melbourne, Australia. Association for Computational Linguistics. find
  14. Akshi Kumar, Shubham Dikshit, and Victor Hugo C Albuquerque. 2021. Explainable artificial intelligence for sarcasm detection in dialogues. ↗
  15. Wireless Communications and Mobile Computing, 2021. find
  16. Silviu Oprea and Walid Magdy. 2019. Exploring author context for detecting intended vs perceived sarcasm. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2854– 2859, Florence, Italy. Association for Computational Linguistics. ↗
  17. Silviu Oprea and Walid Magdy. 2020a. iSarcasm: A dataset of intended sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1279–1289, Online. Association for Computational Linguistics.
  18. Anuja Priyam, GR Abhijeeta, Anju Rathee, and Saurabh Srivastava. 2013. Comparative analysis of decision tree classification algorithms. International Journal of current engineering and technology, 3(2):334– 337. find
  19. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In Proceedings of COLING 2014, the 25th international conference on computational linguistics: Technical papers, pages 213–223.
  20. Arifur Rahaman, Ratnadip Kuri, Syful Islam, Md Javed Hossain, and Mohammed Humayin Kabir. 2021. Sarcasm detection in tweets: A feature-based approach using supervised machine learning models. International Journal of Advanced Computer Science and Applications (IJACSA), 12(6). ↗
  21. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the eighth ACM international conference on web search and data mining, pages 97–106. ↗
  22. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 704–714.
  23. Samer Muthana Sarsam, Hosam Al-Samarraie, Ahmed Ibrahim Alzahrani, and Bianca Wright. 2020. Sarcasm detection using machine learning algorithms in twitter: A systematic review. International Journal of Market Research, 62(5):578–598. ↗
  24. KP Soman, R Loganathan, and V Ajay. 2009. Machine learning with SVM and other kernel methods. PHI Learning Pvt. Ltd. find
  25. Silviu Vlad Oprea and Walid Magdy. 2020b. The effect of sociocultural variables on sarcasm communication online. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW1):1–22. ↗
  26. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ajayan-2022-amrita-cen-semeval-2022-task/