Elamur Research⌕
☰
home / papers / 2022 / GetSmartMSEC at SemEval-2022 Task 6: Sarcasm Detection using Contextual Word Embedding with Gaussian model for Irony Type Identification

GetSmartMSEC at SemEval-2022 Task 6: Sarcasm Detection using Contextual Word Embedding with Gaussian model for Irony Type Identification

Paper · 2022 · Diksha Krishnan, Jerin Mahibha C and Thenmozhi Durairaj

Summary

The paper describes the GetSmartMSEC entry to iSarcasmEval (SemEval-2022 Task 6), English track only. The task has three parts: deciding whether a text is sarcastic (Subtask A), assigning one or more of six ironic-speech categories (Subtask B), and choosing which member of a pair is sarcastic, the original or its plain non-sarcastic rephrase (Subtask C). The training set is small and skewed: 3468 texts, about a quarter of them sarcastic. The authors held out 20% of it for validation. Preprocessing strips escape sequences and stop words, then tokenizes and lemmatizes. For Subtasks A and C they feed ELMo embeddings from TensorFlow Hub into a small network with two dense layers. For Subtask B they use TF-IDF vectors with a Gaussian Naive Bayes classifier. Results were weak: sarcastic-class F1 of 0.2012 on A, macro-F1 of 0.0387 on B, and accuracy of 0.34 on C. The authors acknowledge that transformer systems did better. They suggest richer context, hybrid models and multimodal signals as next steps. The paper is mainly useful as a documented baseline and a cautionary example.

Key points

  • Data: the iSarcasmEval English training set has 3468 instances, 867 sarcastic and 2601 not. The test set has 1400 instances.
    • Irony-type counts are heavily skewed: sarcasm 713, irony 155, rhetorical question 101, overstatement 40, satire 25, understatement 10.
  • Subtasks A and C model: ELMo embedding layer, then a 256-unit ReLU dense layer, then a 2-unit softmax output. Labels are one-hot encoded.
  • Subtask A results: sarcastic F1 0.2012 (rank 36), accuracy 0.705, macro F-score 0.5101.
  • Subtask B results: TF-IDF + Gaussian Naive Bayes gave macro-F1 0.0387 (rank 19). Only the sarcasm label scored above zero (F1 0.2321); all five other categories scored 0.
  • Subtask C results: accuracy 0.34 (rank 15) and F1 0.2794 on the task of picking the sarcastic text from a text/rephrase pair.

Interesting ideas

  • I1. Each sarcastic text comes with an author-written rephrase that keeps the meaning but drops the sarcasm, giving minimal pairs where only the literal surface differs from the intent.
  • I2. The paper's own examples (calling a 3 am bedtime wonderful, or claiming to enjoy a higher tax bill) show sarcasm as praise paired with a situation normally judged bad. The related work it cites models exactly this sentiment-versus-situation contrast.
  • I3. The preprocessing discards stop words before classification, yet cues such as "not", "#not" or "just the best" live in exactly those small words, which plausibly contributes to the weak results.
  • I4. Per-label F1 on Subtask B exposed that the classifier effectively learned only the majority label, with five categories at zero, something a single aggregate number would hide.
  • I5. The pairwise "which of these two is sarcastic" format has a built-in coin-flip reference, so a score like 0.34 is immediately readable as worse than guessing.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): the prag irony module could align the frames of a sarcastic text and its rephrase to learn which literal frame flips into which intended frame, making the irony rule explicit.
  • I2 (frames): an irony detector could fire when a positive evaluative predicate attaches to an event whose world-knowledge valence is negative, giving an explainable reason rather than a score.
  • I3 (tokens): keep function words, hashtags like #not, and intensifiers as first-class tokens with tags, since negation and degree markers carry the pragmatic signal.
  • I4 (learn): evaluation reports should always break scores down per class and flag any label at zero as a collapse, especially for rare irony or speech-act types.
  • I5 (learn): forced-choice pairs make cheap negative controls, because any component scoring below chance on them signals a systematic inversion bug rather than mere weakness.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sri Sivasubramaniya Nadar College of EngineeringChennai · India2 works · 4 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksSentiment & emotion · 608 works

Cite

Cite as Diksha Krishnan, Jerin Mahibha C and Thenmozhi Durairaj. 2022. GetSmartMSEC at SemEval-2022 Task 6: Sarcasm Detection using Contextual Word Embedding with Gaussian model for Irony Type Identification. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.114

@inproceedings{krishnan2022getsmartmsec,
  title = {{GetSmartMSEC at SemEval-2022 Task 6: Sarcasm Detection using Contextual Word Embedding with Gaussian model for Irony Type Identification}},
  author = {Diksha Krishnan, Jerin Mahibha C and Thenmozhi Durairaj},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.114},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.114}
}

References

19 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Deepak Jain, Akshi Kumar, and Geetanjali Garg. 2020. Sarcasm detection in mash-up language using soft-attention based bi-directional lstm and feature-rich cnn. Applied Soft Computing, 91:106198. ↗
  2. Ajiboye Abdulraheem, Ruzaini Abdullah Arshah, and Hongwu Qin. 2015. Evaluating the effect of dataset size on predictive model using supervised learning technique. International Journal of Software Engineering Computer Sciences (IJSECS), 1:75–84. find
  3. A Kalaivani and D Thenmozhi. 2020. Sarcasm identification and detection in conversion context using bert. In Proceedings of the Second Workshop on Figurative Language Processing, pages 72–76.
  4. Ibrahim Abu Farha and Walid Magdy. 2020. From Arabic sentiment analysis to sarcasm detection: The ArSarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39, Marseille, France. European Language Resource Association.
  5. Ibrahim Abu Farha and Walid Magdy. 2021. Benchmarking transformer-based language models for Arabic sentiment and sarcasm detection. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 21–31, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  6. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics.
  7. Nastaran Babanejad, Heidar Davoudi, Aijun An, and Manos Papagelis. 2020. Affective and contextual embedding for sarcasm detection. In Proceedings of the 28th International Conference on Computational Linguistics, pages 225–243, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  8. Arghasree Banerjee, Mayukh Bhattacharjee, Kushankur Ghosh, and Sankhadeep Chatterjee. 2020. Synthetic minority oversampling in addressing imbalanced sarcasm detection in social media. Multimedia Tools and Applications, 79(47):35995–36031. ↗
  9. Yu Du, Tong Li, Muhammad Salman Pathan, Hailay Kidu Teklehaimanot, and Zhen Yang. 2022. An effective sarcasm detection approach based on sentimental context and individual expression habits. Cognitive Computation, 14(1):78–90. ↗
  10. Cyril Goutte, Michel Simard, and Marine Carpuat. 2014. CNRC-TMT: Second language writing assistant system description. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval Avinash Kumar, Vishnu Teja Narapareddy, Veerub-hotla Aditya Srikanth, Aruna Malapati, and Lalita Bhanu Murthy Neti. 2020. Sarcasm detection using multi-head attention based bidirectional lstm. IEEE Access, 8:6388–6397. find
  11. Amit Kumar Jena, Aman Sinha, and Rohit Agarwal. 2020. C-net: Contextual network for sarcasm detection. In Proceedings of the Second Workshop on Figurative Language Processing, pages 61–66, Online. Association for Computational Linguistics.
  12. Chenwei Lou, Bin Liang, Lin Gui, Yulan He, Yixue Dang, and Ruifeng Xu. 2021. Affective Dependency Graph for Sarcasm Detection, page 1844–1849. Association for Computing Machinery, New York, NY, USA. find
  13. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  14. Rolandos Alexandros Potamias, Georgios Siolas, and Andreas . Georgios Stafylopatis. 2020. A transformer-based approach to irony and sarcasm detection. Neural Computing and Applications, 32(23):17309–17320. ↗
  15. Lu Ren, Bo Xu, Hongfei Lin, Xikai Liu, and Liang Yang. 2020. Sarcasm detection with sentiment semantics enhanced multi-level memory network. Neurocomputing, 401:320–326. ↗
  16. Samer Muthana Sarsam, Hosam Al-Samarraie, Ahmed Ibrahim Alzahrani, and Bianca Wright. 2020. Sarcasm detection using machine learning algorithms in twitter: A systematic review. International Journal of Market Research, 62(5):578–598. ↗
  17. Thanh Vu, Dat Quoc Nguyen, Xuan-Son Vu, Dai Quoc Nguyen, Michael Catt, and Michael Trenell. 2018. Nihrio at semeval-2018 task 3: A simple and accurate neural network model for irony detection in twitter.
  18. Yazhou Zhang, Yaochen Liu, Qiuchi Li, Prayag Tiwari, Benyou Wang, Yuhua Li, Hari Mohan Pandey, Peng Zhang, and Dawei Song. 2021. Cfn: A complex-valued fuzzy network for sarcasm detection in conversations. IEEE Transactions on Fuzzy Systems, 29(12):3696–3710. ↗
  19. Xuan Zhao, Jimmy Huang, and Haitian Yang. 2021. Cans: Coupled-attention networks for sarcasm detection on social media. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/krishnan-2022-getsmartmsec-semeval-2022-task-6/