Elamur Research⌕
☰
home / papers / 2022 / IISERB Brains at SemEval-2022 Task 6: A Deep-learning Framework to Identify Intended Sarcasm in English

IISERB Brains at SemEval-2022 Task 6: A Deep-learning Framework to Identify Intended Sarcasm in English

Paper · 2022 · Tanuj Shekhawat, Manoj Kumar, Udaybhan Rathore et al.

Summary

The paper describes the "IISERB Brains" entry to SemEval-2022 Task 6 (iSarcasmEval), which deals with intended sarcasm in English tweets. The labels are unusual: the tweet authors themselves marked their tweets as sarcastic or not, and for sarcastic tweets they also wrote a plain, non-sarcastic paraphrase. The team entered all three English sub-tasks: binary sarcasm detection (A), multi-label classification of ironic categories (B), and choosing the sarcastic member of a sarcastic/paraphrase pair (C). The task's training set was small and skewed. To compensate, they added earlier irony and sarcasm corpora and created synthetic tweets with NLPAUG. They then fine-tuned Hugging Face transformers: BERT, RoBERTa, XLNet and DistilBERT. Sub-task B used one binary classifier per label, and sub-task C ran two sub-task A models in parallel. The submitted BERT systems ranked 19/43, 8/22 and 13/16. After the competition, RoBERTa raised sub-task A sarcastic-class F1 to 0.47. The work is a modest baseline showing how hard author-intended sarcasm remains for fine-tuned encoders.

Key points

  • Organizer data: 3468 English tweets, of which 867 are sarcastic and 2601 are not. Linguistic experts assigned the categories following Leggitt and Gibbs (2000), using six types: irony, satire, sarcasm, overstatement, understatement and rhetorical questions.
  • The extra training data came from SemEval-2018 Task 3 (train and test), the text part of MUStARD, FigLang 2020 Sarcasm, and 867×3 NLPAUG augmentations. After deduplication this gave 19,986 tweets.
  • Augmentation used contextual word replacement with distilbert-base-uncased embeddings, producing three variants per sarcastic tweet. Preprocessing lowercased text except all-caps words, removed NLTK stopwords, links, punctuation and usernames, and skipped stemming and lemmatization.
  • The submitted BERT systems scored as follows:
    • sub-task A: sarcastic F1 0.34;
    • sub-task B: macro-F1 0.0751;
    • sub-task C: accuracy 0.62.
  • After the competition, RoBERTa reached sarcastic F1 0.47 and accuracy 0.86 on sub-task A, compared with BERT's 0.39.

Interesting ideas

  • I1. The labels capture intent from the speaker: authors tag their own sarcasm and supply a literal paraphrase, giving gold pairs of what was said versus what was meant.
  • I2. For sub-task A, the authors' non-sarcastic paraphrases were added to training as extra negatives, so the model sees near-identical texts that differ only in sarcasm.
  • I3. The multi-label problem was split into one binary classifier per category, each with its own label-specific augmented training set, so that rare categories are not swamped by the dominant one.
  • I4. The pairwise "which one is sarcastic" task was handled by scoring each text independently with the single-text model and then comparing the two scores.
  • I5. Augmentation by contextual word replacement multiplied scarce minority-class examples three times.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): the irony module could be evaluated on such pairs, checking whether the frame built from the sarcastic tweet matches the frame from its literal paraphrase once irony is resolved.
  • I2 (learn): minimal pairs like these make good negative controls when evaluating an irony or implicature detector, revealing whether it reacts to the actual ironic cue or only to topic words.
  • I3 (frames): speech-act and irony subtypes (overstatement, rhetorical question, understatement) could get separate explainable detectors, each traceable to its own evidence.
  • I4 (answer): forced-choice test questions could be answered the same way, by scoring each option with an existing reader and picking the higher, with no special pairwise model needed.
  • I5 (learn): this could create perturbed variants of rare silver examples, keeping only those whose UD tree and frames the witness parsers still agree on.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Science Education and Research, BhopalBhopal · India3 works · 20 citations

Also on the byline: Peptcell.

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works

🧭 Topics 2 English · 1,373 works Sentiment & emotion · 608 works

Cite

Cite as Tanuj Shekhawat, Manoj Kumar, Udaybhan Rathore, Aditya Joshi and Jasabanta Patro. 2022. IISERB Brains at SemEval-2022 Task 6: A Deep-learning Framework to Identify Intended Sarcasm in English. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.131 · arXiv 2203.02244

@inproceedings{shekhawat2022iiserb,
  title = {{IISERB Brains at SemEval-2022 Task 6: A Deep-learning Framework to Identify Intended Sarcasm in English}},
  author = {Tanuj Shekhawat, Manoj Kumar, Udaybhan Rathore, Aditya Joshi and Jasabanta Patro},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.131},
  eprint = {2203.02244},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.131}
}

References

22 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSar-casmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. Srijan Bansal, G Vishal, Ayush Suhane, Jasabanta Patro, and Animesh Mukherjee. Code-switching patterns can be an effective route to improve performance https://github.com/manojmahan/iSarca smEval-Intended-Sarcasm-Detection-In-Eng lish-main Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019b. Towards multimodal sarcasm detection (an _Obviously_ perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4619– 4629, Florence, Italy. Association for Computational Linguistics. ↗
  2. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Enhanced sentiment learning using twitter hashtags and smileys. In Coling 2010: Posters, pages 241– 249.
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  4. Debanjan Ghosh, Alexander R Fabbri, and Smaranda Muresan. 2018. Sarcasm analysis using conversation context. Computational Linguistics, 44(4):755–792. ↗
  5. Debanjan Ghosh, Avijit Vajpayee, and Smaranda Muresan. 2020. A report on the 2020 sarcasm detection shared task. arXiv preprint arXiv:2005.05814. ↗
  6. Kannan Govindan, TC Edwin Cheng, Nishikant Mishra, and Nagesh Shukla. 2018. Big data analytics and application for logistics and supply chain management. ↗
  7. Source SemEval-2022 Task 6 Data Augmentation Total |S| 867 0 867 |I| 867 465 1332 |O| 867 120 906 |Rq| 867 303 1170 |St| 867 75 942 |U| 867 30 98 *Table 6: Dataset sources used for sub-task B. Sample size for categories sarcasm, irony, overstatement, rhetorical question, satire and understatement are represented by |S|, |I|, |O|, |Rq|, |St|, |U| respectively.* Results Macro F-1 F1-Sarcasm F1-irony F1-satire F1-understatement F1-overstatement F1-rhetorical question BERT 0.075 0.23 0.096 0.083 0.00 0.00 0.0414 using quantum probability. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 871–880. find
  8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  9. Jasabanta Patro, Srijan Bansal, and Animesh Mukherjee. 2019. A deep-learning framework to detect sarcasm targets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6336–6342, Hong Kong, China. Association for Computational Linguistics. *Table 7: Macro-F1 and class-wise F1 scores for sub-task B* Model RoBERTa XlNet DistilBERT BERT Accuracy 0.47 0.49 0.57 0.62
  10. Luke C Pilling, Chia-Ling Kuo, Kamil Sicinski, Jone Tamosauskaite, George A Kuchel, Lorna W Harries, Pamela Herd, Robert Wallace, Luigi Ferrucci, and David Melzer. 2017. Human longevity: 25 genetic loci associated in 389,166 uk biobank participants. Aging (Albany NY), 9(12):2504. *Table 8: Accuracy of our models on evaluation data provided for sub-task C.* Nikhil Jaiswal. 2020. Neural sarcasm detection using conversation context. In Proceedings of the Second Workshop on Figurative Language Processing, pages 77–82. ↗
  11. Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys (CSUR), 50(5):1–22. ↗
  12. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015a. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757–762.
  13. Ankur Joshi, Saket Kale, Satish Chandel, and D Kumar Pal. 2015b. Likert scale: Explored and explained. British journal of applied science & technology, 7(4):396. ↗
  14. John S Leggitt and Raymond W Gibbs. 2000. Emotional reactions to verbal irony. Discourse processes, 29(1):1–24. ↗
  15. Yaochen Liu, Yazhou Zhang, Qiuchi Li, Benyou Wang, and Dawei Song. 2021. What does your smile mean? jointly detecting multi-modal sarcasm and sentiment Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 704–714.
  16. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. ↗
  17. Himani Srivastava, Vaibhav Varshney, Surabhi Kumari, and Saurabh Srivastava. 2020. A novel hierarchical bert architecture for sarcasm detection. In Proceedings of the Second Workshop on Figurative Language Processing, pages 93–97.
  18. Ishan Tarunesh, Somak Aditya, and Monojit Choudhury. 2021. Trusting roberta over bert: Insights from checklisting the natural language inference task. arXiv preprint arXiv:2107.07229. ↗
  19. Yi Tay, Luu Anh Tuan, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. arXiv preprint arXiv:1805.02856. ↗
  20. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50.
  21. Tony Veale and Yanfen Hao. 2010. Detecting ironic intent in creative comparisons. In ECAI 2010, pages 765–770. IOS Press. ↗
  22. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shekhawat-2022-iiserb-brains-semeval-2022-task/