MarSan at SemEval-2022 Task 6: iSarcasm Detection via T5 and Sequence Learners
Summary
The paper presents the MarSan team's system for SemEval-2022 Task 6 (iSarcasm), which targets intended sarcasm in English and Arabic tweets. The task has three subtasks: a binary sarcastic/non-sarcastic decision (A), multi-label tagging of ironic-speech categories in English (B), and picking the sarcastic member of a pair made of a sarcastic tweet and its plain paraphrase (C). The task data is small and skewed, since only about a quarter of subtask A items are sarcastic. To compensate, the authors first fine-tune the language model on larger external sarcasm corpora: iSarcasm for English and an Arabic tweet set by Farha and Magdy. The resulting T5 or MT5 encoder supplies token vectors. These pass through a Transformer encoder block and a Bi-LSTM, whose outputs are concatenated. Scaled dot-product attention, a tanh dense layer, max-pooling and softmax complete the model. The same network serves all subtasks, with only the input and output layers adjusted. Results were strongest on the pairwise task, especially for Arabic. The error analysis shows the model depends heavily on emoji and sentiment cues.
Key points
- Reported rankings: subtask A 7th in English and 31st in Arabic (the Arabic run had a submission error), subtask B 9th of 22 teams, subtask C 7th of 16 in English and 3rd of 13 in Arabic.
- Data and fine-tuning:
- Training items: 3121 for A English, 2792 for A Arabic and 780 for B.
- A random 10% was held out as dev data because no official dev set existed.
- External corpora used for fine-tuning: iSarcasm (4,484 tweets) and an Arabic set of 10,547 tweets.
- Subtask B used six separate binary classifiers. On the test set:
- understatement (1 sample) and overstatement (10 samples) both got F1 of 0;
- sarcasm got F1 19.81;
- recall was far higher than precision across classes.
- Subtask C (English) per-class F1 was 80.56 and 77.17. In subtask A, 276 non-sarcastic tweets were wrongly flagged as sarcastic.
- Error types the authors identified:
- emoji and sentiment help the model;
- very short tweets and misspellings hurt it;
- tweets whose sarcasm depends on background human knowledge also hurt it.
Interesting ideas
- I1. Framing sarcasm as a choice between a sarcastic text and its literal paraphrase isolates the ironic signal far better than absolute labelling, and it gave the system's best results. — For Mova (frames): The irony detector in prag could be tested and trained on minimal pairs (ironic vs. literal paraphrase), checking that the implicature frame fires on exactly one member.
- I2. The hardest failures were tweets where nothing in the wording signals irony and only world knowledge exposes it, e.g. masks being required in shops but not in pubs. — For Mova (world): A text-world that tracks stated rules and states could flag an evaluative claim contradicted by those states, giving an explainable irony cue where surface sentiment is absent.
- I3. Error analysis was organised into a small taxonomy of cue types (emoji, sentiment, shortness, misspelling, knowledge-only), with examples sorted into TP and FP. — For Mova (learn): Evaluation reports could bucket irony errors by cue type, revealing whether the system genuinely reasons or merely reacts to sentiment words.
- I4. With extremely rare categories, treating each label as its own binary decision and reporting per-class precision and recall exposed that some classes were never learned at all. — For Mova (learn): Per-class tables should be kept for rare speech-act or irony labels, so that one or two silver examples are not mistaken for coverage.
- I5. Fine-tuning first on a larger related corpus, before training on the tiny target set, gave the encoder prior awareness of the task. — For Mova (learn): gparse or the irony tagger could be pre-adapted on a larger noisy silver set of the same genre before training on a small gold set.
Cited by
Cite
@inproceedings{najafi2022marsan,
title = {{MarSan at SemEval-2022 Task 6: iSarcasm Detection via T5 and Sequence Learners}},
author = {Najafi, Maryam and Tavan, Ehsan},
booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
year = {2022},
doi = {10.18653/v1/2022.semeval-1.137},
url = {https://doi.org/10.18653/v1/2022.semeval-1.137}
}References
- Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSar-casmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. find
- Tanvi Dadu and Kartikey Pant. 2020. Sarcasm detection using context separators in online discourse. In Proceedings of the Second Workshop on Figurative Language Processing, pages 51–55. find our card
- Ibrahim Abu Farha and Walid Magdy. 2020. From arabic sentiment analysis to sarcasm detection: The arsarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39. find our card
- Roberto González-Ibánez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581–586. find our card
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. ↗
- Ehsan Tavan, Ali Rahmati, Maryam Najafi, and Saeed Bibak. 2021. Bert-dre: Bert with deep recursive encoder for natural language sentence matching. arXiv preprint arXiv:2111.02188. ↗
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
- Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. ↗
- Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. Cascade: Contextual sarcasm detection in online discussion forums. arXiv preprint arXiv:1805.06413. ↗ our card
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
- Soroush Javdan, Behrouz Minaei-Bidgoli, et al. 2020. Applying transformers and aspect-based sentiment analysis approaches on sarcasm detection. In Proceedings of the Second Workshop on Figurative Language Processing, pages 67–71. find our card
- Avinash Kumar, Vishnu Teja Narapareddy, Veerub-hotla Aditya Srikanth, Aruna Malapati, and Lalita Bhanu Murthy Neti. 2020. Sarcasm detection using multi-head attention based bidirectional lstm. Ieee Access, 8:6388–6397. find
- Yann LeCun, Patrick Haffner, Léon Bottou, and Yoshua Bengio. 1999. Object recognition with gradient-based learning. In Shape, contour and grouping in computer vision, pages 319–345. Springer. find
- Silviu Oprea and Walid Magdy. 2019. isarcasm: A dataset of intended sarcasm. arXiv preprint arXiv:1911.03123. ↗ our card
- Rolandos Alexandros Potamias, Georgios Siolas, and Andreas-Georgios Stafylopatis. 2020. A find
This page in Markdown → https://md.research.elamur.ai/free/en/papers/najafi-2022-marsan-semeval-2022-task-6/