Elamur Research⌕
☰
home / papers / 2022 / I2C at SemEval-2022 Task 6: Intended Sarcasm in English using Deep Learning Techniques

I2C at SemEval-2022 Task 6: Intended Sarcasm in English using Deep Learning Techniques

Paper · 2022 · Adrián Moreno Monterde, Laura Vázquez Ramos, Jacinto Mata et al.

Summary

The authors, from the Universidad de Huelva, took part in Subtask B of SemEval-2022 Task 6 (iSarcasmEval). The subtask is multi-label: an English tweet may belong to any of six ironic speech categories (sarcasm, irony, satire, understatement, overstatement, rhetorical question) or to none. Only 867 tweets carried these labels, and the categories were heavily skewed. To compensate, the team created extra examples by swapping one word for a WordNet synonym and added hand-picked tweets from an external sarcasm corpus. They also compared three levels of text cleaning. They built two systems: a naive Bayes classifier and a set of six binary BERT-base-uncased classifiers using binary relevance. On their own stratified split, BERT with augmentation and external data reached a macro-F1 of 0.75. The official test score was only 0.0699, which placed them 10th of 22 teams. The work shows that data scarcity dominates fine-grained irony typing and that aggressive text cleaning can erase the cues irony depends on.

Key points

  • The labelled portion has 867 tweets. Category counts are sarcasm 713, irony 155, rhetorical question 101, overstatement 40, satire 25 and understatement 10. After manual additions from Oprea and Magdy (2019), the set grew to 904 tweets.
  • Augmentation replaced a single random word with a WordNet synonym and kept the original labels. The authors recommend doing this only once to limit overfitting.
  • The final system used binary relevance: one BERT-base-uncased classifier per label, trained with batch size 32 for 5 epochs on a stratified 80/20 split.
  • In the internal ablation, macro-F1 rose from 0.54 with no extras to 0.71 with minority-class augmentation and 0.75 with external data. It then fell to 0.74 and 0.65 as heavier preprocessing (mention/link removal, then stopword removal) was added.
  • The official test macro-F1 was 0.0699, ranked 10th among 22 participating teams.

Interesting ideas

  • I1. The six ironic speech categories from Leggitt and Gibbs are each defined by how the text relates to the actual state of affairs and whether it targets an addressee, which turns "irony" into a structured, checkable decision.
  • I2. The preprocessing ablation showed that stripping casing, symbols, elongated words and stopwords hurts, because these surface signals carry ironic intent.
  • I3. Augmentation was kept minimal (one synonym swap per new example, applied once) and aimed mainly at rare labels, so that near-duplicates would not inflate scores.
  • I4. Binary relevance is simple but blind to label co-occurrence, and the authors flag that it may miss dependent label combinations.
  • I5. The internal score of 0.75 against the official 0.0699 is a reminder that a small random split, built after augmentation and manual additions, can badly overstate quality.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): crate prag could model irony subtypes as explicit tests: does the stated claim contradict the world of the text, is there a criticised addressee, is a question's invited inference false.
  • I2 (tokens): crate en should keep such features (all-caps, letter stretching, emoji, hashtags like #sarcasm) as token attributes rather than normalising them away, so later stages can use them as evidence.
  • I3 (learn): when building silver data for rare constructions, small controlled perturbations limited to minority cases could add coverage while bounding duplication.
  • I4 (frames): irony subtypes such as sarcasm and rhetorical question often co-occur, so prag could score them jointly or with explicit compatibility rules instead of independent yes/no decisions.
  • I5 (learn): split data before any augmentation, keep perturbed variants on the same side of the split, and use held-out negative controls before trusting a gain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universidad de HuelvaHuelva · Spain2 works · 7 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksSentiment & emotion · 608 works

Cite

Cite as Adrián Moreno Monterde, Laura Vázquez Ramos, Jacinto Mata and Victoria Pachón Álvarez. 2022. I2C at SemEval-2022 Task 6: Intended Sarcasm in English using Deep Learning Techniques. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.119

@inproceedings{monterde2022i2c,
  title = {{I2C at SemEval-2022 Task 6: Intended Sarcasm in English using Deep Learning Techniques}},
  author = {Adrián Moreno Monterde, Laura Vázquez Ramos, Jacinto Mata and Victoria Pachón Álvarez},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.119},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.119}
}

References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abu Farha, Ibrahim, Silviu Oprea, Steven Wilson, Walid Magdy. “SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic”. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics, 2022.
  2. Carvalho, Paula, Luís Sarmento, Mário J. Silva, and Eugénio de Oliveira. 2009. “Clues for Detecting Irony in User-Generated Contents: Oh...!! It’s ‘so Easy’ ;-).” In Proceeding of the 1st International CIKM Workshop on Topic-Sentiment Analysis for Mass Opinion - TSA ’09. New York, New York, USA: ACM Press. find
  3. Wallace, Byron C., Do Kook Choe, Laura Kertz, and Eugene Charniak. 2014. “Humans Require Context to Infer Ironic Intent (so Computers Probably Do, Too).” In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 512–16. Stroudsburg, PA, USA: Association for Computational Linguistics.
  4. Leggitt, John S., and Raymond W. Gibbs. 2000. “Emotional Reactions to Verbal Irony.” Discourse Processes 29 (1): 1–24. https://doi.org/10.1207/s15326950dp2901_1. ↗
  5. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twit- ter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL ’10, pages 107–116, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  6. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. ICWSM - a great catchy name: Semi-supervised recog- nition of sarcastic sentences in online product reviews. In William W. Cohen and Samuel Gosling, editors, Proceedings of the Fourth International Conference on Weblogs and Social Media, ICWSM 2010, Washington, DC, USA, May 23-26, 2010. The AAAI Press. find
  7. Oprea, Silviu, and Walid Magdy. 2019. “ISarcasm: A Dataset of Intended Sarcasm.” ArXiv [Cs.CL]. https://paperswithcode.com/paper/isarcasm-a-dataset-of-intended-sarcasm. ↗
  8. Alzahrani, Esam, and Leon Jololian. 2021. “How Different Text-Preprocessing Techniques Using the BERT Model Affect the Gender Profiling of Authors.” ArXiv [Cs.CL]. http://arxiv.org/abs/2109.13890. ↗
  9. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. "BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding." CoRR abs/1810.04805. ↗
  10. Luaces, Oscar, Jorge Díez, José Barranquero, Juan José del Coz, and Antonio Bahamonde. 2012. “Binary Relevance Efficacy for Multilabel Classification.” Progress in Artificial Intelligence 1 (4): 303–13. https://doi.org/10.1007/s13748-012-0030-x. ↗
  11. Wang, Meng and Fanghui Hu. 2021. "The Application of NLTK Library for Python Natural Language Processing in Corpus Research." Theory and Practice in Language Studies 11 (9): 1041-1049. doi:10.17507/tpls.1109.09. ↗
  12. Hao, Jiangang and Tin Kam Ho. 2019. Machine Learning made Easy: A Review of Scikit-Learn Package in Python Programming Language. Vol. 44. Los Angeles, CA: SAGE Publications. doi:10.3102/1076998619832248. https://journals.sagepub.com/doi/full/10.3102/1076 998619832248 ↗
  13. Stepanek, Hannah. 2020. Thinking in Pandas : How to use the Python Data Analysis Library the Right Way. Berkeley, CA: Apress. doi:10.1007/978-1-4842-5839-2. https://library.biblioboard.com/viewer/087e187ab660-11ea-a44d-0a7fc7c4e64f. ↗
  14. McCrae, John Philip, Alexandre Rademaker, Francis Bond, Ewa Rudnicka, and Christiane Fellbaum. 2019. “English WordNet 2019 – An Open-Source WordNet for English.” In Proceedings of the 10th Global Wordnet Conference, 245–52.
  15. Ashwitha, Shruthi, Shruthi, Makarand Upadhyaya, Abhra Pratip Ray, y Manjunath. 2021. “Sarcasm Detection in Natural Language Processing”. Materials Today: Proceedings 37: 3324–31. https://doi.org/10.1016/j.matpr.2020.09.124. Khatri, Akshay, P, Pranav, y M, Dr. ↗
  16. Anand Kumar. 2020. “Sarcasm detection in tweets with BERT and GloVe embeddings”. https://doi.org/10.48550/ARXIV.2006.11512. ↗
  17. Heckerman, David, and Michael P. Wellman. "Bayesian networks." Communications of the ACM, 1995, 38(3), 27-31 find
  18. Bandy, Jack, and Nicholas Vincent. 2021. “Addressing ‘Documentation Debt’ in Machine Learning Research: A Retrospective Datasheet for BookCorpus.” ArXiv [Cs.CL]. http://arxiv.org/abs/2105.05241. ↗
  19. Wolf, Thomas, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, et al. 2019. “HuggingFace’s Transformers: State-of-the-Art Natural Language Processing.” ArXiv [Cs.CL]. http://arxiv.org/abs/1910.03771. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/monterde-2022-i2c-semeval-2022-task-6/