Elamur Research⌕
☰
home / papers / 2022 / UTNLP at SemEval-2022 Task 6: A Comparative Analysis of Sarcasm Detection Using Generative-based and Mutation-based Data Augmentation

UTNLP at SemEval-2022 Task 6: A Comparative Analysis of Sarcasm Detection Using Generative-based and Mutation-based Data Augmentation

Paper · 2022 · Amirhossein Abaskohi, Arash Rasouli, Tanin Zeraati et al.

Summary

The paper describes team UTNLP's system for SemEval-2022 Task 6, which asks whether a tweet is sarcastic. The main dataset is iSarcasm, where tweet authors labelled their own posts. It is small and skewed toward non-sarcastic examples. The authors compare two ways of enlarging the training set. The first is generative: GPT-2 produces synthetic tweets. The second is mutation-based: existing tweets are perturbed by shuffling words, dropping words or substituting synonyms. They also test preprocessing choices and several model families: SVM with TF-IDF, Word2Vec or BERT features, LSTM/BLSTM with and without CNN layers, attention-augmented models, BERT, RoBERTa and T5. RoBERTa trained on mutated data reached 0.38 F1-sarcastic in the official evaluation. After the competition the authors found that preprocessing had been hurting them, and removing it raised the score to 0.414. The study suggests that simple perturbations help more than synthetic generation for this task. It also shows that adding sarcasm or sentiment corpora from other sources can lower performance on author-labelled data.

Key points

  • Data: iSarcasm originally had 4484 tweets (777 sarcastic, 3707 not), but only 3469 could still be retrieved. Adding the Sarcasm Headlines Dataset or Sentiment140 made results worse, which the authors attribute to different domains and labelling procedures.
  • SVM feature comparison: Without augmentation, the SVM reached 0.2862 F1-sarcastic with BERT features, against 0.2541 for Word2Vec and 0.0924 for TF-IDF.
  • Augmentation comparison on SVM: Mutation variants scored about 0.30–0.306 F1-sarcastic, while GPT-2 generation scored 0.292 with clearly lower accuracy. When re-checked on RoBERTa, dropping words alone worked best.
  • Best models: RoBERTa (twitter-roberta-base) reached 0.414, slightly above the reported LOANT state of the art of 0.4101 on iSarcasm. T5 reached 0.4038 and plain BERT 0.323. The LSTM-family models peaked at 0.2751 (BLSTM+CNN).
  • Preprocessing: None of the preprocessing strategies helped. Skipping them added 0.034 F1 over the competition submission.

Interesting ideas

  • I1. Light, label-preserving edits to real examples, such as removing a word, beat text produced by a generative model, probably because the edits keep the original label and domain intact.
  • I2. Extra data labelled under a different procedure or domain can harm a model even when it looks topically related. Annotation provenance therefore matters as much as volume.
  • I3. iSarcasm records the sarcasm the author intended rather than what a reader detected, and the two notions can differ.
  • I4. A cheap SVM probe was used to choose features, augmentation and preprocessing before training costly transformers.
  • I5. Normalizing noisy tweets removed useful signal, and raw input performed better.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver training sets for irony or frame classifiers could be expanded by deleting optional modifiers rather than asking an LLM teacher for new sentences, with a negative control to confirm the label survives.
  • I2 (tree): Treebanks with different UD dialects or annotation conventions should be converted, or at least tested separately against a held-out set, before being merged into parser training.
  • I3 (frames): The prag irony module could store speaker intent and the cues a reader could actually perceive as separate fields, so each irony decision is explained by which cues triggered it.
  • I4 (learn): Fast perceptron-level runs could screen data-pipeline variants before committing GPU time to gparse training.
  • I5 (tokens): The tokenizer and tagger should keep surface cues like capitalization, punctuation runs and emoticons as features instead of stripping them, since pragmatics may need them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TehranTehran · Iran9 works · 182 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 1,035 works

Cite

Cite as Amirhossein Abaskohi, Arash Rasouli, Tanin Zeraati and Behnam Bahrak. 2022. UTNLP at SemEval-2022 Task 6: A Comparative Analysis of Sarcasm Detection Using Generative-based and Mutation-based Data Augmentation. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.135 · arXiv 2204.08198

@inproceedings{abaskohi2022utnlp,
  title = {{UTNLP at SemEval-2022 Task 6: A Comparative Analysis of Sarcasm Detection Using Generative-based and Mutation-based Data Augmentation}},
  author = {Amirhossein Abaskohi, Arash Rasouli, Tanin Zeraati and Behnam Bahrak},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.135},
  eprint = {2204.08198},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.135}
}

References

32 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2014. Yelp. yelp open dataset. http://www.yelp.com/dataset_challenge. ↗
  2. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSar-casmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. find
  3. Alexandre Berard, Ioan Calapodescu, and Claude Roux. 2019. Naver labs europe’s systems for the wmt19 machine translation robustness task. arXiv preprint arXiv:1907.06488. ↗
  4. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcasm in twitter and amazon. In Proceedings of the fourteenth conference on computational natural language learning, pages 107–116.
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  6. Steven Y Feng, Varun Gangal, Dongyeop Kang, Teruko Mitamura, and Eduard Hovy. 2020. Genaug: Data augmentation for finetuning text generators. arXiv preprint arXiv:2010.01794. Based on the hyperparameters listed in the Section 4, our F1-sarcastic for this model is 0.4038. However, we believe that we may get better results by Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009. ↗
  7. Roberto González-Ibánez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581–586.
  8. Xu Guo, Boyang Li, Han Yu, and Chunyan Miao. 2021. Latent-optimized adversarial neural transfer for sarcasm detection. arXiv preprint arXiv:2104.09261. ↗
  9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. ↗
  10. Suzana Ilić, Edison Marrese-Taylor, Jorge A Balazs, and Yutaka Matsuo. 2018. Deep contextualized word representations for detecting sarcasm and irony. arXiv preprint arXiv:1809.09795. ↗
  11. Sandeepa Kannangara. 2018. Mining twitter for fine-grained political opinion polarity classification, ideology detection and sarcasm detection. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 751–752. find
  12. Hankyol Lee, Youngjae Yu, and Gunhee Kim. 2020. Augmenting data for sarcasm detection with unlabeled conversation context. arXiv preprint arXiv:2006.06259. ↗
  13. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  14. Stephanie Lukin and Marilyn Walker. 2017. Really? well. apparently bootstrapping improves the performance of sarcasm and nastiness classifiers for online dialogue. arXiv preprint arXiv:1708.08572. ↗
  15. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  16. Rishabh Misra and Prahal Arora. 2019. Sarcasm detection using hybrid neural network. arXiv preprint arXiv:1908.07414. ↗
  17. Rishabh Misra and Jigyasa Grover. 2021. Sculpting data for ml: The first act of machine learning. find
  18. Anwar Ur Rehman, Ahmad Kamran Malik, Basit Raza, and Waqar Ali. 2019. A hybrid cnn-lstm model for improving accuracy of movie reviews sentiment analysis. Multimedia Tools and Applications, 78(18):26597–26613. ↗
  19. Antonio Reyes, Paolo Rosso, and Davide Buscaldi. 2012. From humor recognition to irony detection: The figurative language of social media. Data & Knowledge Engineering, 74:1–12. ↗
  20. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in twitter. Language resources and evaluation, 47(1):239–268. ↗
  21. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 704–714.
  22. Yi Tay, Luu Anh Tuan, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. arXiv preprint arXiv:1805.02856. ↗
  23. Piyoros Tungthamthiti, Kiyoaki Shirai, and Masnizah Mohd. 2016. Recognition of sarcasm in microblog-ging based on sentiment analysis and coherence identification. Journal of Natural Language Processing, 23(5):383–405. find
  24. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50.
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  26. Tony Veale, F Amílcar Cardoso, and Rafael Pérez y Pérez. 2019. Systematizing creativity: A computational view. In Computational Creativity, pages 1–19. Springer. ↗
  27. Marilyn Walker, Jean E Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A corpus for research on deliberation and debate. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 812–817.
  28. Silviu Oprea and Walid Magdy. 2019. isarcasm: A dataset of intended sarcasm. arXiv preprint arXiv:1911.03123. ↗
  29. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  30. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. ↗
  31. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. ↗
  32. Renjie Zheng, Hairong Liu, Mingbo Ma, Baigong Zheng, and Liang Huang. 2019. Robust machine translation with domain sensitive pseudo-sources: Baidu-osu wmt19 mt robustness shared task system report. arXiv preprint arXiv:1906.08393. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abaskohi-2022-utnlp-semeval-2022-task-6/