Elamur Research⌕
☰
home / papers / 2024 / Sarcasm Detection in a Disaster Context

Sarcasm Detection in a Disaster Context

Paper · 2024 · Tiberiu Sosea, Junyi Jessy Li and Cornelia Caragea

Summary

Most sarcasm research uses general-domain Twitter or Reddit data. During natural disasters, however, people often voice contempt for the event or for official guidance through irony, and missing that shift in meaning distorts how crisis tweets are read. The authors build HurricaneSARC, which contains 15,000 English tweets sampled at random from a collection streamed during Hurricanes Harvey, Irma and Maria. Five Amazon Mechanical Turk workers labelled each tweet for intended sarcasm, and a majority vote set the final label. Sarcasm turned out to be rare, so the benchmark pairs the sarcastic tweets with an equal number of non-sarcastic ones. BERTweet beats CNN and BiLSTM baselines and reaches up to 0.70 F1. Intermediate-task training on emotion and sentiment data improves it further. Labels are scarce while a disaster unfolds, so the authors also test semi-supervised learning (UST, Noisy Student). It nearly matches full supervision with far fewer labels. Models trained on general-domain sarcasm corpora transfer clearly worse. The work shows that sarcasm detection is domain-sensitive and that unlabeled in-domain text is a practical resource.

Key points

  • HurricaneSARC: 15,000 tweets from the disaster repository of Ray Chowdhury et al. (2019). Only 820 tweets (about 5.5%) were judged sarcastic.
  • Annotation setup:
    • Annotators received sarcasm definitions from dictionary.com and Wikipedia.
    • Workers needed an acceptance rate above 95% and at least 500 completed tasks.
    • Each tweet got five votes, combined by majority.
  • The balanced splits are 934 train, 300 validation and 400 test, with equal sarcastic and non-sarcastic halves. The authors also release the full unbalanced version. Training on all negatives significantly hurt the models.
  • BERTweet is the strongest model, at up to 0.70 F1. Intermediate-task transfer from emotion and sentiment tasks raises performance further.
  • A BERTweet-based UST model trained on only 200 labelled examples comes within 1% F1 of vanilla BERTweet trained on four times more data. Direct transfer from general Twitter/Reddit sarcasm data lowers F1 by about 15%.

Interesting ideas

  • I1. Collecting labels for the author's intended stance inside one narrow situation (a disaster) shows how much irony depends on knowing the circumstances, not just the wording.
  • I2. The motivating example marks sarcasm through clashing pairs of concepts inside one sentence: comfort and laughter set against destruction.
  • I3. Training on emotion and sentiment tasks before the scarce target task works as a cheap bridge to a related but data-poor phenomenon.
  • I4. Self-training on unlabeled in-domain text, with uncertainty-aware selection or noise such as back translation, nearly matches full supervision from a small seed.
  • I5. Measuring general-domain models directly on the new domain gives a clear estimate of the domain gap before any adaptation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The irony detector in prag could judge praise against the situation recorded in the text's world (e.g., "safe, comfortable home" vs. "destroys lives"), not against cue words alone.
  • I2 (world): A check could flag tweets where an attributed positive state of one entity conflicts with a negative event affecting others, and pass that conflict to prag as an irony signal it can explain.
  • I3 (learn): Mova could first train the irony component on more plentiful sentiment or emotion silver labels, then fine-tune it on a small hand-checked irony set.
  • I4 (learn): Witness-agreed silver data could be selected by model uncertainty, and back-translated paraphrases could serve as noise to make the student parser or classifier more robust.
  • I5 (learn): Each new genre (fables, school tests, tweets) could get a direct-transfer baseline row, so later gains from adaptation can be read against it honestly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 works

Cite

Cite as Tiberiu Sosea, Junyi Jessy Li and Cornelia Caragea. 2024. Sarcasm Detection in a Disaster Context. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2308.08156

@inproceedings{sosea2024sarcasm,
  title = {{Sarcasm Detection in a Disaster Context}},
  author = {Tiberiu Sosea, Junyi Jessy Li and Cornelia Caragea},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/4yteakogw6u3},
  eprint = {2308.08156},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.63317/4yteakogw6u3}
}

References

53 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed and Lyle Ungar. 2017. EmoNet: Fine-Grained Emotion Detection with Gated Recurrent Neural Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 718–728, Vancouver, Canada. Association for Computational Linguistics.
  2. Silvio Amir, Byron C Wallace, Hao Lyu, and Paula Carvalho Mário J Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. arXiv preprint arXiv:1607.00976. ↗
  3. Santosh Kumar Bharti, Korra Sathya Babu, and Sanjay Kumar Jena. 2015. Parsing-based sarcasm sentiment recognition in twitter data. In 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 1373–1380. IEEE. find
  4. Paula Carvalho, Luís Sarmento, Mário J Silva, and Eugénio De Oliveira. 2009. Clues for detecting irony in user-generated contents: oh...!! it’s so easy;-. In Proceedings of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion, pages 53–56. ACM. find
  5. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. ↗
  6. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcasm in twitter and amazon. In Proceedings of the fourteenth conference on computational natural language learning, pages 107–116.
  7. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054. Association for Computational Linguistics. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. ↗
  9. Christiane Fellbaum. 2012. Wordnet. The encyclopedia of applied linguistics. find
  10. Aniruddha Ghosh and Tony Veale. 2016. Fracking sarcasm using neural network. In Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 161–169. ↗
  11. Roberto González-Ibánez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581–586.
  12. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. ArXiv, abs/2004.10964. ↗
  13. Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised domain adaptation of contextualized embeddings for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4238–4248, Hong Kong, China. Association for Computational Linguistics. ↗
  14. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. Cascade: Contextual sarcasm detection in online discussion forums. In COLING.
  15. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  16. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In ACL.
  17. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016. Are word embedding-based features useful for sarcasm detection? arXiv preprint arXiv:1610.00883. ↗
  18. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2017. A large self-annotated corpus for sarcasm. CoRR, abs/1704.05579. ↗
  19. Yoon Kim. 2014a. Convolutional neural networks for sentence classification. In EMNLP.
  20. Yoon Kim. 2014b. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. ↗
  21. Roger Kreuz and Gina Caucci. 2007. Lexical influences on the perception of sarcasm. In Proceedings of the Workshop on computational approaches to Figurative Language, pages 1–4.
  22. CC Liebrecht, FA Kunneman, and APJ van Den Bosch. 2013. The perfect solution for detecting sarcasm in tweets# not.
  23. Stephanie Lukin and Marilyn Walker. 2013. Really? well. apparently bootstrapping improves the performance of sarcasm and nastiness classifiers for online dialogue. In Proceedings of the Workshop on Language Analysis in Social Media, pages 30–40, Atlanta, Georgia.
  24. Navonil Majumder, Soujanya Poria, Haiyun Peng, Niyati Chhaya, Erik Cambria, and Alexander F. Gelbukh. 2019. Sentiment and sarcasm classification with multitask learning. CoRR, abs/1901.08014. ↗
  25. Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4238–4243, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗
  26. Abhijit Mishra, Diptesh Kanojia, and Pushpak Bhattacharyya. 2016. Predicting readers’ sarcasm understandability by modeling gaze behavior. In AAAI. find
  27. Saif M Mohammad and Peter D Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3):436–465. ↗
  28. Subhabrata Mukherjee and Ahmed Hassan Awadallah. 2020. Uncertainty-aware self-training for text classification with few labels. arXiv preprint arXiv:2006.15315. ↗
  29. Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9–14.
  30. Silviu Oprea and Walid Magdy. 2020. isarcasm: A dataset of intended sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
  31. Shereen Oraby, Vrindavan Harrison, Lena Reed, Ernesto Hernandez, Ellen Riloff, and Marilyn Walker. 2016. Creating and characterizing a diverse corpus of sarcasm in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 31–41, Los Angeles. Association for Computational Linguistics.
  32. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  33. Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. In Theories of emotion, pages 3–33. Elsevier. ↗
  34. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. arXiv preprint arXiv:1610.08815. ↗
  35. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proc. of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231–5247, Online.
  36. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 213–223.
  37. Jishnu Ray Chowdhury, Cornelia Caragea, and Doina Caragea. 2019. Keyphrase Extraction from Disaster-related Tweets. In The World Wide Web Conference, WWW 2019, pages 1555– 1566, New York, NY, USA. Association for Computing Machinery (ACM). find
  38. Antonio Reyes and Paolo Rosso. 2012. Making objective decisions from subjective data: Detecting irony in customer reviews. Decision support systems, 53(4):754–760. ↗
  39. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in twitter. Language resources and evaluation, 47(1):239–268. ↗
  40. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 704–714.
  41. Hannah Ritchie and Max Roser. 2020. Natural Disasters. Our World in Data. find
  42. Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. CARER: Contextualized affect representations for emotion recognition. In Proc. of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687–3697, Brussels, Belgium.
  43. Edoardo Savini and Cornelia Caragea. 2020. A multi-task learning approach to sarcasm detection (student abstract). Proceedings of the AAAI Conference on Artificial Intelligence, 34(10):13907–13908. find
  44. Edoardo Savini and Cornelia Caragea. 2022. Intermediate-task transfer learning with bert for sarcasm detection. Mathematics, 10(5). ↗
  45. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. ↗
  46. Joseph Tepperman, David Traum, and Shrikanth Narayanan. 2006. " yeah right": Sarcasm recognition for spoken dialogue systems. In Ninth international conference on spoken language processing. find
  47. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm—a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In Fourth International AAAI Conference on Weblogs and Social Media. find
  48. Tony Veale and Yanfen Hao. 2010. Detecting ironic intent in creative comparisons. In ECAI, volume 215, pages 765–770. ↗
  49. Byron C Wallace, Eugene Charniak, et al. 2015. Sparse, contextually informed models for irony detection: Exploiting user communities, entities and sentiment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1035–1044.
  50. Byron C Wallace, Laura Kertz, Eugene Charniak, et al. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 512–516.
  51. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online.
  52. Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. 2020. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10687–10698. find
  53. Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Tweet sarcasm detection using deep neural network. In Proceedings of COLING 2016, The 26th International Conference on Computational Linguistics: Technical Papers, pages 2449– 2460.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sosea-2024-sarcasm-detection-disaster-context/