Elamur Research
☰
home / papers / 2020 / iSarcasm: A Dataset of Intended Sarcasm

iSarcasm: A Dataset of Intended Sarcasm

What it is about

Sarcasm datasets label either by hashtags (#sarcasm, /s) or by annotators; the first mislabels tweets that merely mention sarcasm and misses untagged ones, the second captures how readers perceive a text, not what its author meant. iSarcasm asks Twitter users themselves for one sarcastic and three non-sarcastic tweets of their own, with an explanation of why the sarcastic one is sarcastic and a sincere rephrasing; a linguist then sorts the sarcastic tweets into sarcasm, irony, satire, understatement, overstatement and rhetorical question, discarding unclear ones (777 sarcastic, 3,707 non-sarcastic tweets kept). Models that reach F 0.87 on earlier datasets score at most 0.36 on iSarcasm, and third-party annotators also miss much of the intended sarcasm, which suggests earlier datasets were biased towards obvious cases.

Ideas

For Mova

languages/en/34-irony-and-figurative-language.md §2 (verbal irony and sarcasm), §5 (hyperbole and understatement), §10 (Mova's detector), §12 (figurative language in the tales). Code: prag/src/irony.rs, prag/src/frames.rs (narrator markers). Measures: precision/recall on narrator-intended lines; teacher-vs-intended agreement.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.118.pdf

Summary

Most sarcasm detection corpora are labelled in one of two ways. Some rely on hashtags or specific accounts (distant supervision). Others rely on annotators' judgements. Neither method shows whether the writer actually meant to be sarcastic. Oprea and Magdy separate intended sarcasm, defined from the author's side, from perceived sarcasm, defined from the reader's side. They argue that existing resources mostly capture noisy perception. To address this, they built iSarcasm through a survey. Twitter users submitted their own past tweets, both sarcastic and non-sarcastic. For each sarcastic tweet, the authors explained the sarcasm and rewrote the message literally. Quality checks filtered out unreliable responses. A linguist then assigned the sarcastic items to the sarcasm subtypes of Leggitt and Gibbs Jr. (2000). Crowdworkers who labelled the same tweets matched the authors' labels only moderately well. Strong published detectors that had been reimplemented and validated also dropped sharply on the new data. This suggests that earlier benchmarks overstate progress. It also shows that modelling author intent remains largely unsolved.

Key points

  • Dataset contents: iSarcasm contains 4,484 English tweets labelled by their own authors. The ratio of sarcastic to non-sarcastic tweets is about 1:5. Each sarcastic tweet comes with the author's explanation, a non-sarcastic rephrasing, and a sarcasm category assigned by a linguist. The dataset is released on GitHub (silviu-oprea/iSarcasm).
  • Problems with hashtag labels: In the Riloff dataset, annotators marked 486 tweets that carried sarcasm hashtags as non-sarcastic, compared with 345 they marked as sarcastic. This shows how unreliable hashtag-based labels are.
  • Human baseline: Third-party crowd annotators recovered the authors' labels with an F-score of 0.616.
  • Model results: The best model was MIARN, with an F-score of 0.364 on iSarcasm. On Ptacek it scores 0.874, and on Riloff 0.741.
  • Other models also degrade: 3CNN fell from 0.675/0.788 on SARC-balanced/imbalanced to 0.286 on iSarcasm. Dense-LSTM fell from 0.666 on SemEval-2018 to 0.318. LSTM, Att-LSTM, CNN and SIARN all scored between 0.32 and 0.36.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.118 · arXiv 1911.03123

@inproceedings{oprea2020isarcasm,
  title = {{iSarcasm: A Dataset of Intended Sarcasm}},
  author = {Oprea, Silviu and Magdy, Walid},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.118},
  eprint = {1911.03123},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.118}
}

References

49 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Previous models that achieved high performance in detecting sarcasm on datasets sampling perceived sarcasm (third-party labels) or hash-tagged sarcasm (distant supervision) have failed dramatically to detect sarcasm as meant by its author. This motivates the need to develop more effective methods for detecting intended sarcasm. Potentially, building models that account for sociocultural traits of the authors (available on, or inferred from, their Twitter profiles), or consider other contextual elements to judge the sarcasm in our dataset (Rockwell and Theriot, 2001). Previous research has considered certain contextual elements (Bamman and Smith, 2015b; Amir et al., 2016; Hazarika et al., 2018; Oprea and Magdy, 2019), but only on sarcasm captured by previous labelling methods. We believe the iSarcasm dataset, with its novel method of sampling sarcasm as intended by its Gavin Abercrombie and Dirk Hovy. 2016. Putting Sarcasm Detection into Context: The Effects of Class Imbalance and Manual Labelling on Supervised Machine Classification of Twitter Conversations. In Proceedings of the ACL 2016 Student Research Workshop, pages 107–113. ACL. find
  2. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mario J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In CoNLL, pages 167–177. ACL. find our card
  3. Salvatore Attardo. 2002. Talk is cheap: Sarcasm, alienation, and the evolution of language. Journal of Pragmatics, 34. find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR. find our card
  5. David Bamman and Noah A. Smith. 2015a. Contextualized sarcasm detection on twitter. In ICWSM, pages 574–577. AAAI Press. find
  6. David Bamman and Noah A. Smith. 2015b. Contextualized sarcasm detection on twitter. In ICWSM, pages 574–577. AAAI Press. find
  7. Megan L. Dress, Roger J. Kreuz, Kristen E. Link, and Gina M. Caucci. 2008. Regional variation in the use of sarcasm. JLS, 27(1):71–85. find
  8. Francesco Barbieri, Francesco Ronzano, and Horacio Saggion. 2014a. Italian irony detection in twitter: a first approach. In CLiC-it, page 28. AILC. find
  9. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In LREC. ELRA. find our card
  10. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014b. Modelling sarcasm in twitter, a novel approach. In WASSA, pages 50–58. ACL. find our card
  11. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: A closer look. In HLT, pages 581–586. ACL. find our card
  12. Farah Benamara, Cyril Grouin, Jihen Karoui, Véronique Moriceau, and Isabelle Robba. 2017. Analyse d’opinion et langage figuratif dans des tweets: présentation et résultats du défi fouille de textes deft2017. find
  13. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. Cascade: Contextual sarcasm detection in online discussion forums. In COLING, pages 1837–1848. ACL. find our card
  14. S. K. Bharti, K. S. Babu, and S. K. Jena. 2015. Parsing-based sarcasm sentiment recognition in twitter data. In ASONAM, pages 1373–1380. ACM. find
  15. Delia Irazú Hernández Farı́as, Emilio Sulis, Viviana Patti, Giancarlo Ruffo, and Cristina Bosco. 2015. Valento: Sentiment analysis of figurative language tweets with irony and sarcasm. In SemEval, pages 694–698. ACL. find
  16. S. K. Bharti, K. S. Babu, and S. K. Jena. 2015. Parsing-based sarcasm sentiment recognition in twitter data. In ASONAM, pages 1373–1380. ACM. find
  17. Mondher Bouazizi and Tomoaki Ohtsuki. 2015. Opinion mining in twitter how to make use of sarcasm to enhance sentiment analysis. In ASONAM, pages 1594–1597. ACM. find
  18. F. Bravo-Marquez, E. Frank, S. M. Mohammad, and B. Pfahringer. 2016. Determining word-emotion associations from tweets by multi-label classification. In WI, pages 536–539. IEEE. find
  19. Felipe Bravo-Marquez, Marcelo Mendoza, and Barbara Poblete. 2014. Meta-level sentiment models for big social data analysis. Knowledge-Based Systems, 69:86 – 99. find
  20. Reynier Ortega Bueno, Francisco M. Rangel Pardo, Delia Irazú Hernández Farı́as, Paolo Rosso, Manuel Montes y Gómez, and José Medina-Pagola. 2019. Overview of the task on irony detection in spanish variants. In IberLEF@SEPLN. find
  21. John D Campbell and Albert N Katz. 2012. Are there necessary conditions for inducing a sense of sarcastic irony? Discourse Processes, 49(6):459–480. find
  22. Sepp Hochreiter and Jrgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  23. Julia Jorgensen, George A Miller, and Dan Sperber. 1984. Test of the mention theory of irony. Journal of Experimental Psychology: General, 113(1):112– 120. find
  24. Aditya Joshi, Pushpak Bhattacharyya, Mark Carman, Jaya Saraswati, and Rajita Shukla. 2016a. How do cultural differences impact the quality of sarcasm annotation?: A case study of indian annotators and american text. In LaTeCH, pages 95–99. ACL. find
  25. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In IJCNLP, pages 757–762. ACL. find our card
  26. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016b. Are word embedding-based features useful for sarcasm detection? In EMNLP, pages 1006–1011. ACL. find our card
  27. David S. Kaufer. 1981. Understanding ironic communication. Journal of Pragmatics, 5(6):495 – 510. find
  28. Alessandra Teresa Cignarella, Simona Frenda, Valerio Basile, Cristina Bosco, Viviana Patti, Paolo Rosso, et al. 2018. Overview of the EVALITA 2018 task on irony detection in Italian tweets (IronITA). In Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (EVALITA 2018), volume 2263, pages 1–6. CEUR-WS. find
  29. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2018. A large self-annotated corpus for sarcasm. In LREC. ELRA. find our card
  30. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. ↗
  31. John S Leggitt and Raymond W Gibbs Jr. 2000. Emotional reactions to verbal irony. find
  32. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcasm in twitter and amazon. In CoNLL, pages 107–116. ACL. find our card
  33. Quoc Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. In ICML, pages 1188–1196. PMLR. find our card
  34. Christine Liebrecht, Florian Kunneman, and Antal Van den Bosch. 2013. The perfect solution for detecting sarcasm in tweets #not. In WASSA, pages 29–37. ACL. find our card
  35. Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In LREC. ELRA. find our card
  36. Saif Mohammad and Felipe Bravo-Marquez. 2017. WASSA-2017 shared task on emotion intensity. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 34–49. Association for Computational Linguistics. find
  37. Silviu Oprea and Walid Magdy. 2019. Exploring author context for detecting intended vs perceived sarcasm. In ACL, pages 2854–2859. ACL. detection with densely connected LSTM and multi-task learning. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 51– 56, New Orleans, Louisiana. Association for Computational Linguistics. find
  38. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480–1489, San Diego, California. Association for Computational Linguistics. find our card
  39. Silviu Oprea and Walid Magdy. 2020. The effect of sociocultural variables on sarcasm communication online. Proceedings of The 23rd ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW). find
  40. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In COLING, pages 213–223. ACL. find our card
  41. Antonio Reyes and Paolo Rosso. 2012. Making objective decisions from subjective data: Detecting irony in customer reviews. Decision Support Systems, 53(4):754–760. find
  42. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In EMNLP, pages 704–714. ACL. find our card
  43. Patricia Rockwell and Evelyn M. Theriot. 2001. Culture, gender, and gender mix in encoders of sarcasm: A self-assessment analysis. Communication Research Reports, 18(1):44–52. find
  44. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018. Disan: Directional self-attention network for rnn/cnn-free language understanding. In AAAI. AAAI Press. find
  45. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. In ACL, pages 1010–1020. ACL. find our card
  46. Mike Thelwall, Kevan Buckley, and Georgios Pal-toglou. 2012. Sentiment strength detection for the social web. J. Am. Soc. Inf. Sci. Technol., 63(1):163– 173. find
  47. Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In SemEval, pages 39–50. ACL. find our card
  48. Deirdre Wilson. 2006. The pragmatics of verbal irony: Echo or pretence? Lingua, 116(10):1722–1743. find
  49. Chuhan Wu, Fangzhao Wu, Sixing Wu, Junxin Liu, Zhigang Yuan, and Yongfeng Huang. 2018. THU NGN at SemEval-2018 task 3: Tweet irony find