Elamur Research
☰
home / papers / 2012 / Irony and Sarcasm: Corpus Generation and Analysis Using Crowdsourcing

Irony and Sarcasm: Corpus Generation and Analysis Using Crowdsourcing

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/L12-1386.pdf

Summary

Most earlier sarcasm detection work handled isolated sentences, specific phrases or short Twitter messages. This ignores the fact that irony often only becomes visible with broader context. Filatova addresses this gap by building a document-level corpus of Amazon product reviews through crowdsourcing on Mechanical Turk.

In the first step, workers located pairs of reviews of the same product, one sarcastic and one regular. For the sarcastic review they also supplied the text snippet(s) responsible for the irony and labeled it ironic, sarcastic or both. In a second step, other annotators re-judged the reviews and guessed each review's star rating from its plain text.

Analysis shows that sarcasm clusters in low-rated reviews, while regular reviews tend to be highly rated. Readers can also infer the reviewer's rating quite accurately whether or not the review is sarcastic.

The freely downloadable resource supports sarcasm research at two levels, whole documents and text utterances. It is useful for sentiment analysis and opinion mining.

Key points

  • Corpus of paired Amazon reviews (same product, sarcastic vs. regular) collected via Mechanical Turk. It contains roughly 437 sarcastic and 817 regular reviews, with annotator-supplied snippets explaining the sarcasm.
  • 59.94% of sarcastic reviews give one star, whereas 74% of regular reviews give five stars.
  • Five workers guessed each review's star rating. Their average correlated with the actual rating at 0.889 overall, 0.821 for sarcastic and 0.841 for regular reviews.
  • Many five-star sarcastic reviews target odd or novelty products, such as uranium ore and a short-sleeved T-shirt by The Mountain that shows three wolves and a moon.
  • The corpus is publicly available from the author's Fordham University page.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

59 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 49 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{filatova2012irony,
  title = {{Irony and Sarcasm: Corpus Generation and Analysis Using Crowdsourcing}},
  author = {Filatova, Elena},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  doi = {10.63317/2esatvw48hsm},
  url = {https://doi.org/10.63317/2esatvw48hsm}
}

References

30 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Julian Brooke and Matthew Hurst. 2009. Patterns in the stream: Exploring the interaction of polarity, topic, and discourse in a large opinion corpus. In Proceedings of the ACM Conference on Information and Knowledge Management, 1st International Workshop on Topic-Sentiment Analysis for Mass Opinion Measurement, Hong Kong, November. find
  2. Chris Callison-Burch and Mark Dredze. 2010. Creating speech and language data with Amazon’s Mechanical Turk. In Proceeding of the NAACL workshop on Creating Speech and Language Data With Mechanical Turk (NAACL–HLT 2010), Los Angeles, CA, USA, June. find
  3. Paula Carvalho, Luı́s Sarmento, Mário J. Silva, and Eugénio de Oliveira. 2009. Clues for detecting irony in user-generated contents: Oh...!! It’s “so easy” ;-). In Proceeding of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion, TSA ’09, pages 53–56, New York, NY, USA. ACM. find
  4. Herbert H. Clark and Richard J. Gerrig. 1984. On the pretense theory of irony. Journal of Experimental Psychology: General, 113(1):121–126. find
  5. Marlena Creusere. 1999. Theories of adults understanding and use of irony and sarcasm: Applications to and evidence from research with children. Developmental Review, 19:213–262. find
  6. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in Twitter and Amazon. In Proceeding of Computational Natural Language Learning (CoNLL 2010), Uppsala, Sweden, July. find our card
  7. Shelly Dews and Ellen Winner. 1995. Muting the meaning: A social function of irony. Metaphor and Symbolic Activity, 10(1):3–19. find
  8. Megan Dress, Roger Kreuz, Kristen Link, and Gina Caucci. 2008. Regional variation in the use of sarcasm. Journal of Language and Social Psychology, 27(1):71–85. find
  9. Anindya Ghose, Panagiotis G. Ipeirotis, and Arun Sundararajan. 2007. Opinion mining using econometrics: A case study on reputation systems. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL 2007), Prague, Czech Republic, July. find
  10. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in Twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (), pages 581–586, Portland, Oregon, USA, June. Association for Computational Linguistics. find our card
  11. Paul Grice. 1975. Logic and conversation. Syntax and Semantics, 3: Speech Acts:41–58. find
  12. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining (KDD 2004), pages 168–177, Seattle, WA, USA, August. find
  13. Panagiotis Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on Amazon Mechanical Turk. In Proceedings of the Second KDD Human Computation Workshop (KDD-HCOMP 2010), Washington DC, USA, July. find
  14. Roger Kreuz and Gina Caucci. 2007. Lexical influences on the perception of sarcasm. In Proceedings of the NAACL workshop on Computational Approaches to Figurative Language (HLT–NAACL 2007), Rochester, NY, USA, April. find
  15. Sachi Kumon-Nakamura, Sam Glucksberg, and Mary Brown. 1995. How about another piece of pie: The allusional pretense theory of discourse irony. Journal of Experimental Psychology: General, 124(1):3–21. find
  16. David C. Littman and Jacob L. Mey. 1991. The nature of irony: Toward a computational model of irony. Journal of Pragmatics, 15(2):131–151. find
  17. Joan Lucariello. 1994. Situational irony: A concept of events gone awry. Journal of Experimental Psychology: General, 123(2):129–145. find
  18. Douglas Colin Muecke. 1970. Irony and the ironic. The Critical Idiom, 13. find
  19. Alice R. Myers. 1977. Toward a definition of irony. In Ralph W. Fasold and Roger W. Shuy, editors, Studies in language variation: semantics, syntax, phonology, pragmatics, social situations, ethnographic approaches, pages 171–183. Georgetown University Press. Constantine Nakassis and Jesse Snedeker. 202. Beyond sarcasm: Intonation and context as relational cues in children’s recognition of irony. In Proceedings of the Twenty-sixth Boston University Conference on Language Development, Boston, MA, USA, July. find
  20. Geoffrey Nunberg. 2001. The Way we Talk Now: Commentaries on Language and Culture. Boston: Houghton Mifflin. find
  21. Bo Pang and Lillian Lee. 2008. Opinion Mining and Sentiment Analysis., volume 2. Foundations and Trends R in Information Retrieval. find
  22. Gabriele Paolacci, Jesse Chandler, and Panagiotis G. Ipeirotis. 2010. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), August. find
  23. Ana-Maria Popescu and Oren Etzioni. 2005. Extracting product features and opinions from reviews. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing (HLT–EMNLP 2005), pages 339–346, Vancouver, B.C., Canada, October. Morristown, NJ, USA: Association for Computational Linguistics. find
  24. Luı́s Sarmento, Paula Carvalho, Mário J. Silva, and Eugénio de Oliveira. 2009. Automatic creation of a reference corpus for political opinion mining in user-generated content. find
  25. Dan Sperber and Deirdre Wilson. 1981. Irony and the use-mention distinction. Radical pragmatics, pages 295– 318. find
  26. Joseph Tepperman, David Traum, and Shrikanth S. Narayanan. 2006. ”Yeah right”: Sarcasm recognition for spoken dialogue systems. In Proceedings of InterSpeech, pages 1838–1841, Pittsburgh, PA, USA, September. find
  27. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. ICWSM - A great catchy name: Semi-supervised recognition of sarcastic sentences in product reviews. In Proceeding of AAAI Conference on Weblogs and Social Media (ICWSM–10), Washington, DC, USA, May. find
  28. Akira Utsumi. 1996. A unified theory of irony and its computational formalization. In Proceedings of the 16th conference on Computational linguistics (COLING 1996), pages 962–967, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  29. Janyce Wiebe, Theresa Wilson, Rebecca Bruce, Matthew Bell, and Melanie Martin. 2004. Learning subjective language. Computational Linguistics, 30:277–308. find
  30. Deirdre Wilson and Dan Sperber. 2004. Relevance theory. In L. R. Horn and G. Ward, editors, The Handbook of Pragmatics, pages 607–632. Oxford: Blackwell. find