Elamur Research⌕
☰
home / papers / 2013 / Really? Well. Apparently Bootstrapping Improves the Performance of Sarcasm and Nastiness Classifiers for Online Dialogue

Really? Well. Apparently Bootstrapping Improves the Performance of Sarcasm and Nastiness Classifiers for Online Dialogue

Summary

The paper asks whether a bootstrapping approach built for subjectivity detection in monologic news text (Riloff & Wiebe, R&W) can find sarcasm and nastiness in online debate dialogue, where annotated examples are rare and costly. The authors use Quote/Response pairs from the 4forums portion of the Internet Argument Corpus (IAC), which has Mechanical Turk labels. They first build lexical cue sets in two ways: by statistical selection (χ²) and by having crowd workers pick indicative phrases. These cues feed a high-precision first-stage classifier. The utterances it labels are then used to learn generalized syntactic extraction patterns in R&W's style. A pattern-based classifier built from those patterns is applied to held-out data.

Human-selected cues generalized better, while χ² cues seemed overfitted to the development data. Pattern-based bootstrapping raised both precision and recall for each phenomenon. The study shows that a news-oriented method transfers reasonably well to social dialogue. This points to a route for growing labeled corpora of subjective dialogue acts with little manual annotation.

Key points

  • Data: About 10,003 IAC Q/R pairs from 4forums were analyzed.
    • Roughly 12% are sarcastic and 12% are nasty.
    • Seven Turkers rated each pair: sarcasm as a binary judgment, and nasty/nice on a −5 to 5 scale.
    • Sarcasm was taken as a mean above 0.5, nasty as below −1, and nice as above 1.
  • Sarcasm results: The first-stage classifier with crowdsourced cues reached 54% precision and 38% recall. Bootstrapping with syntactic patterns raised this to 62% precision and 52% recall (F 0.57).
  • Nastiness results: The first stage gave 58% precision and 49% recall. After pattern bootstrapping, this rose to 75% precision and 62% recall (F 0.69). The authors read this as a sign that insults are easier to detect than sarcasm.
  • χ² cues underperformed: Pattern classifiers built on statistically selected cues did much worse, for example 31% precision for sarcasm and 30% for nastiness.
  • Future work: The authors suggest using semantic word categories such as LIWC instead of syntactic ones, running more bootstrapping iterations, and extending the method to other dialogue acts and domains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

30 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 16 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lukin2013really,
  title = {{Really? Well. Apparently Bootstrapping Improves the Performance of Sarcasm and Nastiness Classifiers for Online Dialogue}},
  author = {Lukin, Stephanie and Walker, Marilyn},
  booktitle = {Proceedings of the Workshop on Language Analysis in Social Media},
  year = {2013},
  url = {https://aclanthology.org/W13-1104/}
}

References

25 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G.A. Bryant and J.E. Fox Tree. 2002. Recognizing verbal irony in spontaneous speech. Metaphor and symbol, 17(2):99–119. ↗
  2. P. Carvalho, L. Sarmento, M.J. Silva, and E. de Oliveira. 2009. Clues for detecting irony in user-generated contents: oh...!! it’s so easy;-). In Proc. of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion, p. 53–56. ACM. find
  3. D. Davidov, O. Tsur, and A. Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proc. of the Fourteenth Conference on Computational Natural Language Learning, p. 107– 116. Association for Computational Linguistics. find our card
  4. J. Eisterhold, S. Attardo, and D. Boxer. 2006. Reactions to irony in discourse: Evidence for the least disruption principle. Journal of Pragmatics, 38(8):1239–1256. ↗
  5. E. Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Language Resources and Evaluation Conference, LREC2012. find our card
  6. J.E. Fox Tree and J.C. Schrock. 1999. Discourse Markers in Spontaneous Speech: Oh What a Difference an Oh Makes. Journal of Memory and Language, 40(2):280–295. ↗
  7. J. E. Fox Tree. 2010. Discourse markers across speakers and settings. Language and Linguistics Compass, 3(1):1–13. ↗
  8. P. Gianfortoni, D. Adamson, and C.P. Rosé. 2011. Modeling of stylistic variation in social media with stretchy patterns. In Proc. of the First Workshop on Algorithms and Resources for Modelling of Dialects and Language Varieties, p. 49–59. ACL. find
  9. R.W. Gibbs. 2000. Irony in talk among friends. Metaphor and Symbol, 15(1):5–27. ↗
  10. R. González-Ibáñez, S. Muresan, and N. Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proc. of the 49th Annual Meeting of the ACL: Human Language Technologies: short papers, volume 2, p. 581–586. find our card
  11. A. Razavi, D. Inkpen, S. Uritsky, and S. Matwin. 2010. Offensive language detection using multi-level classification. Advances in Artificial Intelligence, p. 16–27. find
  12. A. Reyes and P. Rosso. 2011. Mining subjective knowledge from customer reviews: a specific case of irony detection. In Proc. of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011), ACL, p. 118–124. find our card
  13. A. Reyes, P. Rosso, and D. Buscaldi. 2012. From humor recognition to irony detection: The figurative language of social media. Data & Knowledge Engineering. ↗
  14. E. Riloff and J. Wiebe. 2003. Learning extraction patterns for subjective expressions. In Proc. of the 2003 conference on Empirical methods in Natural Language Processing-V. 10, p. 105–112. ACL. find
  15. R. Snow, B. O’Conner, D. Jurafsky, and A.Y. Ng. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks In Proc. of the Conference on Empirical Methods in Natural Language Processing, p. 254–263. ACM. find
  16. S.O. Sood, E.F. Churchill, and J. Antin. 2011. Automatic identification of personal insults on social news sites. Journal of the American Society for Information Science and Technology. ↗
  17. Dan Sperber and Deidre Wilson. 1981. Irony and the use-mention distinction. In Peter Cole, editor, Radical Pragmatics, p. 295–318. Academic Press, N.Y. find
  18. E. Spertus. 1997. Smokey: Automatic recognition of hostile messages. In Proc. of the National Conference on Artificial Intelligence, p. 1058–1065. find
  19. M. Thelen and E. Riloff. 2002. A bootstrapping method for learning semantic lexicons using extraction pattern contexts. In Proc. of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, p. 214–221. ACL. find
  20. O. Tsur, D. Davidov, and A. Rappoport. 2010. Icwsm– a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In Proc. of the fourth international AAAI conference on weblogs and social media, p. 162–169. find
  21. Marilyn Walker, Pranav Anand, , Robert Abbott, and Jean E. Fox Tree. 2012. A corpus for research on deliberation and debate. In Language Resources and Evaluation Conference, LREC2012. find our card
  22. J.M. Wiebe, R.F. Bruce, and T.P. O’Hara. 1999. Development and use of a gold-standard data set for subjectivity classifications. In Proc. of the 37th annual meeting of the Association for Computational Linguistics, p. 246–253. ACL. find
  23. J. Wiebe, E. Breck, C. Buckley, C. Cardie, P. Davis, B. Fraser, D. Litman, D. Pierce, E. Riloff, T. Wilson, et al. 2003. Recognizing and organizing opinions expressed in the world press. In Working Notes-New Directions in Question Answering (AAAI Spring Symposium Series). find
  24. T. Wilson, P. Hoffmann, S. Somasundaran, J. Kessler, J. Wiebe, Y. Choi, C. Cardie, E. Riloff, and S. Patwardhan. 2005. Opinionfinder: A system for subjectivity analysis. In Proc. of HLT/EMNLP on Interactive Demonstrations, p. 34–35. ACL. find
  25. G. Xiang, B. Fan, L. Wang, J. Hong, and C. Rose. 2012. Detecting offensive tweets via topical feature discovery over a large scale twitter corpus. In Proc. of the 21st ACM international conference on Information and knowledge management, p. 1980–1984. ACM. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/lukin-2013-really-well-apparently-bootstrapping-improves/