Elamur Research⌕
☰
home / papers / 2014 / Getting Reliable Annotations for Sarcasm in Online Dialogues

Getting Reliable Annotations for Sarcasm in Online Dialogues

Summary

Sarcasm is common in online forum debates, but collecting trustworthy labels for it is difficult because the phenomenon is subjective and sometimes intentionally ambiguous. The authors study how to obtain dependable sarcasm judgments from untrained Mechanical Turk workers on Quote-Response pairs from the Internet Argument Corpus (IAC). They gathered up to 25 crowd annotations for 600 utterances, including items whose status was initially unclear. They then compared several ways of deriving a final label: Majority Class, Kappa-based weighting, EM (Dawid–Skene style), and Karger's iterative algorithm. They also tracked how labels shift as more annotators are added. The more sophisticated worker-reliability models gave no clear advantage over simple majority voting or vote-count thresholds. In most cases, labels settled after roughly 7–10 annotators. This gives practical guidance on annotation budgets for subjective dialogue phenomena. It also prompts a re-examination of the earlier two-vote threshold used in their own previous sarcasm classifier work (L&W).

Key points

  • Data source: The original IAC sarcasm annotation covered 10,003 Q-R pairs, with 5–7 Turkers per item and Krippendorff's α of only 0.23. The new, larger study reached α = 0.387.
  • Earlier threshold: The previous work (Lukin & Walker 2013) labelled a post SARC if at least 2 annotators marked it sarcastic. Karger's labelling proved considerably more conservative than this rule.
  • Accuracy on ambiguous items: For the 400 initially ambiguous items, accuracy against the final Karger labels was about 80% with 3 annotators and close to 90% with 10. Adding annotators beyond 10 brought only small gains.
  • Remaining ambiguity: After 25 annotations, only about 6–9 of the 600 utterances lacked a clear majority across the four methods.
  • Recommendation: Collect 3 annotations per item, and extend to about 7 when the first judgments disagree strongly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{swanson2014getting,
  title = {{Getting Reliable Annotations for Sarcasm in Online Dialogues}},
  author = {Swanson, Reid and Lukin, Stephanie and Eisenberg, Luke and Corcoran, Thomas and Walker, Marilyn},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/5g5wpyeg8tiv},
  url = {https://doi.org/10.63317/5g5wpyeg8tiv}
}

References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brown, P. and Levinson, S. (1987). Politeness: Some universals in language usage. Cambridge University Press. find our card
  2. Bryant, G. and Fox Tree, J. (2002). Recognizing verbal irony in spontaneous speech. Metaphor and symbol, 17(2):99–119. ↗
  3. Callison-Burch, C. (2009). Fast, cheap, and creative: Evaluating translation quality using Amazon’s Mechanical Turk. In Proc. of the 2009 Conf. on Empirical Methods in Natural Language Processing, p. 286–295. find our card
  4. Dagan, I., Glickman, O., and Magnini, B. (2006). The PASCAL recognising textual entailment challenge. Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, Lecture Notes in Computer Science 3944, p. 177–190. find
  5. Davidov, D., Tsur, O., and Rappoport, A. (2010). Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proc. of the Fourteenth Conf. on Computational Natural Language Learning, p. 107–116. find our card
  6. Dawid, A. P. and Skene, A. M. (1979). Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, p. 20–28. ↗
  7. Di Eugenio, B. and Glass, M. (2004). The kappa statistic: A second look. Computational linguistics, 30(1):95– 101. ↗
  8. Filatova, E. (2012). Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Language Resources and Evaluation Conf. , LREC2012. find our card
  9. Gibbs, R. (2000). Irony in talk among friends. Metaphor and Symbol, 15(1):5–27. ↗
  10. González-Ibáñez, R., Muresan, S., and Wacholder, N. (2011). Identifying sarcasm in twitter: a closer look. In Proc. of the 49th Annual Meeting of the Association for Computational Linguistics, volume 2, p. 581–586. find our card
  11. Karger, D. R., Oh, S., and Shah, D. (2011). Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing systems, p. 1953–1961. find
  12. Levinson, S. C. (1981). Some pre-observations on the modelling of dialogue. Discourse Processes, 4:93–116. ↗
  13. Levinson, S. C. (1985). What’s special about conversational inference. In 1987 Linguistics Institute Packet. find
  14. Liu, Q., Peng, J., and Ihler, A. (2012). Variational inference for crowdsourcing. Advances in Neural Information Processing Systems 25, p. 701–709. find
  15. Louis, A. and Nenkova, A. (2011). Automatic identification of general and specific sentences by leveraging discourse annotations. In IJCNLP, p. 605–613. find
  16. Lukin, S. and Walker, M. (2013). Really? Well. Apparently bootstrapping improves the performance of sarcasm and nastiness classifiers for online dialogue. find our card
  17. Language and Social Media Workshop, NAACL 2013, p. 30. find
  18. Riloff, E., Qadir, A., Surve, P., De Silva, L., Gilbert, N., and Huang, R. (2013). Sarcasm as contrast between a positive sentiment and negative situation. find our card
  19. Snow, R., O’Connor, B., Jurafsky, D., and Ng, A. (2008). Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proc. of the Conf. on Empirical Methods in Natural Language Processing, p. 254–263. find
  20. Walker, M., Anand, P., , Abbott, R., and Fox Tree, J. E. (2012). A corpus for research on deliberation and debate. In Language Resources and Evaluation Conf. , LREC2012. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/swanson-2014-getting-reliable-annotations-sarcasm-online/