Elamur Research
☰
home / papers / 2018 / A Large Self-Annotated Corpus for Sarcasm

A Large Self-Annotated Corpus for Sarcasm

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/L18-1102.pdf

Summary

Sarcasm is hard to detect because it is rare and even people often miss it. Earlier datasets were small or artificially balanced, were labelled by outside annotators, or were made of short tweets. The authors built SARC (Self-Annotated Reddit Corpus) from Reddit comments posted between January 2009 and April 2017. Reddit users often end a sarcastic remark with "/s", so the authors treat that marker as a label added by the writer. To reduce noise, they keep only authors who had already used the marker. The tag must sit at the very end of the comment. They also drop replies nested under sarcastic comments and apply standard cleaning, such as removing URLs and keeping only ASCII text. Each comment comes with its author, subreddit, score, date and conversation context. In a manual check of 1,000 comments, about 1% were false positives and 2% were false negatives. The authors also built balanced and unbalanced benchmark subsets and compared simple classifiers with human raters. Bag-of-n-gram models did better than sentence embeddings, but no model reached human performance. SARC is about ten times larger than earlier sarcasm resources.

Key points

  • The raw corpus has about 533M comments, including roughly 1.34M sarcastic ones. That is about 10× more sarcastic examples than any earlier dataset (Twitter, IAC, or Wallace et al. 2015 on Reddit).
  • The data comes in three files: a CSV of labelled comments with their parent comments, a JSON hashtable of the conversation threads around sarcastic comments, and a CSV that links context sequences to labelled replies.
  • In the manual audit, the false negative rate (2.0%) is large compared with the overall share of sarcasm (0.25%). Label noise is therefore a real problem in the unbalanced setting.
  • In a July 2014 sample, 0.927% of Reddit authors used a sarcasm marker, compared with 0.002% of Twitter authors.
  • Human raters agreed moderately on the full benchmark (Fleiss kappa 0.5) and more strongly on the politics subset (0.67). A majority vote of human raters beat all the baseline models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Treebanks

Cited by

87 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 80 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1704.05579

@inproceedings{khodak2018large,
  title = {{A Large Self-Annotated Corpus for Sarcasm}},
  author = {Khodak, Mikhail and Saunshi, Nikunj and Vodrahalli, Kiran},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  doi = {10.63317/2ifwj74dgdbw},
  eprint = {1704.05579},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.63317/2ifwj74dgdbw}
}

References

18 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amir, S., Wallace, B. C., Lyu, H., Carvalho, P., and Silva, M. J. (2016). Modelling context with user embeddings for sarcasm detection in social media. find our card
  2. Arora, S., Liang, Y., and Ma, T. (2017). A simple but tough-to-beat baseline for sentence embeddings. Conference Paper at ICLR 2017. find
  3. Bamman, D. and Smith, N. A. (2015). Contextualized sarcasm detection on twitter. Association for the Advancement of Artificial Intelligence. find
  4. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378– 382. find
  5. González-Ibánez, R., Muresan, S., and Wacholder, N. (2011). Identifying sarcasm in twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 581–586. Association for Computational Linguistics. find our card
  6. Joshi, A., Sharma, V., and Bhattacharyya, P. (2015). Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 757–762. Association for Computational Linguistics. find our card
  7. Joshi, A., Bhattacharyya, P., and Carman, M. J. (2016). Automatic sarcasm detection: A survey. arXiv. find our card
  8. Lukin, S. and Walker, M. (2013). Really? well. apparently bootstrapping improves the performance of sarcasm and nastiness classifiers for online dialogue. In Proceedings of the Workshop on Language in Social Media, pages 30– 40. Association for Computational Linguistics. find
  9. McAuley, J., Pandey, R., and Leskovec, J. (2015). Inferring networks of substitutable and complementary products. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. find
  10. Oraby, S., Harrison, V., Reed, L., Hernandez, E., Riloff, E., and Walker, M. (2016). Creating and characterizing a diverse corpus of sarcasm in dialogue. In Proceedings of the SIGDIAL 2016 Conference, pages 31–41. Association for Computational Linguistics. find our card
  11. Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543. Association for Computational Linguistics. find
  12. Ptác̆ek, T., Habernal, I., and Hong, J. (2014). Sarcasm detection on czech and english twitter. In 25th International Conference on Computational Linguistics: Technical Papers, pages 213–223. find our card
  13. Reyes, A., Rosso, P., and Veale, T. (2013). A multidimensional approach for detecting irony in twitter. Data Knowledge Engineering, 47(1). find
  14. Riloff, E., Qadir, A., Surve, P., Silva, L. D., Gilbert, N., and Huang, R. (2013). Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–741. Association for Computational Linguistics. find our card
  15. Swanson, R., Lukin, S., Eisenberg, L., Corcoran, T. C., and Walker, M. A. (2014). Getting reliable annotations for sarcasm in online dialogues. In Language Resources and Evaluation Conference. find
  16. Walker, M. A., Anand, P., Tree, J. E. F., Abbott, R., and King, J. (2012). A corpus for research on deliberation and debate. In Language Resources and Evaluation Conference. find
  17. Wallace, B. C., Choe, D. K., Kertz, L., and Charniak, E. (2014). Humans require context to infer ironic intent (so computers probably do, too. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 512–516. Association for Computational Linguistics. find
  18. Wallace, B. C., Choe, D. K., and Charniak, E. (2015). Sparse, contextually informed models for irony detection: Exploiting user communities, entities, and sentiment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 1035– 1044. Association for Computational Linguistics. find