Elamur Research⌕
☰
home / papers / 2018 / Sarcasm Target Identification: Dataset and An Introductory Approach

Sarcasm Target Identification: Dataset and An Introductory Approach

Summary

The paper introduces a new task in computational sarcasm. Instead of deciding whether a text is sarcastic, the goal is to find what it mocks. Given a sarcastic text, a system must return the words naming the ridiculed entity or situation. If no such words exist, it returns the fallback label 'Outside'. To benchmark the task, the authors built a dataset of 224 sarcastic book snippets and 506 sarcastic tweets, each manually labeled with its target(s). They also propose a hybrid system that combines two extractors: a set of nine hand-written rules and a per-word statistical classifier. An integrator picks the final target from the candidates each extractor proposes. The system is compared against two baselines: a simple heuristic and a method borrowed from opinion target extraction. The authors argue the results show the task is workable, and they present the system as a reference point for later research. The task matters for aspect-based sentiment analysis, which needs to assign the negative sentiment hidden in sarcasm to the right entity. It could also help dialogue and generation systems respond more appropriately.

Key points

  • Dataset sources: book snippets come from Joshi et al. (2016b) and tweets from Riloff et al. (2013).
    • The released data is at github.com/Pranav-Goel/Sarcasm-Target-Detection.
    • Targets average 1.6 words in snippets and 2.08 words in tweets.
  • Targets carry little sentiment: measured with the McAuley & Leskovec word list, targets are more neutral than the rest of the text.
    • Snippets: 0.0087 for targets vs 0.027 for the rest.
    • Tweets: 0.035 for targets vs 0.53 for the rest.
  • Annotation: three experienced annotators produced the gold labels.
    • They were told to prefer concrete entities over situations.
    • When a text had several targets, all of them were labeled.
  • Human agreement is low: two extra annotators labeled 50 tweets and 50 snippets.
    • Exact match was 28% for snippets and 38% for tweets.
    • Partial overlap was 66% for both.
    • This suggests the task is hard even for people.
  • Main challenges: texts often contain several candidate phrases, some have more than one target, and some have no explicit target word (the 'Outside' case).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{joshi2018sarcasm,
  title = {{Sarcasm Target Identification: Dataset and An Introductory Approach}},
  author = {Joshi, Aditya and Goel, Pranav and Bhattacharyya, Pushpak and Carman, Mark},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  doi = {10.63317/2xck5kccxygt},
  url = {https://doi.org/10.63317/2xck5kccxygt}
}

References

26 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Altun, Y., Tsochantaridis, I., and Hofmann, T. (2003). Hidden markov support vector machines. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 3–10. find
  2. Bird, S. (2006). Nltk: the natural language toolkit. In Proceedings of the COLING/ACL on Interactive presentation sessions, pages 69–72. Association for Computational Linguistics. find
  3. Campbell, J. D. and Katz, A. N. (2012). Are there necessary conditions for inducing a sense of sarcastic irony? Discourse Processes, 49(6):459–480. ↗
  4. Davidov, D., Tsur, O., and Rappoport, A. (2010). Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the fourteenth conference on computational natural language learning, pages 107– 116. Association for Computational Linguistics. find our card
  5. González-Ibánez, R., Muresan, S., and Wacholder, N. (2011). Identifying sarcasm in twitter: a closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers-Volume 2, pages 581–586. Association for Computational Linguistics. find our card
  6. Jin, W., Ho, H. H., and Srihari, R. K. (2009). Opinionminer: a novel machine learning system for web opinion mining and extraction. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1195–1204. ACM. find
  7. Joachims, T. (2006). Training linear svms in linear time. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 217–226. ACM. find
  8. Joshi, A., Sharma, V., and Bhattacharyya, P. (2015). Harnessing context incongruity for sarcasm detection. In ACL (2), pages 757–762. find our card
  9. Joshi, A., Bhattacharyya, P., and Carman, M. J. (2016a). Automatic sarcasm detection: A survey. arXiv preprint arXiv:1602.03426. ↗ our card
  10. Joshi, A., Tripathi, V., Patel, K., Bhattacharyya, P., and Carman, M. (2016b). Are word embedding-based features useful for sarcasm detection? arXiv preprint arXiv:1610.00883. ↗ our card
  11. Khattri, A., Joshi, A., Bhattacharyya, P., and Carman, M. (2015). Your sentiment precedes you: Using an author’s historical tweets to predict sarcasm. In Proceedings of the 6th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 25– 30. ↗ our card
  12. Lu, B., Ott, M., Cardie, C., and Tsou, B. K. (2011). Multi-aspect sentiment analysis with topic models. In Data Mining Workshops (ICDMW), 2011 IEEE 11th International Conference on, pages 81–88. IEEE. find
  13. McAuley, J. J. and Leskovec, J. (2013). From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In Proceedings of the 22nd international conference on World Wide Web, pages 897– 908. ACM. ↗
  14. Michelson, M. and Knoblock, C. A. (2007). Unsupervised information extraction from unstructured, ungrammatical data sources on the world wide web. International Journal on Document Analysis and Recognition, 10(3):211–226. ↗
  15. Mukherjee, A. and Liu, B. (2012). Aspect extraction through semi-supervised modeling. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 339– 348. Association for Computational Linguistics. find
  16. Pang, B. and Lee, L. (2005). Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 115– 124. Association for Computational Linguistics. ↗
  17. Pang, B., Lee, L., et al. (2008). Opinion mining and sentiment analysis. Foundations and Trends R in Information Retrieval, 2(1–2):1–135. ↗
  18. Qiu, G., Liu, B., Bu, J., and Chen, C. (2011). Opinion word expansion and target extraction through double propagation. Computational linguistics, 37(1):9–27. ↗ our card
  19. Rajadesingan, A., Zafarani, R., and Liu, H. (2015). Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pages 97– 106. ACM. find
  20. Riloff, E., Qadir, A., Surve, P., De Silva, L., Gilbert, N., and Huang, R. (2013). Sarcasm as contrast between a positive sentiment and negative situation. In EMNLP, volume 13, pages 704–714. find our card
  21. Shamay-Tsoory, S. G., Tomer, R., and Aharon-Peretz, J. (2005). The neuroanatomical basis of understanding sarcasm and its relationship to social cognition. Neuropsychology, 19(3):288. ↗
  22. Sørensen, T. (1948). A method of establishing groups of equal amplitude in plant sociology based on similarity of species and its application to analyses of the vegetation on danish commons. Biol. Skr., 5:1–34. find
  23. Tan, C., Lee, L., Tang, J., Jiang, L., Zhou, M., and Li, P. (2011). User-level sentiment analysis incorporating social networks. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1397–1405. ACM. find
  24. Titov, I. and McDonald, R. (2008). A joint model of text and aspect ratings for sentiment summarization. proceedings of ACL-08: HLT, pages 308–316. find
  25. Tsur, O., Davidov, D., and Rappoport, A. (2010). Icwsm-a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In ICWSM, pages 162–169. find
  26. Veale, T. and Hao, Y. (2010). Detecting ironic intent in creative comparisons. In ECAI, volume 215, pages 765– 770. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/joshi-2018-sarcasm-target-identification-dataset-introductory/