Elamur Research
☰
home / papers / 2016 / Detecting Sarcasm in Multimodal Social Platforms

Detecting Sarcasm in Multimodal Social Platforms

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1608.02289

Summary

Most sarcasm detectors analyze only text, so they miss cases where an attached image reveals that a caption means the opposite of what it says. The authors use images as situational context for multimodal posts. They gathered English posts tagged #sarcasm or #sarcastic from Instagram, Tumblr and Twitter, removing retweets and reblogs. They then compared how text and images are used on each platform, proposed a typology of the roles images play in sarcastic posts, and ran a crowdsourcing study to measure how often annotators need the picture to recognize sarcasm. This produced gold sets at several agreement levels.

Two fusion frameworks were tested:

  • Visual semantics features (VSF), learned on an external dataset and combined with strong textual features in an SVM.
  • Deep network adaptation (DNA) of an ImageNet-initialized CNN, fine-tuned on text+image posts.

On both noisy (silver) and curated (gold) data, adding visual information generally improved on text-only baselines, with Instagram benefiting most. This is presented as the first computational sarcasm model that uses computer vision.

Key points

  • Raw corpus size: 517,229 Instagram, 63,067 Tumblr and 20,629 Twitter posts. Text and images appear together in 99.74%, 40.22% and 7.56% of them, respectively.
  • DNA on the silver set: text+image fusion achieved the best accuracy, from 74.2% (Instagram) down to 69.7% (Twitter). Twitter's weaker result is linked to having only about 2,000 training posts.
  • Image-only DNA vs. VSF: the image-only network outperformed VSF on every platform: 73.8 vs. 68.8 (IG), 69.2 vs. 65.7 (TU), and 68.7 vs. 61.7 (TW).
  • Gold sets (D-50/D-80/D-100):
    • Instagram: 1-grams + AVR reached 77.8, 78.4 and 77.6.
    • Tumblr: fusion led on D-50 and D-80 but trailed AVR alone on D-100 (74.7 vs. 75.5).
  • Limitations: the DNA text branch was weaker than the SVM text models because it lacks bigrams and richer features. Overfitting on small training sets and GPU memory limits prevented adding those features. The human annotations are to be released to the community.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

52 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 45 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/2964284.2964321 · arXiv 1608.02289

@misc{schifanella2016detecting,
  title = {{Detecting Sarcasm in Multimodal Social Platforms}},
  author = {Schifanella, Rossano and Juan, Paloma de and Tetreault, Joel and Cao, Liangliang},
  year = {2016},
  doi = {10.1145/2964284.2964321},
  eprint = {1608.02289},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/2964284.2964321}
}

References

38 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Attardo, J. Eisterhold, J. Hay, and I. Poggi. Multimodal markers of irony and sarcasm. Humor-international Journal of Humor Research, 16:243–260, 2003. find
  2. D. Bamman and N. A. Smith. Contextualized sarcasm detection on twitter. In M. Cha, C. Mascolo, and C. Sandvig, editors, Proc. of the Ninth Int. Conference on Web and Social Media, ICWSM, pages 574–577. AAAI Press, 2015. find
  3. F. Barbieri, H. Saggion, and F. Ronzano. Modelling sarcasm in twitter, a novel approach. In Proc. of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 50–58, Baltimore, Maryland, June 2014. Association for Computational Linguistics. find our card
  4. D. Borth, T. Chen, R. Ji, and S.-F. Chang. Sentibank: Large-scale ontology and classifiers for detecting sentiment and emotions in visual content. In Proc. of the ACM Int. Conference on Multimedia, MM ’13, pages 459–460, New York, NY, USA, 2013. ACM. find
  5. K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: delving deep into convolutional nets. In BMVC, 2014. find
  6. D. Davidov, O. Tsur, and A. Rappoport. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proc. of the Conference on Computational Natural Language Learning, CoNLL ’10, pages 107–116, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics. find our card
  7. M. Faruqui and C. Dyer. Non-distributional word vector representations. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), volume 2, pages 464–469, Beijing, China, July 2015. Association for Computational Linguistics. find
  8. A. Frome, G. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov. Devise: A deep visual-semantic embedding model. In Advances In Neural Information Processing Systems, NIPS, 2013. find
  9. D. Ghosh, W. Guo, and S. Muresan. Sarcastic or not: Word embeddings to predict the literal or sarcastic meaning of words. In L. Màrquez, C. Callison-Burch, J. Su, D. Pighin, and Y. Marton, editors, EMNLP, pages 1003–1012. The Association for Computational Linguistics, 2015. find
  10. D. Ghosh, W. Guo, and S. Muresan. Sarcastic or not: Word embeddings to predict the literal or sarcastic meaning of words. In Proc. of the Conference on Empirical Methods in Natural Language Processing, pages 1003–1012, Lisbon, Portugal, September 2015. Association for Computational Linguistics. find
  11. R. Gibbs. The Cambridge Handbook of Metaphor and Thought. Cambridge Handbooks in Psychology. Cambridge University Press, 2008. find
  12. R. Gibbs and H. Colston. Irony in Language and Thought: A Cognitive Science Reader. Lawrence Erlbaum Associates, 2007. find
  13. R. González-Ibáñez, S. Muresan, and N. Wacholder. Identifying sarcasm in twitter: A closer look. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), volume 2 of HLT ’11, pages 581–586, Stroudsburg, PA, USA, 2011. Association for Computational Linguistics. find our card
  14. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014. ↗ our card
  15. T. Joachims. Text categorization with suport vector machines: Learning with many relevant features. In Proceedings of the 10th European Conference on Machine Learning, ECML ’98, pages 137–142, London, UK, UK, 1998. Springer-Verlag. find
  16. A. Joshi, P. Bhattacharyya, and M. J. Carman. Automatic sarcasm detection: A survey. CoRR, abs/1602.03426, 2016. ↗ our card
  17. A. Joshi, V. Sharma, and P. Bhattacharyya. Harnessing context incongruity for sarcasm detection. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), volume 2, pages 757–762. The Association for Computer Linguistics, 2015. find our card
  18. S. Karayev, A. Hertzmann, H. Winnemoeller, A. Agarwala, and T. Darrell. Recognizing image style. BMVC, 2014. find
  19. A. Khattri, A. Joshi, P. Bhattacharyya, and M. Carman. Your sentiment precedes you: Using an author’s historical tweets to predict sarcasm. In Proc. of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 25–30, Lisboa, Portugal, September 2015. Association for Computational Linguistics. find
  20. J. P. Kincaid, R. P. Fishburne Jr., R. L. Rogers, and B. S. Chissom. Derivation of new readability formulas (automated readability index, fog count and Flesch reading ease formula) for navy enlisted personnel. Technical report, DTIC Document, 1975. find
  21. R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. CoRR, abs/1411.2539, 2014. ↗ our card
  22. J. R. Landis and G. G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33(1), 1977. find
  23. L. Ma, Z. Lu, L. Shang, and H. Li. Multimodal convolutional neural networks for matching image and sentence. CoRR, abs/1504.06063, 2015. ↗
  24. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). CoRR, abs/1412.6632, 2014. ↗
  25. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119, 2013. find
  26. E. Pavlick and A. Nenkova. Inducing lexical style properties for paraphrase and genre differentiation. In Proc. of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 218–224, Denver, Colorado, May–June 2015. Association for Computational Linguistics. find
  27. A. Rajadesingan, R. Zafarani, and H. Liu. Sarcasm detection on twitter: A behavioral modeling approach. In Proc. of the ACM Int. Conference on Web Search and Data Mining, WSDM ’15, pages 97–106, New York, NY, USA, 2015. ACM. find
  28. A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. CNN features off-the-shelf: an astounding baseline for recognition. CVPR DeepVision workshop, 2014. find
  29. A. Reyes, P. Rosso, and T. Veale. A multidimensional approach for detecting irony in twitter. Lang. Resour. Eval., 47(1):239–268, Mar. 2013. find
  30. E. Riloff, A. Qadir, P. Surve, L. D. Silva, N. Gilbert, and R. Huang. Sarcasm as contrast between a positive sentiment and negative situation. In EMNLP, pages 704–714. ACL, 2013. find our card
  31. E. Shutova, D. Kiela, and J. Maillard. Black holes and white rabbits: Metaphor identification with visual features. In Proc. of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 160–170, San Diego, California, June 2016. Association for Computational Linguistics. find
  32. J. Tepperman, D. Traum, and S. S. Narayanan. ”yeah right”: Sarcasm recognition for spoken dialogue systems. In Proc. of InterSpeech, pages 1838–1841, Pittsburgh, PA, Sept. 2006. find
  33. B. Thomee, B. Elizalde, D. A. Shamma, K. Ni, G. Friedland, D. Poland, D. Borth, and L.-J. Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016. find
  34. O. Tsur, D. Davidov, and A. Rappoport. Semi-Supervised Recognition of Sarcastic Sentences in Online Product Reviews. In M. Hearst, W. Cohen, and S. Gosling, editors, Proc. of the Int. Conference on Weblogs and Social Media (ICWSM-2010). The AAAI Press, Menlo Park, California, 2010. find
  35. T. Veale, K. Feyaerts, and C. Forceville. Creativity and the Agile Mind: A Multi-Disciplinary Study of a Multi-Faceted Phenomenon. Applications of Cognitive Linguistics [ACL]. De Gruyter, 2013. find
  36. P. Verstraten. Film Narratology : film narratives his primary focus, while noting. University of Toronto Press, 2006. find
  37. Z. Wang, Z. Wu, R. Wang, and Y. Ren. Twitter sarcasm detection exploiting a context-based model. In J. Wang, W. Cellary, D. Wang, H. Wang, S. Chen, T. Li, and Y. Zhang, editors, Web Information Systems Engineering - WISE 2015 - 16th Int. Conference, Miami, FL, USA, November 1-3, 2015, Proc., Part I, volume 9418 of Lecture Notes in Computer Science, pages 77–91. Springer, 2015. find
  38. Q. You, J. Luo, H. Jin, and J. Yang. Robust Image Sentiment Analysis using Progressively Trained and Domain Transferred Deep Networks. In AAAI Conference on Artificial Intelligence (AAAI), 2015. find