Elamur Research⌕
☰
home / papers / 2015 / Sarcastic or Not: Word Embeddings to Predict the Literal or Sarcastic Meaning of Words

Sarcastic or Not: Word Embeddings to Predict the Literal or Sarcastic Meaning of Words

Summary

The authors recast sarcasm detection as a word sense disambiguation problem. Instead of labeling whole utterances, they ask whether a particular target word in a tweet carries its literal or its sarcastic sense. They refer to this formulation as LSSD, short for Literal/Sarcastic Sense Disambiguation.

To find words that can flip meaning, they had MTurk workers rewrite hashtag-labeled sarcastic tweets so that each rewrite stated what the author actually meant. They then aligned the originals with the rewrites using unsupervised methods (Barzilay & McKeown co-training, filtered with Giza++/Moses translation scores) to extract semantically opposite word pairs.

For each target word, they built training data from Twitter: sarcastic tweets marked with #sarcasm or #sarcastic, and literal tweets that were either unmarked or carried sentiment hashtags. They compared several distributional semantics approaches. An SVM with a modified kernel based on word embeddings performed best, beating a strong lexical SVM baseline by roughly 7–10% F1.

This matters because sentiment systems that miss sarcasm misread opinions. The released datasets support further work on the problem.

Key points

  • Crowdsourcing step: 1,000 sarcastic tweets, each rephrased by 5 Turkers, produced 5,000 SM-IM pairs. Co-training extracted 367 paraphrase pairs. Filtering at φ ≥ 0.8 plus manual checks left 80 opposite pairs, which gave 70 target words after lemmatization.
  • Data collection: about 2.54M tweets were gathered via the Twitter API with an 80/10/10 split. Requiring at least 400 instances per sense left 37 usable targets, ranging from "love" (26,802) to "mature" (427).
  • Best F1 was 97% for S vs. L but only 84% for S vs. L_sent. This confirms that literal tweets carrying sentiment are much harder to separate from sarcastic ones.
  • Gains over the SVM baseline were 10% F1 (S vs. L) and 8% F1 (S vs. L_sent). Embedding methods held up well even on targets with little training data.
  • Pre-trained GloVe vectors (2B tweets, 100 dimensions) averaged 88.9% F1 on S vs. L, about 7% below embeddings trained on the task's own tweets. Data is published at github.com/debanjanghosh/sarcasm_wsd.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Word sense disambiguation

Cited by

25 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 10 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1116

@inproceedings{ghosh2015sarcastic,
  title = {{Sarcastic or Not: Word Embeddings to Predict the Literal or Sarcastic Meaning of Words}},
  author = {Ghosh, Debanjan and Guo, Weiwei and Muresan, Smaranda},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1116},
  url = {https://doi.org/10.18653/v1/d15-1116}
}

References

28 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 597–604. Association for Computational Linguistics. find our card
  2. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find our card
  3. Regina Barzilay and Kathleen R McKeown. 2001. Extracting paraphrases from a parallel corpus. In Proceedings of the 39th Annual Meeting on Association for Computational Linguistics, pages 50–57. Association for Computational Linguistics. find our card
  4. Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27. Software available at http://www.csie.ntu.edu.tw/˜cjlin/libsvm. ↗
  5. Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational linguistics, 16(1):22–29. ↗ our card
  6. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. find our card
  7. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL ’10. find our card
  8. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 42–47. Association for Computational Linguistics. ↗
  9. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: A closer look. In ACL (Short Papers), pages 581–586. Association for Computational Linguistics. find our card
  10. Weiwei Guo and Mona Diab. 2012a. Learning the latent semantics of a concept from its definition. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2, pages 140–144. Association for Computational Linguistics. ↗
  11. Weiwei Guo and Mona Diab. 2012b. Modeling sentences in the latent space. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 864–872. Association for Computational Linguistics. ↗
  12. Zellig S Harris. 1954. Distributional structure. Word, 10:146–162. find
  13. Aminul Islam and Diana Inkpen. 2008. Semantic text similarity using corpus-based word similarity and string similarity. ACM Transactions on Knowledge Discovery from Data (TKDD), 2(2):10. ↗
  14. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757–762, Beijing, China, July. Association for Computational Linguistics. find our card
  15. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions, pages 177–180. Association for Computational Linguistics. find
  16. Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3:211–225. ↗
  17. CC Liebrecht, FA Kunneman, and APJ van den Bosch. 2013. The perfect solution for detecting sarcasm in tweets# not. find our card
  18. Diana Maynard and Mark A Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of LREC. find our card
  19. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  20. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168. ↗
  21. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013c. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. ↗
  22. Franz Josef Och and Hermann Ney. 2000. Giza++: Training of statistical translation models. find
  23. James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates, 71:2001. find
  24. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), 12. find
  25. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in twitter. Language resources and evaluation, 47(1):239–268. ↗
  26. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 704–714. Association for Computational Linguistics. find our card
  27. Simo Tchokni, Diarmuid O Séaghdha, and Daniele Quercia. 2014. Emoticons and phrases: Status symbols in social media. In Eighth International AAAI Conference on Weblogs and Social Media. find
  28. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698. ↗ our card

↑ back to top

This page in Markdown →