Elamur Research⌕
☰
home / papers / 2016 / Harnessing Sequence Labeling for Sarcasm Detection in Dialogue from TV Series ‘Friends'

Harnessing Sequence Labeling for Sarcasm Detection in Dialogue from TV Series ‘Friends'

Summary

The paper looks at sarcasm in multi-party conversation. In dialogue, an utterance often reads as ironic only because of what came before it. Earlier work treated each sentence as a separate classification instance and added context as extra features. Instead, the authors frame the task as sequence labeling: a whole scene goes into the model, and it outputs a sarcastic or non-sarcastic tag for every utterance.

To test this, they built a manually annotated corpus from transcripts of the sitcom 'Friends', taken from OpenSubtitles. They used three feature sets: their own lexical, conversational-context and speaker features, plus features from two earlier sarcasm studies. With each set, they compared two sequence labelers (SVM-HMM, SEARN) against three classifiers (SVM with oversampling, SVM with undersampling, Naïve Bayes).

The sequence labelers came out ahead for every feature set, mainly because their recall was much higher. This suggests that models which respect turn order are a better fit for sarcasm detection in conversational data such as chats or tweet threads.

Key points

  • Dataset: 913 scenes containing 17,338 utterances, of which 1,888 are labeled sarcastic. Scenes average 18.6 utterances. Two linguists who had never seen the show did the labeling. Their agreement on a 105-scene subset was 0.44. Chandler is the most sarcastic character, at 22.24% of his lines.
  • Best scores with the authors' own features: SVM-HMM reached an F-score of 84.2%, while the strongest classifier, oversampled SVM, reached 79.8%.
  • Features from prior work: switching to sequence labeling raised F-score by 2.8% with the Buschmeier et al. (2014) features and by 4% with the González-Ibáñez et al. (2011) features. Recall rose by roughly 10% and 12% respectively.
  • Manual analysis: the authors examined 100 utterances that the sequence labelers got right but the classifiers missed. In 71% of them, the sarcasm could only be understood from the conversational context.
  • Availability: because of copyright, the annotations are offered on request without the dialogue text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

19 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 10 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k16-1015

@inproceedings{joshi2016harnessing,
  title = {{Harnessing Sequence Labeling for Sarcasm Detection in Dialogue from TV Series ‘Friends'}},
  author = {Joshi, Aditya and Tripathi, Vaibhav and Bhattacharyya, Pushpak and Carman, Mark J.},
  booktitle = {Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning},
  year = {2016},
  doi = {10.18653/v1/k16-1015},
  url = {https://doi.org/10.18653/v1/k16-1015}
}

References

25 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rehan Akbani, Stephen Kwek, and Nathalie Japkow-icz. 2004. Applying support vector machines to imbalanced datasets. In Machine learning: ECML 2004, pages 39–50. Springer. find
  2. Anupam Khattri, Aditya Joshi, Pushpak Bhattacharyya, and Mark James Carman. 2015. Your sentiment precedes you: Using an authors historical tweets to predict sarcasm. In 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA), page 25. ↗ our card
  3. Yasemin Altun, Ioannis Tsochantaridis, Thomas Hofmann, et al. 2003. Hidden markov support vector machines. In ICML, volume 3, pages 3–10. find
  4. David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. In Ninth International AAAI Conference on Web and Social Media. find
  5. Sotiris Kotsiantis, Dimitris Kanellopoulos, Panayiotis Pintelas, et al. 2006. Handling imbalanced datasets: A review. GESTS International Transactions on Computer Science and Engineering, 30(1):25–36. find
  6. Konstantin Buschmeier, Philipp Cimiano, and Roman Klinger. 2014. An impact analysis of features in a classification approach to irony detection in product reviews. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 42–49. find our card
  7. Roger J Kreuz and Gina M Caucci. 2007. Lexical influences on the perception of sarcasm. In Proceedings of the Workshop on computational approaches to Figurative Language, pages 1–4. Association for Computational Linguistics. find our card
  8. Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of LREC 2016. find our card
  9. Zelin Wang, Zhijian Wu, Ruimin Wang, and Yafeng Ren. 2015. Twitter sarcasm detection exploiting a context-based model. In Web Information Systems Engineering–WISE 2015, pages 77–91. Springer. find
  10. Yi Mao and Guy Lebanon. 2006. Isotonic conditional random fields and local sentiment flow. In Advances in neural information processing systems, pages 961–968. ↗
  11. Ainur Yessenalina, Yisong Yue, and Claire Cardie. 2010. Multi-level structured models for document-level sentiment classification. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1046–1056. Association for Computational Linguistics. find
  12. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and trends in information retrieval, 2(1-2):1–135. ↗
  13. Zhengchen Zhang, Minghui Dong, and Shuzhi Sam Ge. 2014. Emotion analysis of children’s stories with context information. In Asia-Pacific Signal and Information Processing Association, 2014 Annual Summit and Conference (APSIPA), pages 1–7. IEEE. find
  14. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. The Journal of Machine Learning Research, 12:2825–2830. find
  15. Ge S S Zhang Z, Dong M. 2014. Emotion analysis of children’s stories with context information. In Asia-Pacific Signal and Information Processing Association, volume 38, pages 1–7. IEEE. find
  16. James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates, 71:2001. find
  17. Jun Zhao, Kang Liu, and Gen Wang. 2008. Adding redundant features for crfs-based sentence sentiment classification. In Proceedings of the conference on empirical methods in natural language processing, pages 117–126. Association for Computational Linguistics. find
  18. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pages 97–106. ACM. find
  19. Rachel Rakov and Andrew Rosenberg. 2013. ” sure, i did the right thing”: a system for sarcasm detection in speech. In INTERSPEECH, pages 842–846. find
  20. Antonio Reyes, Paolo Rosso, and Davide Buscaldi. 2012. From humor recognition to irony detection: The figurative language of social media. Data & Knowledge Engineering, 74:1–12. ↗
  21. Joseph Tepperman, David R Traum, and Shrikanth Narayanan. 2006. ” yeah right”: sarcasm recognition for spoken dialogue systems. In INTER-SPEECH. Citeseer. find
  22. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm-a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In ICWSM. find
  23. Tony Veale and Yanfen Hao. 2010. Detecting ironic intent in creative comparisons. In ECAI, volume 215, pages 765–770. ↗
  24. Byron C Wallace, Laura Kertz Do Kook Choe, and Eugene Charniak. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 512–516. find
  25. Byron C Wallace. 2015. Sparse, contextually informed models for irony detection: Exploiting user communities,entities and sentiment. In ACL. find

↑ back to top

This page in Markdown →