Elamur Research⌕
☰
home / papers / 2018 / Deep contextualized word representations for detecting sarcasm and irony

Deep contextualized word representations for detecting sarcasm and irony

Summary

Sarcasm and irony often flip the literal polarity of a text, which makes them hard for sentiment and opinion-mining systems. Earlier neural detectors relied on word-level embeddings and could miss surface signals such as capitalization, quotation marks, emoticons, emojis and hashtags. Ilić et al. address this with a character-aware model. Pre-trained ELMo embeddings, which are built from characters via CNNs and contextualized by a deep biLM, are kept frozen and fed into a BiLSTM. The BiLSTM states are max-pooled and passed through a two-layer feed-forward classifier for binary prediction. Preprocessing is deliberately light: no lowercasing, stemming or punctuation stripping.

The model was tested on seven benchmarks drawn from Twitter, Reddit and online debate dialogues. It achieved state-of-the-art results on six of them without attention, handcrafted features or multi-task learning. The only exception was SemEval-2018, where the winning system kept higher recall and F1. Adding hashtag-based, soft-labeled tweets did not help. This suggests that more manually annotated data, rather than noisy data, is needed. Code is released as elmo4irony.

Key points

  • Architecture: frozen ELMo (1,024-dim vectors, pre-trained on the 1 Billion Word Benchmark) → BiLSTM → max-pooling → 2 feed-forward layers of 512 units. Training used Adam with lr 0.001 and a modified Twokenizer for emoji handling.
  • Datasets:
    • Twitter: SemEval-2018 Task 3, Riloff, Ptáček
    • Reddit: SARC 2.0, SARC 2.0 pol
    • IAC dialogues: SC-V1, SC-V2
  • Results:
    • Ptáček: F1 0.872 vs 0.860 (Tay et al.)
    • Riloff: F1 0.759 vs 0.732
    • SC-V2: F1 0.760 vs 0.728
    • SC-V1: F1 0.644 vs 0.632
    • SARC 2.0: accuracy 0.773 vs 0.758 (Khodak et al.)
    • SARC 2.0 pol: accuracy 0.785 vs 0.765
  • SemEval-2018:
    • A 10-model ensemble reached F1 0.696, behind the Wu et al. winner at 0.705, but with higher precision (0.696 vs 0.630).
    • As a single model, it beats Wu et al.'s single-model F1 of 0.674.
  • Data augmentation: about 225k #sarcasm/#irony tweets were filtered by hashtag overlap and added to the three Twitter sets. This gave no gains in any of them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

19 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 16 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6202

@inproceedings{ilic2018deep,
  title = {{Deep contextualized word representations for detecting sarcasm and irony}},
  author = {Ilić, Suzana and Marrese-Taylor, Edison and Balazs, Jorge and Matsuo, Yutaka},
  booktitle = {Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis},
  year = {2018},
  doi = {10.18653/v1/w18-6202},
  url = {https://doi.org/10.18653/v1/w18-6202}
}

References

29 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodra-halli. 2017. A Large Self-Annotated Corpus for Sarcasm. find our card
  2. S. K. Bharti, B. Vachha, R. K. Pradhan, K. S. Babu, and S. K. Jena. 2016. Sarcastic sentiment detection in tweets streamed in real time: a big data approach. Digital Communications and Networks, 2(3):108– 121. ↗
  3. Florian Kunneman, Christine Liebrecht, Margotvan Mulken, and Antalvan den Bosch. 2015. Signalling sarcasm : From hyperbole to hashtag. Information Processing & Management. find
  4. Santosh Kumar Bharti, Korra Sathya Babu, and Sanjay Kumar Jena. 2015. Parsing-based Sarcasm Sentiment Recognition in Twitter Data. Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2015 - ASONAM ’15, pages 1373–1380. find
  5. Joan Lucariello. 1994. Situational Irony: A Concept of Events Gone Awry. Journal of Experimental Psychology: General, 123(2):129–145. ↗
  6. Pranali Chaudhari and Chaitali Chandankhede. 2017. Literature Survey of Sarcasm Detection. pages 2041–2046. find
  7. Saif M. Mohammad, Felipe Bravo-Marquez, Salameh Mohammad, and Svetlana Kiritchenko. 2018. SemEval-2018 Task 1 : Affect in Tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 1–17. find our card
  8. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling. Proceedings of the Annual Conference of the International Speech Communication Association, INTER-SPEECH, pages 2635–2639. find our card
  9. Shereen Oraby, Vrindavan Harrison, Lena Reed, Ernesto Hernandez, Ellen Riloff, and Marilyn Walker. 2016. Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue. Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, (September):31– 41. find our card
  10. Elena Filatova. 2012. Irony and Sarcasm: Corpus Generation and Analysis Using Crowdsourcing. Lrec, pages 392–398. find our card
  11. Bo Pang and Lillian Lee. 2006. Opinion Mining and Sentiment Analysis. Foundations and Trends in Information Retrieval, 1(2):91–231. find
  12. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 42–47, Portland, Oregon, USA. Association for Computational Linguistics. ↗
  13. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. find
  14. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm Detection on Czech and English Twitter. Proceedings of the 25th International Conference on Computational Linguistics: Technical Papers (COL-ING 2014), Citeseer, pages 213–223. find our card
  15. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in Twitter. Language Resources and Evaluation, 47(1):239–268. ↗
  16. H. Paul Grice. 1975. Logic and Conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Semantics, volume 3, pages 41–58. Academic Press, New York. ↗
  17. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as Contrast between a Positive Sentiment and Negative Situation. Emnlp, (Emnlp):704– 714. find our card
  18. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339. Association for Computational Linguistics. find
  19. Cameron Shelley. 2001. The bicoherence theory of situational irony. Cognitive Science, 25(5):775–818. ↗
  20. Aditya Joshi, Pushpak Bhattacharyya, and Mark James Carman. 2016. Automatic Sarcasm Detection: A Survey. 50(5). ↗ our card
  21. Yi Tay, Anh Tuan Luu, , Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1010–1020. Association for Computational Linguistics. find our card
  22. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing Context Incongruity for Sarcasm Detection. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Short Papers), 51(4):757–762. find our card
  23. Piyoros Tungthamthiti, Kiyoaki Shirai, and Masnizah Mohd. 2010. Recognition of Sarcasm in Microblog-ging Based on Sentiment Analysis and Coherence Identification. Journal of Natural Language Processing, 23(5):383–405. find
  24. Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 39–50. find our card
  25. Tony Veale. 2016. Fracking Sarcasm using Neural Network Fracking Sarcasm using Neural Network. Acl, (May):161–169. find our card
  26. Byron C. Wallace. 2015. Computational irony: A survey and new perspectives. Artificial Intelligence Review, 43(4):467–483. find
  27. Chuhan Wu, Fangzhao Wu, Sixing Wu, Junxin Liu, Zhigang Yuan, and Yongfeng Huang. 2018. THU NGN at SemEval-2018 Task 3 : Tweet Irony Detection with Densely THU NGN at SemEval-2018 Task 3 : Tweet Irony Detection with Densely Connected LSTM and Multi-task Learning. (March):51–56. find
  28. Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Tweet sarcasm detection using deep neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2449–2460. find our card
  29. The COLING 2016 Organizing Committee. find

↑ back to top

This page in Markdown →