Elamur Research
☰
home / papers / 2018 / SemEval-2018 Task 3: Irony Detection in English Tweets

SemEval-2018 Task 3: Irony Detection in English Tweets

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/S18-1005.pdf

Summary

Irony makes a text mean something different from what it literally says, and this hurts NLP applications such as sentiment analysis. At SemEval-2014, sentiment classifiers scored much lower on ironic tweets than on ordinary ones. This paper reports the first shared task on irony detection, held at SemEval-2018. Systems saw only tweet text, without metadata. In Task A they decided whether a tweet is ironic. In Task B they chose one of four labels: verbal irony through a polarity clash, other verbal irony, situational irony, or not ironic.

The organisers gathered tweets tagged #irony, #sarcasm or #not and annotated them by hand with a fine-grained scheme to filter out misleading hashtags. They then added non-ironic tweets and removed the collection hashtags. Forty-three teams entered Task A and 31 entered Task B. The best systems reached F1 0.705 on the binary task and 0.507 on the multiclass task, so telling irony types apart proved much harder than detecting irony. The resulting benchmark gives figurative-language research a manually verified reference dataset.

Key points

  • The source "hashtag corpus" holds 3,000 English tweets from 2,676 users, collected between 01/12/2014 and 04/01/2015. Three linguistics students annotated them in brat, and reliability was checked with Fleiss' Kappa.
  • The balanced dataset of 4,792 tweets was split into about 3,833 training tweets and a test set. Removing ambiguous items reduced the test set to 784 tweets, of which 40% are ironic.
  • In Task B, ironic by clash was recognised best (top F1 0.697). Situational irony (0.376) and other verbal irony (0.114) were much harder.
  • Strong systems combined neural architectures (LSTMs, CNNs, ensembles) with embeddings trained on large Twitter corpora and sentiment-lexicon features such as VADER, SenticNet and aFinn.
  • Unconstrained runs that used extra training data (9 in Task A, 4 in Task B) did not beat constrained ones. This is likely because the added data had been annotated differently.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

175 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 166 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1005

@inproceedings{hee2018semeval,
  title = {{SemEval-2018 Task 3: Irony Detection in English Tweets}},
  author = {Hee, Cynthia Van and Lefever, Els and Hoste, Véronique},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1005},
  url = {https://doi.org/10.18653/v1/s18-1005}
}

References

50 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mário J. Silva. 2016. Modelling Context with User Embeddings for Sarcasm Detection in Social Media. CoRR, abs/1607.00976. ↗ our card
  2. Delia Irazú Hernańdez Farı́as, Viviana Patti, and Paolo Rosso. 2016. Irony detection in twitter: The role of affective content. ACM Transactions on Internet Technology, 16(3):19:1–19:24. find
  3. David Bamman and Noah A. Smith. 2015. Contextualized Sarcasm Detection on Twitter. In Proceedings of the Ninth International Conference on Web and Social Media (ICWSM’15), pages 574–577, Oxford, UK. AAAI. find
  4. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1615–1625, Copenhagen, Denmark. ACL. find our card
  5. Francesco Barbieri and Horacio Saggion. 2014. Modelling Irony in Twitter. In Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the ACL, pages 56–64, Gothenburg, Sweden. ACL. find
  6. Christos Baziotis, Nikolaos Athanasiou, Pinelopi Papalampidi, Athanasia Kolovou, Georgios Paraskevopoulos, Nikolaos Ellinas, and Alexandros Potamianos. 2018. NTUA-SLP at SemEval-2018 Task 3: Deep Character and Word-level RNNs with Attention for Irony Detection in Twitter. In Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval-2018, New Orleans, LA, USA. ACL. find
  7. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. find
  8. Aniruddha Ghosh. 2018. IronyMagnet at SemEval-2018 Task 3: A Siamese network for Irony detection in Social media. In Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval-2018, New Orleans, LA, USA. ACL. find
  9. Mondher Bouazizi and Tomoaki Ohtsuki. 2016. Sarcasm detection in twitter: “all your products are incredibly amazing!!!” - are they really? In Global Communications Conference, GLOBECOM 2015, pages 1–6. IEEE. find
  10. Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden, and Antonio Reyes. 2015. SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 470–478, Denver, Colorado. ACL. find
  11. Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1993. Signature verification using a ”siamese” time delay neural network. In Proceedings of the 6th International Conference on Neural Information Processing Systems, NIPS’93, pages 737–744, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  12. Aniruddha Ghosh and Tony Veale. 2016. Fracking Sarcasm using Neural Network. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 161–169, San Diego, California. ACL. find our card
  13. Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992. Class-based N-gram Models of Natural Language. Computational Linguistics, 18(4):467–479. find
  14. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying Sarcasm in Twitter: A Closer Look. In Proceedings of the 49th Annual Meeting of the ACL: Human Language Technologies (HLT’11), pages 581–586, Portland, Oregon. ACL. find our card
  15. Erik Cambria, Soujanya Poria, Rajiv Bajpai, and Bjo-ern Schuller. 2016. SenticNet 4: A Semantic Resource for Sentiment Analysis Based on Conceptual Primitives. In Proceedings of COLING 2016, 26th International Conference on Computational Linguistics, pages 2666–2677, Osaka, Japan. ACL. find
  16. Raj Kumar Gupta and Yinping Yang. 2017. Crys-talNest at SemEval-2017 Task 4: Using Sarcasm Detection for Enhancing Sentiment Classification and Quantification. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 626–633. ACL. find
  17. Jean Carletta. 1996. Assessing Agreement on Classification Tasks: The Kappa Statistic. Computational Linguistics, 22(2):249–254. find our card
  18. Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2002. SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16(1):321– 357. find our card
  19. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04, pages 168–177, New York, NY, USA. ACM. find
  20. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised Recognition of Sarcastic Sentences in Twitter and Amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning (CoNLL’10), pages 107–116, Uppsala, Sweden. ACL. find our card
  21. Clayton J. Hutto and Eric Gilbert. 2014. VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. In Proceedings of the 8th International Conference on Weblogs and Social Media (ICWSM-14), pages 216–225. AAAI. find
  22. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825–2830. find
  23. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. Automatic Sarcasm Detection:A Survey. ACM Computing Surveys (CSUR), 50(5):73:1– 73:22. find our card
  24. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of Tricks for Efficient Text Classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. ACL. find
  25. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. ACL. find
  26. Jihen Karoui, Benamara Farah, Véronique MORICEAU, Nathalie Aussenac-Gilles, and Lamia Hadrich-Belguith. 2015. Towards a Contextual Pragmatic Model to Detect Irony in Tweets. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 644–650, Beijing, China. ACL. find
  27. Peter Potash, Alexey Romanov, and Anna Rumshisky. 2017. SemEval-2017 Task 6: #HashtagWars: Learning a Sense of Humor. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 49–57. ACL. find
  28. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 213–223, Dublin, Ireland. Dublin City University and ACL. find our card
  29. Florian Kunneman, Christine Liebrecht, Margot van Mulken, and Antal van den Bosch. 2015. Signaling sarcasm: From hyperbole to hashtag. Information Processing Management, 51(4):500–509. find
  30. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1). find
  31. Harsh Rangwani, Devang Kulshreshtha, and Anil Kumar Sing. 2018. NLPRL-IITBHU at SemEval-2018 Task 3: Combining Linguistic Features and Emoji pre-trained CNN for Irony Detection in Tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval-2018, New Orleans, LA, USA. ACL. find
  32. Bing Liu. 2012. Sentiment Analysis and Opinion Mining. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. find
  33. Diana Maynard and Mark Greenwood. 2014. Who cares about Sarcastic Tweets? Investigating the Impact of Sarcasm on Sentiment Analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4238–4243, Reykjavik, Iceland. European Language Resources Association. find our card
  34. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A Multidimensional Approach for Detecting Irony in Twitter. Language Resources and Evaluation, 47(1):239–268. find
  35. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as Contrast between a Positive Sentiment and Negative Situation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’13), pages 704–714, Seattle, Washington, USA. ACL. find our card
  36. Abhijit Mishra, Diptesh Kanojia, Seema Nagar, Kuntal Dey, and Pushpak Bhattacharyya. 2016. Harnessing Cognitive Features for Sarcasm Detection. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1095–1104, Berlin, Germany. ACL. find
  37. Saif Mohammad and Felipe Bravo-Marquez. 2017. Emotion Intensities in Tweets. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics, *SEM @ACM 2017, pages 65–77. find
  38. Omid Rohanian, Shiva Taslimipoor, Richard Evans, and Ruslan Mitkov. 2018. WLV at SemEval-2018 Task 3: Dissecting Tweets in Search of Irony. In Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval-2018, New Orleans, LA, USA. ACL. find
  39. Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. 2016. SemEval-2016 Task 4: Sentiment Analysis in Twitter. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1–18, San Diego, California. ACL. find
  40. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 Task 4: Sentiment Analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502–518, Vancouver, Canada. ACL. find our card
  41. Finn Årup Nielsen. 2011. A new ANEW: evaluation of a word list for sentiment analysis in microblogs. In Proceedings of the ESWC2011 Workshop on ‘Making Sense of Microposts’: Big things come in small packages, volume 718, pages 93–98. find
  42. Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. 2014. SemEval-2014 Task 9: Sentiment Analysis in Twitter. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 73–80, Dublin, Ireland. ACL and Dublin City University. find
  43. Cameron Shelley. 2001. The bicoherence theory of situational irony. Cognitive Science, 25(5):775–818. find
  44. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. BRAT: A Web-based Tool for NLP-assisted Text Annotation. In Proceedings of the 13th Conference of the European Chapter of the ACL, EACL’12, pages 102–107, Avignon, France. ACL. find
  45. Cynthia Van Hee. 2017. Can machines sense irony? Exploring automatic irony detection on social media. Ph.D. thesis, Ghent University. find
  46. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2016a. Guidelines for Annotating Irony in Social Media Text, version 2.0. Technical Report 16-01, LT3, Language and Translation Technology Team– Ghent University. find
  47. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2016b. Monday mornings are my fave #not: Exploring the Automatic Recognition of Irony in English tweets. In Proceedings of COLING 2016, 26th International Conference on Computational Linguistics, pages 2730–2739, Osaka, Japan. find our card
  48. Thanh Vu, Dat Quoc Nguyen, Xuan-Son Vu, Dai Quoc Nguyen, Michael Catt, and Michael Trenell. 2018. NIHRIO at SemEval-2018 Task 3: A Simple and Accurate Neural Network Model for Irony Detection in Twitter. In Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval-2018, New Orleans, LA, USA. ACL. find
  49. Byron C. Wallace. 2015. Computational irony: A survey and new perspectives. Artificial Intelligence Review, 43(4):467–483. find
  50. Chuhan Wu, Fangzhao Wu, Sixing Wu, Junxin Liu, Zhigang Yuan, and Yongfeng Huang. 2018. THU NGN at SemEval-2018 Task 3: Tweet Irony Detection with Densely Connected LSTM and Multi-task Learning. In Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval-2018, New Orleans, LA, USA. ACL. find