Elamur Research⌕
☰
home / papers / 2018 / NLPRL-IITBHU at SemEval-2018 Task 3: Combining Linguistic Features and Emoji pre-trained CNN for Irony Detection in Tweets

NLPRL-IITBHU at SemEval-2018 Task 3: Combining Linguistic Features and Emoji pre-trained CNN for Irony Detection in Tweets

Summary

This paper presents a system for SemEval-2018 Task 3, which asks whether English tweets are ironic. Subtask A is binary (ironic vs. not). Subtask B has four classes: non-ironic, verbal irony via polarity contrast, situational irony and other verbal irony. Irony lacks the overt mockery of sarcasm, which makes it harder to detect, yet recognising it matters for opinion analysis on social media. The system is constrained: it uses only the 3,834 training tweets supplied by the organizers. It merges two feature sources. The first is 72 handcrafted linguistic and user-behaviour features, such as sentiment contrast, readability, GloVe-based incongruity, character repetition, punctuation and marker words. The second is 2304-dimensional activations taken from DeepMoji, a CNN pre-trained on emoji prediction. An XGBoost classifier makes the predictions, and SMOTE oversampling counters the class skew in Subtask B. The team placed 4th in both subtasks, with F1 0.65 on A and macro F1 0.47 on B. The work suggests that pretrained representations combined with classic features suit small labelled datasets.

Key points

  • Data: The training set contains 1923 non-ironic, 1390 polarity-contrast, 316 situational and 205 other-verbal-irony tweets. The test set has 784 tweets, of which 311 are ironic (164 / 85 / 62 across the three irony classes).
  • Metrics: Subtask A is scored by F1 on the ironic class and Subtask B by macro-averaged F1.
  • Results (test set):
    • Subtask A: linguistic features alone reach 0.59, DeepMoji alone 0.62 and the combination 0.65, against a baseline of 0.59 (Linear SVC over BoW).
    • Subtask B: linguistic 0.30, DeepMoji 0.42, combination 0.42, and combination + SMOTE 0.47, against a baseline of 0.33.
  • Resources and tools:
    • DeepMoji (Felbo et al., 2017), trained on 1.2 billion emoji tweets
    • SentiStrength, Warriner affect lexicon, GloVe, WordNet similarity, TextBlob
    • XGBoost with default parameters, chosen via 10-fold CV
    • SMOTE from imblearn with k_neighbors=5
  • Main takeaways:
    • Adding DeepMoji activations to the linguistic features beats linguistic features alone by 0.03 on A and 0.12 on B.
    • SMOTE applied to the neural representations brought the Subtask B gain over the baseline to 14 points.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

5 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1104

@inproceedings{rangwani2018nlprl,
  title = {{NLPRL-IITBHU at SemEval-2018 Task 3: Combining Linguistic Features and Emoji pre-trained CNN for Irony Detection in Tweets}},
  author = {Rangwani, Harsh and Kulshreshtha, Devang and Singh, Anil Kumar},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1104},
  url = {https://doi.org/10.18653/v1/s18-1104}
}

References

22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. We reported the use of handcrafted features and pre-trained CNN activations for predicting the irony in tweets. We implemented a variety of features based on user behavior as well as the linguistic structure in a tweet. We further exploit the SMOTE oversampling technique to handle the class imbalance problem in Subtask-B, which involves categorizing a tweet into Non Ironic, Verbal Irony, Situational Irony and Other Verbal Irony. We then feed the features into XGBoost classifier for both the tasks. The benefit of using CNN models pre-trained on sarcasm, sentiment, and emotion domains can be clearly seen, yielding an improvement of 3% and 9% over task baselines. Our final submitted system stood 4 th in both the subtasks in the SemEval 2018 shared task on “Irony Detection in English Tweets”. • Our submitted models achieve 4th position in public leaderboard 5 on both Task-A and Task-B and beat the task baselines by about 6% and 14%, respectively, on both tasks on the test set. find
  2. Leveraging DeepMoji model for Irony detection domain yields a considerable improvement over purely linguistic features (0.03 and 0.12). This is because the model is trained on over a billion tweets on sarcasm and four other domains. As stated earlier, sarcasm is a verbal form of irony (Long and Graesser, 1988), and transfer learning works as domains are quite similar. find
  3. Konstantin Buschmeier, Philipp Cimiano, and Roman Klinger. 2014. An impact analysis of features in a classification approach to irony detection in product reviews. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 42–49. find our card
  4. Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321–357. • The use of SMOTE oversampling technique leads to an F-score of 0.47 in Task-B, which is an improvement of 0.05 over (Linguistic + Pretrained CNN) model. ↗ our card
  5. Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794. ACM. • The improvement in scores due to linguistic features are not as pronounced in Subtask-B, as compared to Subtask-A. One of the possible reasons for this is that linguistic features are not able to capture the fine grained differences between different forms of irony. ↗
  6. Delia Irazú Hernańdez Farı́as, Viviana Patti, and Paolo Rosso. 2016. Irony detection in twitter: The role of affective content. ACM Transactions on Internet Technology (TOIT), 16(3):19. find
  7. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. arXiv preprint arXiv:1708.00524. 5 https://competitions.codalab.org/ competitions/17468#results Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval-2018, New Orleans, LA, USA. Association for Computational Linguistics. ↗ our card
  8. Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys (CSUR), 50(5):73. ↗ our card
  9. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016. Are word embedding-based features useful for sarcasm detection? arXiv preprint arXiv:1610.00883. ↗ our card
  10. Byron C Wallace, Eugene Charniak, et al. 2015. Sparse, contextually informed models for irony detection: Exploiting user communities, entities and sentiment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1035–1044. find
  11. Roger J Kreuz and Gina M Caucci. 2007. Lexical influences on the perception of sarcasm. In Proceedings of the Workshop on computational approaches to Figurative Language, pages 1–4. Association for Computational Linguistics. find our card
  12. Roger J Kreuz and Sam Glucksberg. 1989. How to be sarcastic: The echoic reminder theory of verbal irony. Journal of experimental psychology: General, 118(4):374. ↗
  13. Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 english lemmas. Behavior research methods, 45(4):1191–1207. ↗
  14. Guillaume Lemaı̂tre, Fernando Nogueira, and Christos K. Aridas. 2017. Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning. Journal of Machine Learning Research, 18(17):1–5. find
  15. Debra L Long and Arthur C Graesser. 1988. Wit and humor in discourse processing. Discourse processes, 11(1):35–60. ↗
  16. Steven Loria, P Keen, M Honnibal, R Yankovsky, D Karesh, E Dempsey, et al. 2014. Textblob: simplified text processing. Secondary TextBlob: Simplified Text Processing. find
  17. Rada Mihalcea, Courtney Corley, Carlo Strapparava, et al. 2006. Corpus-based and knowledge-based measures of text semantic similarity. In AAAI, volume 6, pages 775–780. find
  18. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  19. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pages 97–106. ACM. find
  20. RJ Senter and Edgar A Smith. 1967. Automated readability index. Technical report, CINCINNATI UNIV OH. find
  21. Mike Thelwall, Kevan Buckley, Georgios Paltoglou, Di Cai, and Arvid Kappas. 2010. Sentiment strength detection in short informal text. Journal of the Association for Information Science and Technology, 61(12):2544–2558. ↗
  22. Cynthia Van Hee. 2017. Can machines sense irony? : exploring automatic irony detection on social media. Ph.D. thesis, Ghent University. find

↑ back to top

This page in Markdown →