Elamur Research⌕
☰
home / papers / 2018 / WLV at SemEval-2018 Task 3: Dissecting Tweets in Search of Irony

WLV at SemEval-2018 Task 3: Dissecting Tweets in Search of Irony

Summary

The paper describes the University of Wolverhampton (WLV) entry to SemEval-2018 Task 3 on irony detection in English tweets. The task has two subtasks: a binary ironic/non-ironic decision (A) and a four-way classification of irony type (B). The authors trained on the organisers' data of 3,834 training tweets, roughly balanced, and tested on 784 tweets. Their supervised approach splits each tweet into a left and a right half and computes features for each half separately, which keeps some information about word order. Features include surface cues, sentiment intensity and a deliberately broad binary contrast indicator. This indicator detects sentiment clashes inside or between the text, hashtags and emojis. These are combined with pretrained word2vec and emoji2vec vectors and fed to soft-voting ensembles of LR and SVM. For subtask A the system placed third of 44 teams, mainly thanks to very high recall. In subtask B it ranked ninth by F1 but second by accuracy. The work shows that cheap, interpretable contrast features remain competitive. Its error analysis also shows that situational irony often depends on world knowledge or linked content.

Key points

  • Pipeline: NLTK tweet tokeniser and ekphrasis for surface features and hashtag segmentation. Sentiment comes from Stanford CoreNLP, the Emoji Sentiment Ranking (751 emojis) and the Vader lexicon, which supplies Rise/Fall/Stable patterns.
  • Subtask A: The half-tweet embeddings are averaged. The system reached F1 0.65 (3rd/44), with recall 0.836 (2nd) but precision only 0.53. The high recall reflects a tendency to over-label tweets as ironic.
  • Subtask B: The embeddings are concatenated, and the system adds bigram tf-idf plus LDA and NMF topic features. It scored F1 0.4153 (9th/32) and accuracy 0.6709 (2nd).
  • Per-class test F1 in subtask B: non-ironic 0.7652, polarity clash 0.4651, situational 0.2595, other 0.0299.
  • Error analysis and availability: The authors point to some questionable gold labels and suggest multimodal features, such as the content behind URLs, as future work. Code and data are released on GitHub (omidrohanian/irony_detection).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

7 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1090

@inproceedings{rohanian2018wlv,
  title = {{WLV at SemEval-2018 Task 3: Dissecting Tweets in Search of Irony}},
  author = {Rohanian, Omid and Taslimipoor, Shiva and Evans, Richard and Mitkov, Ruslan},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1090},
  url = {https://doi.org/10.18653/v1/s18-1090}
}

References

18 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fréderic Godin, Baptist Vandersmissen, Wesley De Neve, and Rik Van de Walle. 2015. Multimedia lab @ acl wnut ner shared task: Named entity recognition for twitter microposts using distributed word representations. In Proceedings of the Workshop on Noisy User-generated Text, pages 146–153. find
  2. Christos Baziotis, Nikos Pelekis, and Christos Doulk-eridis. 2017. Datastories at semeval-2017 task 4: Deep lstm with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 747–754, Vancouver, Canada. Association for Computational Linguistics. find
  3. H. P. Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Se-Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502–518. mantics: Vol. 3: Speech Acts, pages 41–58. Academic Press, New York. ↗ our card
  4. Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. 2002. Gene selection for cancer classification using support vector machines. Machine learning, 46(1-3):389–422. ↗
  5. Cameron Shelley. 2001. The bicoherence theory of situational irony. Cognitive Science, 25(5):775–818. ↗
  6. John Haiman. 1998. Talk is cheap: Sarcasm, alienation, and the evolution of language. Oxford University Press on Demand. find
  7. Cynthia Van Hee, Els Lefever, and Vronique Hoste. 2018. Semeval-2018 task 3: Irony detection in English tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA. find our card
  8. Patrick Hanks. 2013. Lexical analysis: Norms and exploitations. MIT Press. find
  9. Byron C Wallace, Laura Kertz, Eugene Charniak, et al. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 512–516. find
  10. Linda Hutcheon. 1994. Irony’s edge: The theory and politics of irony. Psychology Press. find
  11. Aditya Joshi, Vaibhav Tripathi, Pushpak Bhattacharyya, Mark Carman, Meghna Singh, Jaya Saraswati, and Rajita Shukla. 2016. How challenging is sarcasm versus irony classification?: A study with a dataset from English literature. In Proceedings of the Australasian Language Technology Association Workshop 2016, pages 123–127. find our card
  12. Petra Kralj Novak, Jasmina Smailović, Borut Sluban, and Igor Mozetič. 2015. Emoji sentiment ranking 1.0. Slovenian language resource repository CLARIN.SI. find
  13. Edward Loper and Steven Bird. 2002. Natural language processing toolkit. http://www.nltk.org/. ↗
  14. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55–60. find our card
  15. Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. 2016. Semeval-2016 task 4: Sentiment analysis in twitter. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1–18. find
  16. Federico Alberto Pozzi, Elisabetta Fersini, Enza Messina, and Bing Liu. 2016. Sentiment analysis in social networks. Morgan Kaufmann. ↗
  17. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on Czech and English twitter. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 213–223. find our card
  18. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714. find our card

↑ back to top

This page in Markdown →