Elamur Research⌕
☰
home / papers / 2018 / IIIDYT at SemEval-2018 Task 3: Irony detection in English tweets

IIIDYT at SemEval-2018 Task 3: Irony detection in English tweets

Paper · 2018 · Edison Marrese-Taylor, Suzana Ilic, Jorge Balazs et al.

Summary

The paper describes the IIIDYT entry to SemEval-2018 Task 3, subtask A. The task is a binary decision on whether an English tweet is ironic. The authors avoided heavy hand-crafted linguistic or contextual features and used representation learning instead. Their model is a multi-layer bidirectional LSTM whose last hidden state encodes the tweet, and it is initialised with GloVe embeddings trained on Twitter. They added a few simple binary cues about casing, digits and repetition, and they gave frequent unknown words vectors sampled near the region of rare GloVe words. The data was a balanced set of about 4.8k tweets originally collected through irony-related hashtags. On their own validation split, an ensemble of four such networks beat the official TF-IDF SVM baseline. On the official test set, however, performance collapsed, and the system finished near the bottom of the ranking. The authors read this as a sign that a generic architecture trained on so little data does not generalise. They argue that irony detection needs task-specific design, large-scale pre-training or transfer learning, and possibly larger noisy self-labelled corpora.

Key points

  • The dataset has 2,396 ironic and 2,396 non-ironic tweets. The ironic tweets break down into 1,728 verbal irony by polarity contrast, 267 other verbal irony and 401 situational irony.
  • Pre-processing used the NLTK TweetTokenizer. Usernames, URLs and trigger words such as not, sarcasm and irony were removed, while case was kept and no stemming was applied.
  • The best configuration used 100-dimensional GloVe vectors, 150 hidden units and dropout 0.1. It was trained with Adam at learning rate 0.0001 with early stopping on an 80/20 split, and the final system was an ensemble of four random seeds.
  • On validation the model reached F1 0.7262 against the baseline's 0.6263. On the test set its F1 dropped to 0.2905, with accuracy 0.3520, among 43 teams.
  • In the ablation, sentence-level binary features helped slightly (F1 0.6848 with them versus 0.6820 without). Token-level features hurt (0.6843 with them versus 0.7008 without).

Interesting ideas

  • I1. Frequent words missing from the pre-trained vocabulary get a vector drawn at random from a sphere centred on the centroid of the rarest 10% of known words, so they land in a plausible rare-word region instead of sharing one UNK vector.
  • I2. The hashtags and words that were used to collect the ironic tweets are deleted before training, so the model cannot simply learn the collection label.
  • I3. Casing is deliberately preserved because all-caps words or spans can signal an ironic or exaggerated tone.
  • I4. Features matching the granularity of the decision (whole-sentence cues) helped, while finer per-token cues added noise.
  • I5. Situational irony is separated from verbal irony because it rests on a scenario contradicting expectations rather than on surface wording.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The tagger could initialise representations of unseen but recurring word forms near the profile of known rare words, rather than collapsing them all into one unknown class.
  • I2 (learn): Mova's silver-data pipeline could strip whatever cue produced a label before training, and treat the stripped version as a negative control against shortcut learning.
  • I3 (frames): The prag irony module could take emphatic casing from the token layer as one explainable cue, alongside polarity contrast between the frame and its context.
  • I4 (frames): Irony and speech-act decisions could be driven by utterance-level properties, such as a contrast between evaluation and situation, rather than by accumulating word-level hints.
  • I5 (world): Such cases could be flagged when the world model's expected outcome of an event chain conflicts with the narrated outcome, for example winning the lottery and then dying the next day.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan45 works · 8,692 citations🔬National Institute of InformaticsTokyo · Japan21 works · 3,045 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 English · 1,373 works Sentiment & emotion · 608 works

Cite

Cite as Edison Marrese-Taylor, Suzana Ilic, Jorge Balazs, Helmut Prendinger and Yutaka Matsuo. 2018. IIIDYT at SemEval-2018 Task 3: Irony detection in English tweets. Proceedings of the 12th International Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1087

@inproceedings{marresetaylor2018iiidyt,
  title = {{IIIDYT at SemEval-2018 Task 3: Irony detection in English tweets}},
  author = {Edison Marrese-Taylor, Suzana Ilic, Jorge Balazs, Helmut Prendinger and Yutaka Matsuo},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1087},
  url = {https://doi.org/10.18653/v1/s18-1087}
}

References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111–3119. Curran Associates, Inc. ↗ our card
  2. David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. Icwsm (International AAAI Conference on Web and Social Media), pages 574–577. find
  3. Jeremy Barnes, Roman Klinger, and Sabine Schulte im Walde. 2017. Assessing state-of-the-art sentiment models on state-of-the-art sentiment datasets. pages 2–12. find
  4. Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359. ↗
  5. Adam Paszke, Gregory Chanan, Zeming Lin, Sam Gross, Edward Yang, Luca Antiga, and Zachary Devito. 2017. Automatic differentiation in pytorch. Advances in Neural Information Processing Systems 30, (Nips):1–4. find
  6. Erik Cambria, Dipankar Das, Sivaji Bandyopadhyay, and Antonio Feraco. 2017. Guide to Sentiment Analysis. find
  7. Springer International Publishing AG 2017, Cham, Switzerland. find
  8. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. our card
  9. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. our card
  10. H. Paul Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Semantics, volume 3, pages 41–58. Academic Press, New York. ↗
  11. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. ↗ our card
  12. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  13. Geoffrey E Hinton. 1984. Distributed representations. find
  14. Sepp Hochreiter and J Urgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. ↗
  15. Oren Tsur, Ari Rappoport, and Dmitry Davidov. 2010. Icwsm a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. International AAAI Conference on Weblogs and Social Media, (9):162–169. find
  16. Aaron Courville Ian Goodfellow, Yoshua Bengio. 2017. Deep Learning, volume 521. MIT Press. find
  17. Aditya Joshi, Pushpak Bhattacharyya, and Mark James Carman. 2016. Automatic sarcasm detection: A survey. ACM Computing Surveys, V. ↗ our card
  18. Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2015. Guidelines for annotating irony in social media text. LT3 Technical Report, 15(2). find
  19. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Short Papers), 51(4):757–762. our card
  20. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval-2018, New Orleans, LA, USA. Association for Computational Linguistics. our card
  21. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodra-halli. 2017. A large self-annotated corpus for sarcasm. our card
  22. Byron C. Wallace. 2015. Computational irony: A survey and new perspectives. Artificial Intelligence Review, 43(4):467–483. find
  23. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference for Learning Representations, San Diego. ↗
  24. Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. ↗
  25. Joan Lucariello. 1994. Situational irony: A concept of events gone awry. Journal of Experimental Psychology: General, 123(2):129–145. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/taylor-2018-iiidyt-semeval-2018-task-3/