📄 KLUEnicorn at SemEval-2018 Task 3: A Naive Approach to Irony Detection

Summary

This paper presents KLUEnicorn, a constrained submission to SemEval-2018 Task 3 on irony detection in English tweets. Task A is binary (ironic vs. not); Task B separates three irony types from non-irony. Only the official training set of 3,834 tweets was used, with no author or other context. After cleaning, the tweets were tagged with Ark TweetNLP. The system extracts lightweight features:

• TF-IDF n-grams and tweet length
• punctuation and Twitter-specific counts
• sentiment from AFINN and SentiWordNet
• named entities and number expressions
• 19 Wiktionary adverb categories
• sums of word2vec vectors from two models, one trained on ironic and one on non-ironic tweets

An F-test keeps the top 15% of features, and a Gaussian naïve Bayes classifier makes the decision. Ablations found the embedding-based features most useful. Officially, the system fell below the bag-of-words SVM baseline on Task A and ranked 27th and 23rd. However, a two-step variant using all features beat the baseline on Task B. The study shows how far shallow, tweet-only cues go and where they fail, especially on rare situational irony.

Key points

• Data: 1,911 ironic and 1,923 non-ironic training tweets. The ironic ones split into 1,390 polarity-contrast, 316 other verbal and 205 situational. Only about 14% of ironic tweets had URLs, mostly links to images or tweets.
• Embeddings: two word2vec models built with gensim, using 100 dimensions, window 5 and hierarchical softmax. Each tweet vector is the sum of its word vectors, normalised to unit length.
• Classifier comparison (10-fold CV, Task A): naïve Bayes, SVM, decision tree and random forest were compared. Random forest on all features scored best at 70.06 F1. Removing the embedding group cost up to about 10 points for random forest.
• Feature importance (random forest): embedding dimensions carried 91% of the importance weight. Tweet length in characters ranked first, followed by positive word sentiment. Among adverb types, demonstrative adverbs were most informative.
• Test results:
  • Task A: F1 55.71 vs. a benchmark of 58.91.
  • Task B: the feature-selected two-step version beat the benchmark F1 (34.08) by about 6 points.
  • Situational irony reached only about 11% F1, while non-irony was found with nearly 80% recall.

Interesting ideas

• I1. Build two separate representations of a text, one from a model of literal usage and one from a model of ironic usage, and let the gap between them signal irony.
• I2. Sort adverbs into fine-grained semantic classes (location, temporal, degree, evaluative, speech-act, etc.) instead of one ADV tag, because time and place adverbs may hint at situational irony while degree adverbs hint at hyperbole.
• I3. Run each feature-group ablation in both directions, training on the group alone and on everything except the group, to tell which groups are informative and which are redundant.
• I4. Classify hierarchically: first decide ironic vs. literal, then decide the irony type, so the scarce subtypes do not swamp the main decision.
• I5. Split hashtags using regular expressions plus a dictionary that mixes a standard wordlist with terms mined from in-domain tweets, so non-standard spellings still get segmented.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): the prag irony module could compute a literal and an ironic frame reading for each utterance and flag irony when the expected world-state clash appears, which keeps the decision explainable.
• I2 (tokens): crate en could attach a Wiktionary-derived adverb class to each lemma, giving prag and world ready cues for hyperbole, stance and event timing.
• I3 (learn): this two-way ablation could become a standard evaluation protocol for Mova's rule and feature families, alongside the negative controls.
• I4 (frames): prag could first gate whether an utterance is non-literal at all and only then pick a speech-act or irony subtype, making each choice separately inspectable.
• I5 (tokens): the tokenizer could segment hashtags and run-together tokens into real words for the UD tree, with a domain-mined lexicon added to the standard one.