📄 Lancaster at SemEval-2018 Task 3: Investigating Ironic Features in English Tweets

Summary

The paper describes Lancaster's entry to SemEval-2018 Task 3, which asks systems to label English tweets as ironic or not. Using only the tweet text from the organisers' annotated data, the authors built a hand-designed, explainable feature set and fed it to an untuned LinearSVC from scikit-learn. The features fall into four groups: tweet-surface cues (punctuation, emoji, elongated words, capitalisation), sentiment cues (VaderSentiment scores, emoji sentiment, polarity shifts between words), complexity/style measures (readability, WordNet ambiguity, lexical density), and large data-driven Bag-of-X sets. The submitted combination scored F1 0.5914, beating a bag-of-words baseline. Surface tweet features alone did best. The authors then ablated by irony subtype. The classifier mainly relies on polarity contrast and almost fails once such tweets are removed. The work matters less for its score than for its diagnosis: lexical features add little, and situational or non-contrastive irony likely needs contextual and world knowledge.

Key points

• Feature-set results (F1): Baseline 0.5516, Tweet 0.6046 (highest), Bag-of-X 0.5522, Sentiment 0.5007, Complexity 0.5125, Submission 0.5914, Reduced 0.5901, Combined 0.5889.
• Feature set sizes:
  • Submission (Tweet + Sentiment + Complexity): 126 features.
  • Reduced (emoji and punctuation token features dropped): 27 features.
  • Combined: 3,537 features.
• Tools used:
  • NLTK Tweet Tokeniser and NLTK POS tagger.
  • USAS semantic tagger for Bag-of-X semantic tags.
  • VaderSentiment for sentiment scores.
  • Kralj Novak et al. emoji sentiment lexicon.
  • WordNet sense counts for ambiguity.
  • A random forest for comparison.
• Recall on ironic tweets differed by subtype: 82% for polarity-contrast irony versus 50% for other kinds.
• In the retraining experiment, polarity-contrast tweets were removed from both training and test data. The classifier then caught only 4 of 147 ironic test tweets.
• Character elongation and positive words such as "love", "great" and "fun" carried heavy weight toward the ironic class.

Interesting ideas

• I1. Represent a sentence as a sequence of positive/neutral/negative words and count the flips from positive to negative, so that "love … awful" yields a measurable clash signal that can be explained.
• I2. Break the evaluation down by irony subtype, then retrain with the easy subtype removed, to reveal whether a model has learned irony itself or just one shortcut.
• I3. Choose targeted, predefined features over n-grams so that the linear model's top weights can be read and justified directly.
• I4. Collapse repeated punctuation such as "..." into a single token, and keep elongated spellings ("Greeeeat") as a separate emphasis count instead of discarding them as noise.
• I5. Many ironic tweets describe a negative situation without negative words (going to the dentist), so polarity must come from knowledge about the concept rather than from the lexicon.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): crate prag could compute polarity flips along UD-linked arguments, such as a positive governing verb with a negative complement, as one explicit, traceable cue for its irony judgement.
• I2 (learn): this is a good negative-control pattern; removing the dominant easy class from training and test data shows whether a learned rule truly generalises.
• I3 (learn): inspecting the weights of the perceptron or repair-rule learner in the same way keeps every learned decision explainable, consistent with Mova's no-bag-of-words stance.
• I4 (tokens): crate en could normalise the elongated form to its lemma while attaching an emphasis attribute, which later stages can use as an irony or intensity cue.
• I5 (world): attaching default evaluative associations to events and entities in the knowledge layer would let prag spot a clash between stated delight and an implicitly unpleasant event.