📄 HashCount at SemEval-2018 Task 3: Concatenative Featurization of Tweet and Hashtags for Irony Detection

Summary

The paper describes a SemEval-2018 Task 3 entry that detects irony in English tweets. Task A is a binary ironic/non-ironic decision. Task B sorts ironic tweets into polarity-contrast verbal irony, other verbal irony, and situational irony. The authors argue that hashtags carry much of the ironic signal but are often unsegmented or out of vocabulary. They therefore build each input in two parts: a word-vector sequence for the tweet body, followed by a separate hashtag slot called the hashtag vector placeholder (HVP). Hashtags reach this slot through a simple fallback chain: dictionary lookup, then splitting at capital letters, then averaging character vectors. URLs are dropped, and @-mentions are not treated as evidence. The resulting sequences feed BiLSTM and CNN classifiers. Both models beat the organisers' TF-IDF + SVM baseline, with official F1 scores of 0.5939 on Task A and 0.3925 on Task B. An analysis of the Gold test data shows that tweets with extra hashtags are classified correctly more often. This supports treating hashtags as first-class metadata rather than noise.

Key points

• Corpus and splits: 4,792 tweets (2,396 ironic, 2,396 non-ironic), annotated following Van Hee et al. (2016).
  • The split is 3,834 for training and 958 for testing, and all training was constrained.
  • The ironic subtypes contain 1,728 polarity-contrast, 267 other-verbal and 401 situational tweets.
• Input features: tweets are tokenized with NLTK and embedded with 100-dim GloVe vectors trained on 27B Twitter tokens.
  • Each input has 50 positions: 30 for the tweet body and 20 for the HVP.
  • Camel-case hashtags are either averaged into one vector ("sum") or listed word by word ("enum").
• Cross-validation results (10-fold):
  • On Task A, the best system was BiLSTM-sum with emoji, at F1 0.6852 versus 0.6200 for the baseline.
  • On Task B, the best system was BiLSTM-enum with emoji, at macro-F1 0.4378.
  • BiLSTM generally beat CNN, while including emoji showed no consistent effect.
• Official results: Task A scored 0.5939 F1 (rank 23/42) and Task B scored 0.3925 (rank 10/28).
  • The Task B submission was CNN-sum without emoji, and it had notably lower recall.
• Gold-data analysis: accuracy on tweets with more than one hashtag was higher than on all tweets.
  • Ironic tweets rose from 63.6% to 76.2%, and non-ironic tweets from 65.5% to 69.7%.
  • Polarity-contrast irony benefited most from effective hashtags, while the "Other" type was rarely recognised.

Interesting ideas

• I1. Keep salient metadata (hashtags) in its own fixed slot appended after the main text, instead of mixing it inline, so the model can weigh the user's explicit commentary separately from the literal message.
• I2. Use a cheap, training-free fallback chain for unknown compound tokens: try the lowercased form, then split on capital letters, and only then fall back to a character-level representation.
• I3. Validate a feature by comparing accuracy on the test subset where it is present against accuracy on the whole set, broken down by class.
• I4. When counting a cue, discount the label-leaking default token (here the mandatory #irony/#sarcasm tag) so the analysis measures genuine signal rather than an annotation artifact.
• I5. The authors note that detecting semantic contrast alone cannot explain situational irony, which is descriptive in tone rather than provocative.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): the irony module in prag could treat hashtag-like tags or trailing comments as a separate evaluative channel and test whether their polarity contradicts the frame built from the sentence body.
• I2 (tokens): the en tokenizer could segment camel-case hashtags and handles into lemmatizable words before tagging, logging which fallback fired so the step stays explainable.
• I3 (learn): evaluation reports could routinely slice results by the presence of a phenomenon (e.g. hashtags, negation), as a sanity check that a component helps where it should.
• I4 (learn): silver-data and negative-control checks should strip tokens that trivially reveal the label before claiming a feature works.
• I5 (world): situational irony could be detected by comparing expected versus actual event outcomes in the text's world model, rather than by looking for polarity clashes in the wording.