📄 CTSys at SemEval-2018 Task 3: Irony in Tweets

Summary

This system paper reports an entry in SemEval-2018 Task 3, which asks whether English tweets are ironic (subtask A, binary) and, if so, which kind of irony they show (subtask B: situational, verbal, other, or none). The team used the organizers' corpus of 3,842 tweets and built three separate feature families. The first was a frequency-based bag of words. The second was a rule-based sentiment-contrast detector, which splits a tweet at a conjunction and checks whether the two halves have opposite polarity. The third was averaged Word2Vec embeddings. Each feature family was fed to four classifiers: Naive Bayes, SVM, Decision Trees and 1-NN. A two-stage voting scheme then combined the predictions, first across classifiers and then across feature families. The final scores were modest: F1 of 0.4675 in subtask A (35th place) and 0.3014 in subtask B (22nd place). The paper is still useful as a clear baseline. Its own error analysis points at what flat features miss, namely dependencies between words and representations of whole sentences.

Key points

• The data is the SemEval-2018 Task 3 corpus: 3,842 English tweets collected through the hashtags #irony, #sarcasm and #not. Subtask B labels are 0 = non-ironic, 1 = situational, 2 = verbal, 3 = other.
• There are three feature models: a bag-of-words built from frequency hash-maps of ironic vs non-ironic words, with words frequent in both classes discarded; polarity contrast computed with a Sentiment Intensity Analyzer using a 0.5 threshold; and 100-dimensional Word2Vec vectors averaged per tweet.
• The classifiers are Naive Bayes, SVM, Decision Trees and K-NN, where k=1 tuned best. On the training set, the best single pairing was bag-of-words with Naive Bayes.
• Official test results:
  • Subtask A: accuracy 0.5089, precision 0.4102, recall 0.5434, F1 0.4675.
  • Subtask B: accuracy 0.4923, F1 0.3014.
• Preprocessing was adapted to each feature. Negations and conjunctions such as "not" and "but" were kept for the sentiment feature, and lemmatization was skipped there because it can alter polarity.

Interesting ideas

• I1. Verbal irony is treated as a clash between two evaluative halves of an utterance: the tweet is cut at a conjunction and the dominant polarity of each side is compared, which gives a cheap and inspectable irony cue.
• I2. Preprocessing is made feature-specific: negations and contrastive conjunctions are not thrown away as stop words, and lemmas are avoided where they would erase sentiment-bearing surface forms.
• I3. The paper separates situational irony (an event that defies expectation) from verbal irony (saying the opposite of what is meant), and the four-way task scored far below the binary one with purely lexical features.
• I4. A two-level vote over feature families and classifiers still landed near 0.47 F1, and the authors name ignored word dependencies as the core weakness.
• I5. The full feature-by-classifier grid was scored on training data, which makes it hard to tell real gains from overfitting, as the 1-NN choice suggests.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): prag could compare the evaluative polarity of the two sides of a conj/advcl/cc structure in the UD tree instead of splitting the flat string, and record the clash as an explicit irony signal with its two evidence spans.
• I2 (tokens): the en crate should keep surface form, lemma and negation marking side by side, so that downstream evaluative and irony logic never loses polarity information to normalization.
• I3 (world): situational irony could be detected as a mismatch between an event in the text's world and the expected outcome drawn from compiled knowledge (verb classes, ontology), something string features cannot see.
• I4 (learn): this is a reminder that agreement among many weak, structure-blind witnesses does not create meaning. Silver data from witness voting should be checked against negative controls rather than trusted for agreement alone.
• I5 (learn): every component comparison should be reported on held-out data alongside a trivial baseline, so that a "best" configuration is not an artifact of memorization.