📄 Irony Detector at SemEval-2018 Task 3: Irony Detection in English Tweets using Word Graph

Summary

This system paper tackles irony detection in English tweets for SemEval-2018 Task 3. It covers both the binary ironic/non-ironic subtask (A) and the four-way subtask (B): polarity-contrast verbal irony, other verbal irony, situational irony, and non-irony. Each tweet becomes a directed, unweighted word graph whose edges link words within a small sliding window. All training tweets of a class are combined into one class graph. A new tweet is scored against every class graph with several graph-similarity measures: containment and three maximum-common-subgraph variants. These scores, plus SenticNet polarity/subjectivity and LDA-based Hellinger distances to each class, form a compact feature vector. TPOT searches for the classifier pipeline, and SMOTE handles class imbalance. The system placed 23rd on subtask A (F1 0.5955) and 22nd on subtask B (F1 0.3101). The scores are modest, but the paper shows that comparing a text's graph with class-level prototype graphs yields usable, low-dimensional features for a pragmatic phenomenon.

Key points

• Data: 3,000 tweets gathered through #irony, #sarcasm and #not between Dec 2014 and Apr 2015, from 2,676 users. Annotators were three linguistics students using the brat tool, with kappa 0.72. The class counts are 1,728 polarity-contrast, 267 other verbal, 401 situational, and 604 non-ironic.
• The organizers balanced the data with a background corpus to 4,792 tweets (2,396 ironic and 2,396 non-ironic). They split it 80/20 into 3,833 training and 958 test instances.
• Graph construction uses a vicinity window of 2 with directed edges that keep word order. The similarity features are:
  • containment similarity (non-normalized), with shared edges divided by the size of the smaller graph;
  • MCS node similarity;
  • MCS undirected-edge similarity;
  • MCS directed-edge similarity.
• TPOT chose a tuned XGBoost model for subtask A. For subtask B it chose a stack of XGBoost, extra trees and random forest, trained with SMOTE oversampling.
• Results:
  • Subtask A: accuracy 0.6135, precision 0.5091, recall 0.7170, F1 0.5955.
  • Subtask B: accuracy 0.4158, precision 0.4055, recall 0.3526, F1 0.3101.

Interesting ideas

• I1. Each class is summarized as one merged graph, and a text is described by its similarity to each class prototype, giving a tiny feature vector whose values can be traced to matched edges.
• I2. The same overlap is measured at three strictness levels: nodes only, edges ignoring direction, and edges with direction. The gaps between these levels show how much structure, not just vocabulary, is shared.
• I3. Containment divides shared edges by the smaller graph's size, so a short text is not penalized for being compared with a huge aggregate graph.
• I4. The task separates verbal irony that rests on a polarity contrast from other kinds, and the system adds sentence polarity scores as explicit features.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): an irony cue in prag could compare a sentence's UD dependency edges against prototype graphs built from ironic and literal examples, then report which shared edges drove the score.
• I2 (learn): when witness parsers disagree, scoring their trees at node, undirected-edge and directed-edge levels would separate attachment disagreements from head-direction flips before silver data is accepted.
• I3 (back): in text → graph → text round-trip checks, containment of the original meaning graph's edges in the regenerated one gives a size-fair preservation score.
• I4 (frames): prag could flag a polarity clash between a stated evaluation and the described event's valence as a named, explainable irony trigger, with non-contrast irony handled separately.