📄 Binarizer at SemEval-2018 Task 3: Parsing dependency and deep learning for irony detection

Summary

The paper describes team Binarizer's entry to SemEval-2018 Task 3, Subtask A: a binary decision on whether an English tweet is ironic. The authors assume that irony often arises when different parts of a tweet carry clashing emotions. Each tweet is parsed with TweeboParser, a dependency parser for tweets trained on Tweebank. Its output is frequently a multi-rooted graph, and the words under each root are grouped into one phrase. Every phrase is encoded with the pretrained DeepMoji emoji-prediction model, taking its 2,304-dimensional attention output in place of the softmax. Up to nine phrase vectors are concatenated and passed to a fully connected classifier. One variant adds an embedding of the whole tweet. Trained on 3,833 tweets, the system placed ninth of 43 by accuracy and seventeenth by F1. A submission glitch meant only 446 of the 784 test tweets were scored. The work matters because it treats syntactic segmentation as a way to expose sentiment contrast inside a short text.

Key points

• Parser: TweeboParser produces unlabelled dependencies and tends to leave hashtags, URLs and emoticons out of the tree. It was trained on a subset of Tweebank (929 tweets, 12,318 tokens).
• Encoder: DeepMoji was trained on 1.2 billion emoji-bearing tweets. Its feature vector concatenates three layers: 256 embedding dimensions plus two BiLSTM layers of 1,024 each, giving 2,304.
• Classifier: Phrase slots are fixed at nine, with zero-padding or truncation. The fully connected layers run 20,736 → 9,216 → 2,304 → 256 → 2, all with tanh, trained by SGD with learning rate 0.01 and momentum 0.5.
• Variants:
  • System α uses phrase vectors only and gave higher precision.
  • System β also includes a whole-tweet vector and gave higher recall.
• Data: Tweets were collected via the #irony, #sarcasm and #not hashtags, then manually cleaned. The split was 3,833 for training and 784 for testing, scored by accuracy, precision, recall and F1.

Interesting ideas

• I1. Each root of a multi-rooted parse is treated as its own utterance, so a parser's fragmentation becomes a free segmentation of noisy text. This could help wherever texts mix several sentences or fragments.
• I2. Irony is modelled as contrast between the emotional colouring of separate segments, not as a property of the whole text. This gives an inspectable signal for where the mismatch lies.
• I3. A model pretrained on a cheap proxy label (emoji prediction) is used as a sentiment feature extractor for a scarce target label. This transfers well to any task with little annotated data.
• I4. Comparing segment-only and segment-plus-whole-text inputs showed a precision/recall trade-off. Context therefore changes which errors a system makes, not just how many.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): The tree stage could expose disconnected roots as explicit utterance units, rather than forcing a single tree, and pass each unit separately downstream.
• I2 (frames): The irony component in prag could compare the polarity of frames from different utterances and flag a clash, naming the two conflicting segments as its explanation.
• I3 (learn): Abundant weak signals such as emoji, punctuation or hashtags could serve as silver supervision for evaluative-stance features, which are then checked against a small gold set.
• I4 (learn): Evaluation could run paired ablations with and without global context and report precision and recall separately, so the operating point fits the downstream use.