📄 UWB at SemEval-2018 Task 3: Irony detection in English tweets

Summary

This paper presents the University of West Bohemia entry to SemEval-2018 Task 3, which asks systems to find irony in English tweets. Subtask A is a yes/no decision. Subtask B separates non-ironic tweets from three ironic types: irony by clash, situational irony, and other verbal irony. The author built a "strongly constrained" system, a label he introduced himself. It learns only from the task's training data and uses no lexicons, embeddings or other external resources. A Maximum Entropy classifier from the Brainy library is fed hand-designed features. Many of these come from UDPipe output: POS tags, lemmas, morphological features and dependency trees. Others are character and word n-grams, word shapes, and cues at the start and end of the tweet. Measured by accuracy, the submitted system took fourth place on Subtask A and seventh place on Subtask B, although it ranked lower on F1. Ablation studies identify which feature groups helped. The work shows that a transparent, resource-free, syntax-aware linear model can compete with systems that use embeddings.

Key points

• Data:
  • Training: 3,834 tweets, almost balanced (1,923 non-ironic, 1,911 ironic).
  • Test: 784 tweets, 60.3% non-ironic.
  • Subtask B training split: 1,390 clash, 316 situational, 205 other-irony tweets.
• Subtask A, submitted run (settings chosen by 10-fold cross-validation on training data):
  • Accuracy 0.6875, ranked 4th.
  • F1 0.6202, ranked 11th.
  • A post-hoc "best" configuration tuned on test data reached accuracy 0.7003 and F1 0.6264.
• Subtask B, submitted run:
  • Accuracy 0.6263, ranked 7th.
  • F1 0.3902, ranked 13th.
  • The test-tuned best run reached accuracy 0.6403 and F1 0.4080.
• Feature set:
  • Character n-grams (orders 1–5) and word n-grams.
  • Word-shape n-grams from Stanford CoreNLP.
  • POS counts and POS counts binned by quantile.
  • Bag of morphological features over verbs, and bag of parse-tree tags.
  • Words around the root and around verbs in the tree.
  • First and last five words, a short list of interjection and punctuation cues, and TF-IDF.
• Ablation results:
  • Character n-grams mattered most.
  • Binned POS counts, first words and verb morphology helped in both subtasks.
  • Root context, TF-IDF and unigrams helped in A, and word-shape n-grams helped in B.
  • Several submitted features turned out useless or harmful.

Interesting ideas

• I1. Instead of the whole sentence, the model collects only the parent, siblings and children of the root and of each verb, filtered by UPOS, so the features focus on the syntactic core where an ironic clash tends to sit.
• I2. Verb morphology from UD features (mood, tense, polarity, voice, verb form) served as a cheap and readable signal of irony.
• I3. Raw counts of each POS tag are turned into a three-way one-hot code using tertiles from the training data, which makes counts comparable across texts of different lengths.
• I4. The first and last five words of a tweet are treated as separate features, because the setup and the ironic payoff often sit at its edges.
• I5. The paper reports both a cross-validation-selected run and a test-tuned "best" run, and gives a remove-one ablation table for each, which reveals how far feature selection was optimistic.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): The prag irony detector could inspect the predicate's immediate dependents in the UD tree for evaluative words that clash, keeping each cue tied to a specific arc rather than a bag of words.
• I2 (frames): The tense, mood and polarity already present on predicate nodes could be passed to the irony and implicature rules as explicit, explainable evidence.
• I3 (learn): When repair rules or perceptron features need numeric properties such as dependent counts or sentence length, tertile binning keeps them discrete and easy to inspect.
• I4 (frames): The speech-act and irony modules could weight clause-final material, such as a tag phrase or a closing interjection, as a dedicated position cue.
• I5 (learn): Evaluations could always keep the honestly selected configuration apart from any held-out-tuned one, and attach per-component ablations as a check against overfitting.