📄 ECNU at SemEval-2018 Task 3: Exploration on Irony Detection from Tweets via Machine Learning and Deep Learning Methods

Summary

The paper presents the ECNU team's systems for SemEval-2018 Task 3, which deals with irony in English tweets. Subtask A is a binary decision (ironic or not). Subtask B splits ironic tweets into polarity contrast, situational irony and other verbal irony, plus a non-ironic class. The authors built a large set of hand-crafted features: word n-grams weighted by Relevant Frequency, NER tags from Stanford CoreNLP, hashtags and segmented hashtag words, punctuation flags, emoticons, elongated words, eight sentiment lexicons, and Word2Vec/GloVe vectors. They fed these to many scikit-learn and Liblinear classifiers, choosing features by hill climbing. For subtask A they also trained a Bi-LSTM on glove.twitter.27B embeddings and combined it with the classical models through voting.

The classical models looked strong in cross-validation (about 0.70 F1) but dropped sharply on the test set, a clear sign of overfitting. Neural models held up better. The submitted ensemble ranked mid-table, and subtask B was weak on rare classes. The work is a useful record of which surface cues do not carry over from training to test data in irony detection.

Key points

• Data: 3,834 training tweets with no dev set and 784 test tweets. In subtask B the classes are very unbalanced: situational irony is 8.2% and other verbal irony 5.3% of training data.
• Metrics: subtask A uses F1 on the ironic class only. Subtask B uses macro-F1 over all four classes.
• Feature work:
  • a 7-dimensional punctuation flag vector (e.g. "!?", final "?");
  • a 67-dimensional emoticon feature;
  • elongated words, defined as characters repeated 3–11 times;
  • hashtags split into words with WordSegment;
  • feature subsets chosen by greedy hill climbing under 10-fold cross-validation.
• Best cross-validation result came from scikit-learn Naïve Bayes (0.7111), but the submitted ML+NN ensemble reached only 0.593 F1 on test (rank 20).
• After the contest:
  • a Bi-LSTM with extra dropout reached up to 0.637 F1, which would have ranked about 6th;
  • adding an attention layer lowered scores;
  • in subtask B, oversampling (×6 and ×10) still left other verbal irony near 0.07 F1.

Interesting ideas

• I1. Punctuation is turned into a small set of explicit binary flags (single vs repeated "!" or "?", mixed "!?", tweet-final mark), giving an interpretable signal of emotional intensity instead of an opaque embedding.
• I2. Hashtags written as one string (e.g. "ilikemonday") are split back into words, so they can carry meaning instead of being unique, unseen tokens.
• I3. In the ensemble, positive neural predictions were kept as they were, and only negatives were re-voted by the other models, deliberately raising recall on the rare class.
• I4. The large gap between shuffled cross-validation scores and test scores was read as a diagnosis of overfitting. Sparse features such as specific hashtags rarely recur between training and test.
• I5. Four classical voters plus one neural model gave an odd committee, so binary votes could not tie.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): The irony and speech-act layer in prag could read such flags as named evidence for exaggerated or rhetorical tone, citable in the explanation.
• I2 (tokens): The en tokenizer could segment hashtags and camel-cased compounds into words that the tagger and parser can then attach in the UD tree.
• I3 (learn): When several witness parsers disagree, a trusted witness could be given veto power for one kind of decision, while the others vote only on the remainder.
• I4 (learn): Evaluation should report held-out scores alongside in-sample ones and flag features that seldom reappear in unseen text, as a guard against memorised cues.
• I5 (learn): Silver-data agreement among witness parsers could likewise use an odd number of voters, or an explicit tie-breaker, so every arc gets a decision.