📄 ELiRF-UPV at SemEval-2018 Tasks 1 and 3: Affect and Irony Detection in Tweets

Summary

The ELiRF-UPV team built one neural system for two SemEval-2018 tweet tasks. Task 1, Affect in Tweets, covers English and Spanish and asks for emotion intensity, sentiment (valence) intensity and multi-label emotion classification. Task 3 is English irony detection, with a binary subtask and a four-way subtask.

Each tweet becomes a matrix of word vectors. Every vector joins a Word2Vec embedding with scores taken from polarity and emotion lexicons. This matrix feeds a two-layer CNN followed by an LSTM and a dense output layer. Hyperparameters, lexicon choice and loss were tuned per subtask on the development sets. The ordinal subtasks were not trained separately; they reuse the regression models and cut their scores into classes with thresholds.

The authors' main finding is that training directly on a differentiable version of the official metric (Jaccard, F1 or macro-F1) beat cross-entropy. The system placed mid-table in English and near the top in Spanish. The work shows that a modest model with lexical features and an aligned loss can compete across several affect tasks.

Key points

• Embeddings:
  • English used Word2Vec vectors from Godin et al., pre-trained on 400M tweets.
  • Spanish used the team's own skip-gram vectors: 300 dimensions, trained on 87M crawled tweets.
• Lexicons:
  • English kept all eight: AFFIN, Bing Liu, MPQA, Sentiment140, SentiWordnet, NRC Emotion, NRC Hashtag Emotion and LIWC2007.
  • Spanish kept only ElHPolar and ISOL; MLSenticon was dropped after tuning.
• Architecture: a two-layer CNN (64–256 filters, filter height 3) with batch normalization between layers, then an LSTM with dropout 0.2, trained with RMSProp. A plain LSTM was better only for EI-Reg on development data (67.57 vs 64.12 Pearson for English).
• Metric as loss (development results):
  • English E-C: 55.23 Jaccard with a Jaccard loss vs 52.11 with cross-entropy.
  • Task 3 subtask B: 45.45 macro-F1 with a macro-F1 loss vs 44.59 with cross-entropy.
• Test results:
  • Spanish Task 1 ranked 2nd in V-Reg (74.20), V-Oc (72.90) and E-C (45.80).
  • English EI-Reg scored 69.60 (13th of 42).
  • Irony subtask A scored 62.94 F1 (7th of 44); subtask B scored 42.11 (8th of 32).

Interesting ideas

• I1. Optimizing a smooth stand-in for the official evaluation measure, instead of a generic loss, gave consistent gains on set-overlap and F-style metrics, so training targets the score that is actually reported.
• I2. Ordinal labels were produced by thresholding one regression model, with cut points taken from the lowest score each class reaches in training data, so a single model serves two subtasks.
• I3. Each token vector was extended with lexicon-derived polarity and emotion values, so hand-built knowledge enters the model per word rather than as a document-level count.
• I4. A query item (the target emotion) was appended as an extra input row with a flag column telling it apart from the tweet's words, a cheap way to condition one model on a question.
• I5. The task's irony taxonomy separates irony built on polarity contrast from other verbal irony and from situational irony, which gives a finer target than ironic vs not ironic.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when training the gparse parser or tagger, try a loss that approximates LAS or per-label F1 directly, and check it against a cross-entropy baseline.
• I2 (frames): score sentiment or irony strength once as a continuous value, then map it to discrete levels with thresholds learned from data that can be shown in an explanation.
• I3 (tokens): attach lexicon polarity and emotion attributes to each lemma in the en crate, so later stages can trace which word carried an affective cue.
• I4 (answer): readers could append the question focus as a marked extra element in the input, letting one scorer handle many query types.
• I5 (frames): the irony module in prag could use these three categories, flagging polarity contrast as a clash between the frame's expected and stated evaluation.