📄 connotation_clashers at SemEval-2022 Task 6: The effect of sentiment analysis on sarcasm detection

Summary

The paper tests whether sentiment signals help a classifier detect sarcasm in tweets. The idea is that sarcasm often pairs a positive word with a negative situation, as in loving that one got ill. The authors entered only subtask A (binary sarcastic vs. non-sarcastic) of SemEval-2022 Task 6, with English and Arabic tweet data. They built two pipelines and ran each with and without sentiment:

• CNN: a Keras model with its own learned embeddings. The sentiment variant was first pretrained on IMDB movie reviews and then fine-tuned on sarcasm.
• SVM: a model over tf-idf word n-grams. The sentiment variant received extra one-hot features built from binned Flair sentiment predictions.

Adding sentiment slightly hurt both pipelines. The SVM without sentiment scored best overall. The authors conclude that labelling a whole sentence as positive or negative is too coarse to capture an internal polarity clash. They suggest scoring sentence sub-parts or using a sentiment lexicon. The result is a useful negative finding: the polarity-contrast intuition only pays off if the contrast is located inside the sentence.

Key points

• Data: the English training set has 3468 tweets, 867 of them sarcastic. The Arabic set has 3102 tweets, 745 of them sarcastic.
• Official submission: the plain CNN, without sentiment.
  • English: sarcastic-class F1 0.2024, rank 37 of 43.
  • Arabic: F1 0.3013, rank 23 of 32.
• CNN with sentiment: pretraining on Keras IMDB (25,000 reviews, split 20000/5000) dropped English F1 to 0.1679.
• SVM setup:
  • Data balanced by downsampling to 867 tweets per class (694 train / 173 held-out each).
  • Features: tf-idf over unigrams to 4-grams.
• Model choice: Naive Bayes won the grid search but labelled nearly all test tweets sarcastic. The authors therefore used the SVM with scikit-learn default settings.
• SVM results: F1 0.2738 without the Flair features vs. 0.2721 with them.

Interesting ideas

• I1. Sarcasm as a polarity clash should be checked between parts of a sentence, not on the sentence as a whole, since a global score can average the contrast away.
• I2. A near-neutral overall score can itself be evidence of mixed content. The authors propose treating mid-range outputs (around 0.35–0.65) as a possible contradiction instead of forcing a positive/negative label.
• I3. A model's confidence can be turned into a few discrete bands (very/quite/rather positive or negative) that feed later steps as readable one-hot features, rather than an opaque float.
• I4. The best grid-search model was rejected after a check of its prediction distribution showed it labelled almost everything as one class.
• I5. The authors blame part of the failure on a genre mismatch between the auxiliary data (movie reviews) and the target data (tweets).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): the irony module in prag could score polarity separately on the predicate and on its complement or situation in the UD tree. It could then flag a strong positive-vs-negative mismatch and cite the two clashing constituents as its explanation.
• I2 (frames): an implicature or irony detector could treat "uncertain overall polarity combined with strongly polar parts" as a trigger for an ironic reading, not as noise.
• I3 (frames): speech-act or sentiment judgments in prag could be passed on as named confidence bands. Downstream rules and explanations could then mention them explicitly.
• I4 (learn): add a label-distribution sanity check to evaluation as a cheap negative control, so that degenerate classifiers or repair rules are caught before they enter training.
• I5 (learn): when choosing silver data or LLM-teacher examples for a skill like irony, prefer sources close to the target genre and record genre so mismatches can be audited.