📄 Dartmouth at SemEval-2022 Task 6: Detection of Sarcasm

Summary

The paper describes Team Dartmouth's entry to SemEval-2022 Task 6 (iSarcasmEval), which asks systems to recognise sarcasm in English and Arabic tweets across five subtasks. These are standalone binary detection (Task A), assigning one or more of six sarcasm categories to English tweets (Task B), and choosing the sarcastic member of a tweet pair (Task C). The team fine-tuned three transformer classifiers: RoBERTa BASE for English binary decisions, XLM-RoBERTa BASE for Arabic, and BERT BASE for the multilabel task.

Before encoding, each input was enriched with a second segment. For the binary tasks this segment held the tweet's verbs and adjectives, which were masked in the original text. For Task B it held TF-IDF keywords tied to the tweet's categories. Other choices were softened positive labels, tuned decision thresholds, and Arabic normalisation with CAMeL Tools. The team also tried simple data augmentation, which gave no clear gain.

The systems placed between 9th and 14th on every subtask. The authors attribute the remaining errors to label imbalance, very few examples for rare categories, and noisy annotations. The work shows how far plain fine-tuning goes on sarcasm and where it fails.

Key points

• Models:
  • RoBERTa BASE (125M parameters) for English Tasks A and C.
  • XLM-RoBERTa BASE for Arabic, with CAMeL Tools used for dediacritization, normalisation and POS tagging.
  • BERT BASE (110M parameters) for Task B, which uses six labels: irony, satire, sarcasm, rhetorical questions, overstatement and understatement.
• Training and decision settings:
  • Positive labels were lowered from 1 to 0.8 to soften the effect of noisy examples.
  • The binary threshold was 0.40; for Task C, the tweet in the pair with the higher score was labelled sarcastic.
  • The multilabel threshold was 0.30, set low because satire, understatement and overstatement had only 25, 10 and 40 training examples.
• Ranks: Task A English 13th, Task A Arabic 9th, Task B 14th, Task C English 12th, Task C Arabic 10th.
• Official scores:
  • Task A English: F1-sarcastic 0.386, accuracy 0.804.
  • Task A Arabic: F1-sarcastic 0.350.
  • Task C: accuracy 0.660 (English) and 0.680 (Arabic).
  • Task B: macro-F1 0.0590, with understatement scoring F1 0.0000.
• Augmentation results for Task B: category duplication, hand-written sentences and GPT-3-generated sentences all left macro-F1 close to the baseline of 0.156 (range 0.151–0.159).

Interesting ideas

• I1. Mask a tweet's verbs and adjectives in place and repeat them as a separate second segment, so the encoder must relate the evaluative words to the context they were removed from; the authors warn this may overfit to particular training words. — For Mova (frames): The irony detector in prag could likewise contrast a frame's evaluative predicates with the situation described around them, rather than scoring the words in isolation.
• I2. Lower the target value of positive labels (1 → 0.8) to express doubt about uneven annotation quality without discarding any examples. — For Mova (learn): Silver data on which the witness parsers only partly agree could carry such softened targets, so contested arcs pull the parser less strongly than unanimous ones.
• I3. Turn a pairwise question ("which of these two is sarcastic?") into a comparison of the scores from a single-item classifier, which removes the need for a threshold. — For Mova (answer): Multiple-choice school tests could be answered by ranking each option's plausibility score instead of requiring any option to pass a fixed cutoff.
• I4. The case study examines degenerate test inputs, such as one-word tweets and automated weather reports, to explain errors instead of relying only on aggregate scores. — For Mova (learn): Evaluation reports could include a slice of such low-context or non-linguistic inputs as a negative control, checking that the system abstains rather than inventing frames.
• I5. The Task B keyword segment was built from the tweet's own gold categories, which a model cannot know at test time; this makes the design a cautionary example of features leaking label information. — For Mova (learn): Feature-building and repair-rule induction should be audited so that nothing derived from gold labels reaches the inputs used at inference.