📄 PALI-NLP at SemEval-2022 Task 6: iSarcasmEval- Fine-tuning the Pre-trained Model for Detecting Intended Sarcasm

Summary

The paper presents the system behind PALI-NLP's entry to SemEval-2022 Task 6 (iSarcasmEval). The team worked on Subtask B, a multi-label problem: assign a sarcastic English tweet one or more of six categories (sarcasm, irony, satire, understatement, overstatement, rhetorical question), and give a non-sarcastic tweet no label. The approach is deliberately simple. Pre-trained encoders (BERT, RoBERTa, BERTweet) are fine-tuned with a plain classifier on the [CLS] representation. The training data combines the official iSarcasm-2022 set with extra sarcastic tweets from the earlier iSarcasm corpus.

Most of the gains come from training choices rather than architecture:

• a re-weighted BCE loss,
• multi-sample dropout,
• PGD adversarial perturbation of the embeddings,
• optionally a supervised contrastive loss.

Final predictions come from a rank-weighted ensemble that votes separately on primary and secondary labels. The system placed 1st in Subtask B with macro-F1 0.1630. Its best single model reached 0.1675. The work shows that matching the pre-training domain and tuning the training recipe can outperform complex classifier designs on noisy social-media text.

Key points

• BERTweet clearly outperformed the general-domain encoders (macro-F1 0.1417 vs 0.0991 for RoBERTa-base and 0.0766 for BERT-base-uncased), which the authors attribute to its Twitter pre-training corpus.

• 777 extra samples from iSarcasm (Oprea and Magdy, 2019) were added. Re-weighting the BCE loss raised BERTweet from 0.1417 to 0.1564, and this became the baseline.

• Each technique was tested against that baseline:

  Setting	Macro-F1
  Baseline	0.1564
  6-layer multi-sample dropout	0.1578
  PGD (adversarial rate 0.5)	0.1595
  PGD + multi-sample dropout	0.1675
  SupCon loss alone	0.1529
  SupCon + multi-sample dropout	0.1670

• Training setup: batch size 128, sequence length 64, decision threshold 0.2, layer-wise learning-rate decay, AdamW with Lookahead, early stopping on three validation checks per epoch.

• The label distribution has a long tail. Understatement received an F1 of 0 in every reported configuration.

Interesting ideas

• I1. The authors noticed that the six labels form a two-level hierarchy (sarcasm/irony as primary, the other four as secondary) and ran separate ensemble votes for each level, which makes use of label structure without changing the model.
• I2. Ensemble members get voting weights equal to the square root of the inverse of their validation rank, a cheap scheme that favours stronger models without letting one dominate.
• I3. Mentions, hashtags and URLs are normalized to placeholder tokens, while emojis are deliberately kept because they may signal a gap between literal wording and intended sentiment.
• I4. The ablations show that components interact: the contrastive loss hurt on its own but helped when paired with multi-sample dropout, so single-addition tests can be misleading.
• I5. Pre-training on in-domain text (BERTweet) mattered more than any later fine-tuning trick in this task.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): the irony module in prag could first decide whether an utterance is ironic or sarcastic and only then choose a rhetorical subtype, keeping each decision separately explainable.
• I2 (learn): when several witness parsers disagree while building silver data, their votes could be weighted by rank on a held-out treebank instead of counted equally.
• I3 (tokens): the en tokenizer could map social-media artefacts to typed placeholders but keep emojis as tokens whose sentiment the irony step can compare with the literal text.
• I4 (learn): evaluation of new training tricks or repair rules should include pairwise combinations alongside negative controls before a component is discarded.
• I5 (tree): for informal or tweet-like input, gparse could swap in a domain-matched encoder, or be adapted on such text, before effort goes into parser-side tuning.