📄 Context vs. Human Disagreement in Sarcasm Detection

Summary

Earlier work agrees that context helps recognise sarcasm but rarely asks how much context is actually needed. The authors built a small controlled dataset from 24 two-person scenes from Friends in MUStARD, with character names changed. Each scene appears either as its original short transcript context (SC) or as a hand-written narrative long context (LC). The narrative version restores information that the transcript omits, such as visual, episode-level, or series-level facts. Lay native English speakers who did not know the show wrote free replies. Other raters then scored these replies for sarcasm in three settings: after LC, after SC, or with no context (NC). BERT, RoBERTa and DistilBERT, fine-tuned on the rest of MUStARD, classified the same items.

Humans and models both detect more sarcasm when any context is present. Making the context longer gives humans no extra gain. For models, longer context helps only on items where the human raters disagreed. Rater agreement was low, yet the effect of context stayed stable across raters. This matters for subjective NLP tasks.

Key points

• Data collection: 32 participants wrote replies. Fourteen were dropped for familiarity with the show, leaving replies from 18 people. SC averages about 26 words and LC about 66 words.
• Ratings: each of the 1,296 stimuli (432 per condition) was rated by 3 people on a six-point scale, which was then binarised.
  • Overall Fleiss' Kappa was 0.17, compared with 0.23 in the original MUStARD.
  • Agreement fell as context was added: NC 0.18, SC 0.13, LC 0.10.
• Effect of context on humans: roughly 38% of replies judged non-sarcastic without context became sarcastic once context was shown.
  • Mixed-effects models (lme4) found context significant.
  • LC beat SC for only one of the three raters.
• Model results: models were trained over 4 seeds × 5 folds for 10 epochs. On combined labels, BERT's macro-F was LC 0.55, SC 0.54 and NC 0.34; the other models showed the same pattern.
• Agreed vs. disagreed items:
  • On items where raters agreed, LC and SC both beat NC.
  • On split items, LC raised BERT's majority-label choice to 0.54, versus 0.51 for SC and 0.50 for NC.
  • Model choices matched each individual rater at similar rates, which explains why the overall pattern held despite low agreement.

Interesting ideas

• I1. The same utterance is presented under graded context conditions (none, transcript-only, full narrative), so the effect of context is measured directly rather than assumed.
• I2. The long context was written to recover facts the transcript lacks, on the view that sarcasm depends on shared background knowledge rather than just surrounding words.
• I3. Items are scored against each annotator separately and against the combined label. They are also split into unanimous and 2-vs-1 cases, and on split cases the model's choice counts as a better or worse option rather than simply right or wrong.
• I4. Annotators who might recognise the source material were excluded, so that prior knowledge could not shape their judgments.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): run the irony detector in prag on a sentence alone, with its preceding turn, and with the full story, and log which piece of context flips the decision, so each irony call can be explained by a cited cause.
• I2 (world): treat the world-of-the-text state (expectations, prior events, holdings) as the "long context" for irony, so a flagged contradiction can point to the specific world fact being violated.
• I3 (learn): when witness parsers disagree on silver data, keep the split arcs as graded majority/minority targets and report accuracy separately on unanimous and contested items, instead of forcing one gold answer.
• I4 (answer): when testing QA on well-known fables, add negative controls with renamed or altered tales, so that correct answers reflect reading the text rather than recalling the story.