📄 Humans Require Context to Infer Ironic Intent (so Computers Probably do, too)

Summary

The paper argues that verbal irony often cannot be detected from the words of an utterance alone. Detection also depends on context outside the text, such as who is speaking and what the discussion is about. To test this, the authors built the reddit irony corpus (β 1.0): 3,020 comments from six subreddits, split into 10,401 sentences. Three undergraduates labelled each sentence as ironic or not. The custom annotation tool let annotators ask for extra context: the surrounding thread, the linked page, or the author's earlier comments. The tool logged every such request along with a confidence rating. Two logistic regressions showed two things. Annotators asked for context much more often on comments that turned out to contain irony. A bag-of-words classifier also made its errors disproportionately on those same comments. The conclusion is that intrinsic text features put a ceiling on irony detection, and future models should use speaker and conversation context. The corpus was released publicly to support that work.

Key points

• Corpus: 3,020 comments and 10,401 sentences from r/politics, r/conservative, r/progressive, r/atheism, r/Christianity and r/technology. Sentences were segmented naively by punctuation.
• Agreement: raw average agreement was 0.844, but average pairwise Cohen's Kappa was only 0.341. Annotators were deliberately given little guidance on what counts as irony.
• Context requests: annotators asked for context on 30% of comments on average, ranging from 12% to 56% per person. Their confidence rose after viewing it, and they often changed their label; one annotator flipped from ironic to unironic 86 times.
• Regression results: the coefficient linking context requests to irony was 1.508 (95% CI 1.326–1.690, p < 0.001). The coefficient linking context requests to classifier errors was 0.971 (CI 0.810–1.133, p < 0.001), after adjusting for the true label.
• Baseline classifier: binary unigram and bigram features (top 50,000, stop-words removed) plus punctuation and emoticon flags. It reached recall of about 0.496 and average precision of about 0.315. The tokens it relied on most heavily for prediction included "!", "shocked", "oh", "yeah" and "guys".

Interesting ideas

• I1. During annotation, record when a human had to look beyond the text, and treat that request as a measurable signal of how much the item depends on context.
• I2. Regress a model's errors on a human-difficulty indicator while controlling for the gold label, to show why a model fails rather than only how often.
• I3. Model the speaker's history as context: an utterance that contradicts the author's known stance is likely ironic, as in the Ted Cruz example.
• I4. Ask annotators for both a forced judgement before seeing context and a final judgement after it, then compare the two to measure how much context changes interpretation.
• I5. Some sarcasm markers are shallow but intuitive, such as exclamation marks and words like "yeah" or "oh". They serve as weak cues but cannot decide on their own.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): silver-data tooling could flag items where witness parsers or LLM teachers needed extra discourse to agree, and keep those as a separate "context-dependent" evaluation slice.
• I2 (learn): error analysis could test whether failures cluster on sentences that humans mark as context-needing, which would separate missing-knowledge errors from parser bugs.
• I3 (frames): the irony module in prag could compare a speech act's literal evaluation with stances attributed to the same speaker elsewhere in the text, and flag a clash as possible irony with an explicit reason.
• I4 (answer): fable and tale QA tests could pair a sentence-only question with a full-story question, checking whether readers really use the world of the text.
• I5 (frames): treat such markers as low-weight evidence feeding an explainable irony hypothesis, never as the sole trigger for reversing a frame's polarity.