📄 SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter

Summary

This SemEval-2015 shared task asks how well sentiment analysis works on tweets that are rich in irony, sarcasm and metaphor. In such tweets, the literal wording often signals a different polarity than the meaning the author intends. Systems were not asked to detect which figure of speech a tweet uses. Instead, they had to give every tweet a detailed sentiment rating ranging from -5 to +5, which covers 11 possible values.

The organisers harvested candidate tweets with the Twitter4j API, using figurative hashtags and marker words. They then widened the seed hashtags with an LSA model. The resulting tweets were annotated on CrowdFlower, with spam filtering and reliability-weighted score aggregation. Submissions were scored by cosine similarity and by mean squared error against the gold scores.

The best team, ClaC, reached a cosine of 0.758. Across systems, sarcastic and ironic tweets proved fairly tractable, while metaphorical and "other" tweets remained much harder. The released data gives researchers a benchmark for studying how figurative language interacts with sentiment.

Key points

• Datasets:
  • A trial set of 1025 tweets came from users known for figurative style, such as comedians. Seven in-house annotators labelled it, and native English speakers' scores counted double.
  • The training and test sets contain 8000 and 4000 tweets, annotated on CrowdFlower.
• Collection: Tweets were gathered in June 2014 using hashtags like #sarcasm, #irony, #not and #yeahright, plus words such as "literally" or "figuratively." URLs were stripped, but hashtags were kept. Very short tweets were discarded.
• Spam control:
  • Trial tweets served as hidden test questions, and workers below 70% accuracy on them were removed.
  • Workers whose scores never varied (zero standard deviation) were flagged as scammers.
  • So were workers with 60% or more outlier judgements.
  • After cleaning, each tweet kept 5–7 annotations.
• Results: ClaC ranked first (cosine 0.758, MSE 2.117), followed by UPF (0.711) and LLT_PolyU (0.687). Sarcasm and irony cosines were roughly 0.87–0.92 for top systems, while metaphor scores were mostly under 0.66.
• Evaluation: Systems were ranked with two metrics, cosine similarity and MSE. The detailed results cover 15 teams and their multiple runs.

Interesting ideas

• I1. Measure figurativeness extrinsically. Rather than labelling which device a tweet uses, the task scores sentiment on tweets where figurative usage is common. This avoids disputes over fuzzy category boundaries and could apply to any phenomenon that is hard to define, such as hate speech subtypes.
• I2. Collect data with cheap signals. Self-tagging hashtags and marker words act as weak labels for gathering candidate data, and a semantic model (here LSA) expands the seed query set. This is a cheap bootstrap for collecting low-resource or niche corpora, for example Ukrainian social media.
• I3. Weight annotators by reliability. Each annotator gets a reliability score equal to their share of in-range answers on gold test questions. The final label is a weighted average using those scores, which is more principled than a plain mean when crowd quality varies.
• I4. Filter spam on several fronts. The scheme combines gold-question accuracy, zero-variance detection, and per-item outlier rates relative to the item's spread. Together these give a reusable recipe for cleaning crowdsourced ordinal labels.
• I5. Report results per subset. Splitting scores by sarcasm, irony, metaphor and other shows where systems actually fail, which a single aggregate number would hide. This makes the approach useful for diagnosing any benchmark with a mixed composition.