📄 SemEval-2007 Task 14: Affective Text

Summary

This SemEval-2007 shared task asks systems to recognise emotions and valence in English news headlines, as a way to study how emotions relate to lexical semantics. The organisers built a corpus of headlines from sources such as the New York Times, CNN, BBC News and Google News. It has a 250-item development set and a 1,000-item test set. Six annotators scored each test headline with slide bars: Anger, Disgust, Fear, Joy, Sadness and Surprise on a 0–100 scale, and valence on a −100 to 100 scale. The task was deliberately unsupervised, with no training data released, though teams could build their own. An optional word list from WordNet Affect was provided. Systems were scored in two ways: Pearson correlation against the averaged gold scores (fine-grained), and accuracy, precision and recall after thresholding (coarse-grained). Five teams took part, with five valence systems and three emotion systems. The best valence correlation was about 0.48 (CLaC). Emotion labelling proved much harder. Annotators agreed only moderately, yet systems still fell well below that agreement, which leaves clear headroom for future work.

Key points

• Inter-annotator agreement was measured by correlating each annotator with the mean of the other five. Valence reached 78.01. Among emotions, Sadness was highest (68.19) and Surprise lowest (36.07).
• The coarse-grained setting converts scores into classes:
  • emotions become binary, with present meaning a score of 50 or more;
  • valence becomes −1/0/1, with cut-offs at ±50.
  • Precision and recall leave out the neutral class, because accuracy is inflated by the many neutral headlines.
• Valence Pearson r by system: CLaC 47.70, UPAR7 36.96, SWAT 35.25, CLaC-NB 25.41, SICS 20.68. The supervised CLaC-NB had low correlation but the highest coarse F1 (42.43).
• The systems took very different approaches:
  • UPAR7: rules over Stanford parser dependencies, using SentiWordNet and WordNet Affect.
  • SICS: a TFIDF word space built from the LA Times corpus.
  • CLaC: a 10,809-word fuzzy sentiment lexicon plus 450 valence shifters scoped with MiniPar.
  • UA: web-search PMI.
  • SWAT: a unigram model trained on 1,000 extra self-annotated headlines, with Roget-based synonym expansion.

Interesting ideas

• I1. Graded slide-bar annotation instead of binary labels. Annotators recorded how strongly each emotion was present, not just whether it was. The same graded data can feed both regression-style and classification-style evaluation, which is useful for any subjective labelling project.
• I2. Leave-one-out agreement against the group mean. Correlating each annotator with the average of the others gives a robust agreement figure for continuous scores. It also doubles as a realistic upper bound for judging how far systems are from human performance.
• I3. Releasing no training data on purpose. Withholding training data pushes participants toward lexical and semantic resources rather than plain text classification. This is a useful design for low-resource languages, where shared tasks could test the quality of lexicons rather than the size of corpora.
• I4. Seed-vector valence in a word space. SICS summed the vectors of 8 positive and 8 negative seed words, then labelled each headline by cosine similarity to those two vectors. This cheap bootstrap needs only an unannotated corpus and could be ported quickly to a new language.
• I5. Valence shifters with an explicit action and scope. CLaC gave each negation or intensifier a defined action and a parse-determined scope. This makes lexicon-based sentiment scoring more accurate and easy to inspect, and it suits morphologically rich languages if paired with a dependency parser.