📄 Using “Annotator Rationales” to Improve Machine Learning for Text Categorization

Summary

Building supervised classifiers needs many labeled examples, and labeling is costly. Zaidan, Eisner and Piatko propose having annotators state not only the label of an example but also why they chose it. While classifying a document, the annotator highlights the text spans ("rationales") that drove the decision. The authors added such highlights to the movie review polarity corpus of Pang and Lee.

To use rationales in an SVM, they build "contrast" documents by masking one or all highlighted spans. They then require the model to score the original document more confidently than each contrast version, by a tunable margin. These extra constraints become weighted pseudo-examples, so a standard soft-margin SVM solver can still be used. The method improves sentiment classification significantly, even when only part of the training set carries rationales. A separate study showed reasonable agreement between annotators on which spans matter.

The broader point is that, in some settings, an annotator's time may be better spent explaining a few labels than labeling more examples. This gives a new way to ease the annotation bottleneck, alongside active and semi-supervised learning.

Key points

• Data: version 2.0 of the polarity dataset, drawn from work by Pang et al. (2002) and by Pang and Lee (2004).
  • It has 1000 positive and 1000 negative IMDb reviews by 312 authors, split into 10 folds.
  • Author A0 marked rationales on folds 0–8 (1,800 documents).
  • A0 averaged 8.55 rationales per review, with a median of 8 and a mode of 7.
• Method: each contrast example yields a constraint w·(x − v) ≥ µ(1 − ξ), with its own slack cost C_contrast.
  • Dividing by µ turns these constraints into ordinary pseudo-examples (x − v)/µ.
  • A 0 is prepended to each pseudo-example instead of the bias 1.
  • The values of µ, C and C_contrast are tuned on held-out data.
• Agreement study: 150 randomly chosen documents were annotated by four annotators.
  • Their thoroughness varied widely, from 5.02 rationales per document (A1) to 11.36 (AY).
  • 91.4% of A1's rationales overlapped with spans marked by at least one other annotator.
• Annotators needed no knowledge of the classifier's features; they simply boldfaced phrases in an HTML editor.
• The rationale-annotated corpus is released publicly at cs.jhu.edu/~ozaidan/rationales.
• Proposed future work is a generative, log-linear model that jointly predicts labels and rationales.
  • It would model the annotator's limited patience, sloppiness and bias.
  • It would remove the need for contrast examples.

Interesting ideas

• I1. Explanations as cheap supervision. Asking annotators to highlight supporting evidence while they label costs little extra effort. It addresses the credit-assignment problem directly, so it could help any low-resource labeling effort, such as Ukrainian text classification.
• I2. Contrast examples via masking. Removing the evidence a human marked as important should make the model less confident. This turns a vague "focus here" hint into concrete training constraints, and the idea can be reused wherever spans can be deleted or masked.
• I3. Reducing a new constraint type to the standard solver. Rescaling the difference vectors lets off-the-shelf SVM software handle the margin between originals and contrasts. This is a handy pattern for adding side information without writing a custom optimizer.
• I4. Feature-agnostic annotation. Annotators mark raw text rather than judging features. The same rationale data can therefore be reused with different feature sets or learners, which makes the resource last longer.
• I5. "Lazy annotator" simulation. The authors kept only a random fraction of A0's rationales and found no notable difference in results. This suggests annotation guidelines could deliberately ask for only the most obvious evidence to save time, and that it is worth testing such cost–benefit trade-offs before large annotation campaigns.