📄 Annotating Expressions of Appraisal in English

Summary

The Appraisal framework from systemic functional linguistics classifies how English speakers express evaluation. It covers attitude (affect, judgement, appreciation), engagement (stance toward other people's views) and graduation (strengthening or weakening). Before this work, no machine-readable Appraisal annotations were publicly available, so there was nothing to test automatic Appraisal analysers against. The authors built a single-genre corpus of book reviews from four British newspapers. Two trained annotators labelled it independently, marking spans of any length with one of 32 fine-grained types, plus polarity or scaling.

Agreement could not be measured with Cohen's κ, because the units are free-form spans. The authors instead borrowed MUC-7-style precision, recall and F-score, and counted overlapping spans as matches. Agreement fell steadily as the categories grew more specific. It was higher for attitude than for engagement or graduation.

The released corpus, and the analysis of why annotators disagreed, give a realistic picture of how hard fine-grained evaluation analysis is. This matters for later sentiment and subjectivity research.

Key points

• The corpus has 38 unedited book reviews, gathered on two dates in 2006 from four papers: The Times, The Telegraph, The Independent and The Guardian. It totals 36,997 tokens in 1,245 sentences.
• Annotator d produced 3,176 annotations and annotator j produced 2,886. Annotation ran in two rounds, with a discussion of disagreements between them; first-round labels were left unchanged.
• Choosing any span with any of 32 classes gives roughly 6.5 × 10⁸ possible choices in this corpus, which rules out κ. MUC-7 measures were used instead.
• Mean F-score dropped from 0.698 at the most abstract level of the hierarchy to 0.395 at the most specific level. If only annotations both judges share are trusted, 2,223 usable annotations remain.
• Typical disagreements include:
  • Where a class boundary lies: how strong a word must be to count as Pronounce or Mass.
  • Who is being evaluated: Satisfaction versus Propriety, as in "neglected and wronged by critics".
  • Span size: Proximity (Space) versus Capacity.

Interesting ideas

• I1. When annotation units are free spans and κ is meaningless, treat agreement like an information-extraction evaluation (precision, recall, F) and accept partially overlapping spans as matches.
• I2. Report agreement separately at each depth of a label hierarchy, because coarse categories can be reliable while fine ones are not.
• I3. Treat the intersection of two independent annotations as a smaller but trustworthy subset instead of discarding the whole noisy resource.
• I4. Many disagreements came from unspecified targets (who is appraised) and from forcing one type per unit. This argues for annotations that record both the evaluator and the evaluated entity and allow several readings.
• I5. Represent sentiment as a structured frame (attitude type, orientation, force, focus, polarity), as in the reviewed Appraisal Groups work, instead of as unigram features.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Witness parsers that disagree on span edges (phrases, frame arguments) could be scored with overlap-tolerant P/R/F. Near-identical analyses would then not be counted as conflicts in silver-data voting.
• I2 (world): Ontology or verb-class labels could store a confidence per level. Reasoning would fall back to the coarsest level that is reliably agreed rather than trusting a shaky fine label.
• I3 (learn): This fits naturally with multi-witness silver data. Keep only items where independent sources coincide as training gold, and route the rest to an LLM teacher tier.
• I4 (frames): An evaluation frame in prag could have explicit appraiser and target roles and permit multiple attitude types per expression. "Wronged by critics" could then yield both the victim's dissatisfaction and a reproach of the critics.
• I5 (frames): Graduation words like "deceptively", "sort of" or "massive" could attach as force/focus modifiers on a frame. This keeps evaluative meaning explainable rather than buried in bag-of-words scores.