📄 Committed Belief Annotation and Tagging

Summary

Search and text processing that only look for propositions miss an important layer of meaning. "GM may lay off workers" and "GM will lay off workers" contain the same proposition, yet they license very different conclusions. This pilot study treats the writer's stance toward each proposition as a target of annotation and prediction. Each predicate gets one of three labels: committed belief (CB), non-committed belief (NCB), or not applicable (NA). Desires, questions and requirements fall under NA, and only the writer's perspective is considered.

The authors wrote an annotation manual and labelled a small multi-genre English corpus of about 10K words. Annotators agreed strongly. A supervised tagger then learned both to find annotatable tokens and to assign their class, using a ±3 token window and lexical, POS and chunk features. The best feature set reached 53.97 overall F, roughly 23 points above a token-only baseline. POS carried most of the gain, and NCB was much harder to predict than the other classes.

The work shows that writer commitment can be learned from surface syntax. It also points to deeper structure as the next step.

Key points

• The corpus is about 10K words drawn from newswire, blogs, email, letters and transcribed dialogue. 70% of it was doubly annotated, covering 6188 potentially annotatable tokens, and annotators agreed 95.8% of the time on both annotatability and class.

• The most common disagreements were:

  • NONE vs CB: 103 tokens
  • NCB vs NONE: 48 tokens
  • NCB vs CB: 32 tokens

• The feature types were:

  • token (TOK)
  • character NGRAM (NG)
  • AlphaNumeric (AN)
  • POS
  • verb type (VT, with all modals treated as modal)
  • CHUNK

  Results were averaged over 5 test folds.

• The TOK baseline scored 30.3 overall F. TOK+POS reached 53.5, and TOK+AN+POS+CHUNK reached the best score of 53.97. Adding NG to richer feature sets lowered performance.

• The best per-class scores were about 56 for CB and about 59.6 for NA, but only about 22.7 for NCB. Most errors came from missing an annotatable token: under TOK+POS, 55% of CB items were tagged O.

Interesting ideas

• I1. Annotators labelled what the writer believes about each event rather than marking cue words, so the data can later reveal how language signals commitment without circular bias.
• I2. All reported speech is labelled NCB by convention, even when world knowledge suggests the writer agrees, which keeps the labels consistent and decidable.
• I3. In predicative constructions the annotatable unit is the nominal, adjectival or prepositional head rather than "be", which also covers small clauses.
• I4. Errors are split into identification errors (is this token annotatable?) and classification errors, and identification turned out to be the main bottleneck for both humans and the model.
• I5. A belief verb embedded under a reporting verb (e.g. "believed" inside "said") was mislabelled as CB because flat chunks miss the scope. The authors expect deeper syntax to fix this.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): attach a writer-commitment label (CB/NCB/NA) to every frame in prag, and learn its cues from the data rather than hard-coding a cue list.
• I2 (world): propositions under say/claim verbs should enter the world of the text as facts attributed to a source, not as asserted facts, so answers can cite who holds them.
• I3 (tree): this matches the UD copula analysis, so commitment labels can sit directly on the UD predicate head with no special case for "be".
• I4 (learn): score frame/predicate detection separately from label assignment, so silver data and repair rules target the step that actually fails.
• I5 (frames): propagate commitment down ccomp/xcomp chains in the UD tree, weakening CB to NCB under reporting or modal heads, as an explainable rule rather than a window feature.