📄 Detecting Hedge Cues and their Scopes with Average Perceptron

Summary

The paper describes a Fudan University system for the closed track of the CoNLL-2010 Shared Task on hedging, meaning language that marks a claim as speculative rather than established. The task has two parts: deciding whether a sentence contains uncertainty, and finding hedge cues together with the stretch of text each cue governs.

The authors handle both parts with linear classifiers trained by a modified averaged perceptron. For sentences, the score is the sum of per-word feature scores. Cues are found by tagging each word, and scopes by classifying every word–cue pair using surface-context and dependency-parse features. A heuristic then turns the labelled words into contiguous blocks. Training used the BioScope biomedical data and Wikipedia data, with no unlabeled text.

On biomedical text the system reached 77.86% F1 for uncertain sentences and 77.44% for cues, but only 19.27% for scopes. The authors trace the scope failure mainly to the block-building heuristic. The work shows that one simple, inspectable learner can cover several levels of the task, and that converting word-level decisions into spans is where things break.

Key points

• Data came from the CoNLL-2010 Shared Task: BioScope abstracts and full articles, plus Wikipedia paragraphs. About 18% of biomedical and 22% of Wikipedia sentences are speculative (2484 of 11,111 Wikipedia sentences). Biomedical cues are mostly single words, Wikipedia cues are longer, and a scope averages about 15.42 words.
• Training uses an averaged perceptron (Collins, 2002), changed so that weights are accumulated only after each full pass over the training data. This keeps training fast and limits overfitting.
• Features come from templates: 6 for the sentence classifier (word and POS unigrams/bigrams in a ±2 window) and 12 for the cue tagger. Scope classification adds features from a first-order projective dependency parser (McDonald et al., 2005).
• With gold cues, word-level scope F1 was 70.44 with context features alone and 75.94 with dependency features alone, where precision reached 93.77. Combining both gave 79.55. The official block-level scope result with predicted cues was only 19.27 F1 (P 21.87, R 17.23).
• Error analysis:
  • Cue mistakes come mostly from polysemy and collocations, for example "support", "can" and "should".
  • Words with no hedging meaning still received large weights, which the authors attribute to data sparsity.
  • Their own word-level cue scores are inflated compared with the official phrase-level scoring, especially on Wikipedia.

Interesting ideas

• I1. Scoring a sentence as the sum of its words' feature scores means any decision can be traced back to the specific words and contexts that pushed it toward "uncertain".
• I2. Framing scope detection as classifying each (word, cue) pair, with dependency-path features, turns an awkward span problem into many small decisions; the dependency features were notably precise.
• I3. The authors' heuristic merges labelled words into one maximal contiguous block per cue, which collapses nested or multiple scopes into a single blob and largely explains the low score.
• I4. Accumulating weights only once per training epoch, instead of after every example, keeps most of averaging's generalization benefit at much lower cost.
• I5. The gap between word-level scores and block-level official scores shows that metrics must be computed at the unit that downstream consumers actually use.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): a modality/hedging flag on a claim frame could carry a per-word contribution list, so the system can say which tokens ("suggest", "putative") made the claim tentative.
• I2 (tree): hedge and negation scope could be read off the UD tree as the cue's governed subtree, with pair features as a fallback where the subtree is ambiguous.
• I3 (world): when marking facts in the text's world as uncertain, scope should follow tree structure and allow nesting, so that only the hedged sub-claims get downgraded rather than the whole sentence.
• I4 (tree): the perceptron parser could use per-epoch averaging to speed up retraining on new silver treebanks without losing stability.
• I5 (learn): evaluations of scope, span or argument modules should report exact-span scores alongside token scores, and inspect top-weighted features for spurious cues as a sanity check.