📄 HedgeHunter: A System for Hedge Detection and Uncertainty Classification

Summary

HedgeHunter is a CoNLL-2010 shared task system that separates reliable statements from speculative ones, so that information extraction can drop or down-weight doubtful facts. It works in two supervised stages. First, it labels each word as a hedge cue or not. This uses logistic regression trained with Quasi-Newton optimisation over the word form, lemma and POS of the target word and of three neighbours on each side; tags come from a Stanford JavaNLP maximum-entropy tagger trained on the PennTreebank. Second, it labels each sentence as certain or uncertain. The features are how many cues stage one found, the identities of the first four cues, and binary vocabulary features.

The system was tested on biomedical abstracts, full articles and Wikipedia "weasel word" data, both in-domain and with merged training sets. Cue detection had good precision on biomedical text but weak recall, and it dropped sharply on Wikipedia. Sentence classification was moderately below the best shared-task systems. The work matters because it shows that hedging is largely lexical and that the number of cues carries signal. It also argues for scoped, graded uncertainty instead of sentence-level yes/no labels.

Key points

• Data. The corpus has 11,871 abstract sentences, 2,670 full-article sentences and 11,111 Wikipedia sentences. Wikipedia cues come from editor-applied weasel-word tags, which are noisy and sparse. A random 10% was held out for testing.
• Cue detection, word level (P / R / F1).
  • HedgeHunter: abstracts .8758 / .5800 / .6979; articles .8704 / .4052 / .5529; Wikipedia .5453 / .2434 / .3369.
  • Morante and Daelemans (2009): abstracts .9081 / .7984 / .8477.
• Closed-task sentence classification (F1).
  • HedgeHunter: biomedical .7997, Wikipedia .5390.
  • Best shared-task systems: Tang .8636 (biomedical), Georgescul .6017 (Wikipedia).
• Cross-domain sentence classification (F1).
  • HedgeHunter: biomedical .6833, Wikipedia .5272.
  • Best shared-task systems: Li .8545 (biomedical), Ji .5874 (Wikipedia).
  • On Wikipedia, HedgeHunter had higher precision than Ji (.7173 vs .6266).
• Feature findings.
  • The heaviest-weighted sentence features were cue presence and cue count. This contradicts Szarvas (2008), who found cue counts unhelpful.
  • Vocabulary features partly made up for the low cue recall.
  • The author attributes the recall gap to the lack of chunk, constituency or dependency features.

Interesting ideas

• I1. The sentence stage is trained on cues predicted by stage one rather than gold cues, so training conditions match deployment and unlabeled text can later be folded in.
• I2. More hedge cues in one sentence raised the chance it was uncertain, which suggests uncertainty is graded rather than binary.
• I3. A hedge governs only part of a sentence, so discarding every relation that shares a sentence with a cue is too blunt.
• I4. Multi-word cues like "ought to" are hard to catch with independent per-word labels; the author expects dependency links to reveal which tokens form one cue.
• I5. The planned semi-supervised loop has three steps: mine unlabeled words with high class-conditional likelihood, treat them as new cues, then retrain both stages. The author explicitly flags drift as a precision risk.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): When training frame or world components, feed them Mova's own parsed and tagged output rather than gold trees, so learned weights reflect real upstream errors.
• I2 (world): Store a scalar certainty on each fact in the world of the text, strengthened by each stacked epistemic marker ("might possibly"), instead of a true/doubtful flag.
• I3 (frames): Use the UD subtree headed by or attached to a hedge cue (may, suggest, seems) to mark only the frames inside that scope as non-factual, leaving sibling clauses asserted.
• I4 (tree): Detect hedge cues as dependency-linked units (e.g. aux plus mark/xcomp patterns), so that one cue object is passed to prag rather than scattered token flags.
• I5 (learn): Expanding a hedge lexicon from silver data is worth trying, but each round should be gated by negative controls (known-certain sentences) to catch drift.