📄 Uncertainty Detection as Approximate Max-Margin Sequence Labelling

Summary

The paper describes a system for the CoNLL-2010 shared task on uncertainty detection. Task 1 flags whole sentences as uncertain: hedged sentences in biological text and "weasel" sentences in Wikipedia, where a claim rests on a vague, unverifiable authority. Task 2 finds the hedge cue words inside a sentence and marks the span each cue governs.

The authors chose fast linear max-margin learners throughout:

• Biology: a sparse L1-regularised SVM.
• Wikipedia: an L2-regularised model.
• Cues and scopes: an approximate L2 max-margin sequence labeller over BIO tags.

Features combine surface forms with output from the Connexor FDG dependency parser. Feature templates were chosen by greedy forward search.

The corrected system reaches about 86.0 F1 for biology and 58.2 for Wikipedia. On Task 2 it scores 78.5 for cues and 39.6 for full scope resolution. Training takes minutes and prediction takes seconds on an ordinary PC. The work shows that linguistically informed but cheap structured prediction is a workable approach to hedges. It also shows that much of the remaining error comes from tokenisation, format conversion and strict scoring rather than from the model itself.

Key points

• Models: The biological sentence classifier uses an L1-regularised hinge-loss SVM from the library by Fan et al. (2008). L1 was preferred because a small set of lexical cues dominates the bio annotation, while the noisier Wikipedia weasel data fared slightly better with L2.
• Class imbalance: Each class is weighted by the inverse of its share of the training data, as a rough way of optimising F1.
• Feature templates: SENLEN gives sentences three length-dependent bias terms, with cut points at 18 and 32 tokens. Other templates are DOCPT, TOKEN, LEMMA, QUANT, HEAD, DEPREL, SYN and MORPH, plus Cartesian products of templates. The structural features come from the Connexor FDG parser.
• Results: The official results were 85.2 (bio), 55.4 (Wikipedia) and 62.5 for cues. The full Task 2 score was only 2.1 because of bugs. After fixes the scores rose to 85.97, 58.25, 78.52 (cues) and 39.64 (scopes).
• Error analysis:
  • About 80 scope errors come from misaligned tokens such as parentheses when converting back to the BioScope XML.
  • Weasels were over-predicted because the parse did not separate subject and non-subject quantified NPs.
  • Negation was captured only through adjacent bigrams, so longer-range negations were missed.

Interesting ideas

• I1. Feature templates are added greedily, and each candidate set is scored as the cross-validated mean F1 minus one standard deviation, which favours configurations that are stable across folds rather than ones that are lucky on average.
• I2. The "certain" class has no explicit markers and is treated as the default, captured by a bias term, while only the marked "uncertain" class receives lexical and syntactic features.
• I3. Predicted BIO spans are rebuilt with a stack that forces scopes to be either disjoint or nested and never to begin or end inside a cue, with a whole-sentence fallback when this fails.
• I4. The choice between sparse L1 and dense L2 regularisation follows from how the signal is distributed: a few strong lexical cues call for L1, while diffuse, noisy signals suit L2.
• I5. The authors argue that exact-match span scoring is too harsh, since shifting a boundary over one unimportant token voids an otherwise sensible analysis, and propose token-level or fuzzy scoring instead.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Use the same mean-minus-spread criterion when choosing parser feature sets or self-induced repair rules, so that a gain must be consistent across data splits before it is accepted.
• I2 (frames): Speech-act and epistemic-stance detection in prag could treat plain assertion as the unmarked default and attach explicit evidence only to hedged, reported or doubted statements, which keeps the explanation short and readable.
• I3 (frames): Scopes of hedges, negation and modal operators over frames could be decoded with the same well-formedness stack, and the fallback cases could be logged as explainable low-confidence decisions.
• I4 (tokens): For tagger or lexical-class features dominated by a handful of trigger words, sparse regularisation could give smaller and more inspectable models whose surviving weights read like a lexicon.
• I5 (learn): Evaluation of scopes, constituents and frame arguments could report token-overlap scores alongside exact match, so that boundary noise from tokenisation is separated from real semantic errors.