📄 A Lucene and Maximum Entropy Model Based Hedge Detection System

Summary

Lin Chen and Barbara Di Eugenio built a system for the CoNLL-2010 shared task on hedge detection. The task is to decide whether a sentence expresses uncertainty, and their system also points to the specific hedge cues that cause it. The cue annotations in the training data serve as seeds. A candidate generator then rewrites them: it widens single-word cues with neighbouring tokens, drops proper nouns, and substitutes POS tags for some words. Each candidate is scored by support (how many sentences it matches) and confidence (what share of those sentences are uncertain). Apache Lucene stores every sentence with words and POS tags at the same index positions, so a cue can mix words and tags and still match with small gaps. A separate MaxEnt classifier for each retained cue makes the final call on every sentence the cue matches. The submitted run placed third on Wikipedia. Later threshold tuning raised the F-scores to 0.612 on Wikipedia and 0.802 on the biological data. The work shows that explicit, inspectable cue patterns combined with small local classifiers give competitive uncertainty detection.

Key points

• Lucene setup:
  • Each sentence is one Lucene document, with OpenNLP tokenization and tagging.
  • Every word and its POS tag share an index position (position increment 0).
  • SpanNearQuery permits a position gap of 2 between cue tokens.
• Single tokens made poor cues:
  • Thresholds of support 20 and confidence 0.4 left only 42 tokens on Wikipedia.
  • Thresholds of 5 and 0.3 gave 279 tokens, but these included junk such as is and his.
• POS replacement covers VBN, NNS, DT and CD. Examples of resulting cues are it is VBN and some of NNS.
• MaxEnt features, trained with OpenNLP MaxEnt for 100 iterations per cue:
  • sentence location (title or body);
  • sentence length;
  • cue position (start, middle or end);
  • word and POS collocations in a window of 5.
• Results:
  • The official run (confidence 0.35, support 5) reached F 0.5741 on Wikipedia, ranking third. On the biological data it reached F 0.7692 and ranked third from last, because only the smaller training file was used.
  • The best thresholds were confidence 0.5 and support 5. At that setting the generator produced 52 cues with 100% confidence on Wikipedia and 332 on the biological data.

Interesting ideas

• I1. Placing a word and its POS tag at the same index position lets a single query freely mix literal words and tag slots, which turns corpus pattern counting into cheap lookups.
• I2. Scoring mined patterns by support and confidence, then sweeping both thresholds, makes the trade-off between coverage and reliability explicit: higher confidence raised precision and lower support raised recall.
• I3. Seed annotations are generalized by three deliberate edits: extending with context, deleting proper nouns, and replacing content words with POS tags. Each edit is then re-scored against the corpus rather than trusted blindly.
• I4. Training one small classifier per cue keeps each decision local and traceable to the pattern that fired. As a result, the system reports why a sentence counts as uncertain, not just that it does.
• I5. Tolerating a small gap between cue tokens lets cues like "is said" survive inserted modifiers such as "widely".

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Indexing silver treebanks this way would let us quickly count how often a candidate repair-rule pattern such as "it is VBN" occurs and how often it co-occurs with a known error.
• I2 (learn): Self-induced repair rules could be admitted only above tuned support and confidence thresholds, with the sweep itself reported in evaluation.
• I3 (frames): The prag crate could grow its inventory of epistemic and hedging markers (e.g. "it is believed", "many NNS") from a few seeds, keeping only variants the data supports.
• I4 (world): Statements entering the world of a text could carry an uncertainty flag tied to the specific hedge cue and a per-cue check, so answers can say a fact is only reported as believed.
• I5 (tree): Hedge patterns could be matched along dependency paths (aux/advmod attachments to the verb) instead of surface gaps, giving the same robustness in an explainable, tree-based form.