📄 Named Entity Recognition through Classifier Combination

Summary

The paper addresses named entity recognition for the CoNLL-2003 shared task in English and German. Its approach is to merge several statistical taggers that differ in how they make decisions. Four classifiers are used: RRM (a regularized winnow method), MaxEnt, fnTBL (transformation-based learning in a boundary-then-classification setup) and an HMM in the style of Bikel et al. The HMM is dropped for German. All of them label words by their position inside an entity and share a rich feature set: word windows, lemmas, POS tags, chunks, affixes and word-shape flags. The authors compare several combination schemes: equal voting, weighted voting, two linear-interpolation models whose parameters are estimated by 5-fold cross-classification, and RRM used as a meta-learner. Using only training-data features, combination lifts English development F from about 89.9 to about 91.6, roughly a 17% error reduction. When gazetteers and outputs of external NE taggers are added, the RRM combiner reaches 93.9 on English development data. German gains less, because its data is sparser. The work shows that diverse learners plus a learned combiner can absorb many information sources.

Key points

• Individual classifiers on English (dev/test F):
  • HMM: 82.0 / 74.6
  • fnTBL: 88.1 / 81.2
  • MaxEnt: 90.8 / 85.6
  • RRM: 92.1 / 85.5
• Combination results on English dev without gazetteers (best single classifier at 89.94 F):
  • equal voting: about 91.0, averaged over 30 random tie-breaking trials
  • weighted voting: 91.56
  • Model 1, using P(C|w): 90.9
  • Model 2, using P(C|C_i): 91.64
  • RRM combiner: 91.63
• Extra resources:
  • English: gazetteers of 50,000 cities, 80,000 proper names and 3,500 organizations, plus two IBM NE taggers trained on 1.7M words with 32 categories.
  • With these, the RRM combiner reaches 93.9 F on English dev (about 21% error reduction), while Model 2 manages only 92.4.
• Final overall F:
  • English: 93.87 dev, 88.76 test
  • German: 71.51 dev, 72.41 test
  • German recall is weak, with MISC recall around 42%.
• German specifics:
  • The rate of unknown words is about double that of English.
  • Capitalization is a weak cue, since all German nouns are capitalized.
  • A small gazetteer collected by hand in about two person-hours added roughly 1.5 F on testa and 0.6 F on testb.

Interesting ideas

• I1. Choose ensemble members for diversity along several axes. The authors pick learners that differ in decision style (few discriminant features vs. all features), in how they handle sparseness, in their search strategy, and in their output type. This is a useful checklist when building any ensemble, because members that disagree in different ways leave more room for gains.
• I2. Train the combiner on cross-classified outputs. Each base model is trained on four fifths of the training data and tags the remaining fifth, so the combiner learns from realistic, non-overfit predictions. This trick transfers to any stacking setup where training data is limited.
• I3. Let votes carry partial credit. Instead of a hard vote, each classifier's output is mapped to a distribution over the true class, P(C|C_i). This captures systematic confusions between classes and beat plain voting here.
• I4. Use a strong learner as the meta-combiner. Feeding base outputs, POS/chunk tags, gazetteers and external taggers into RRM integrated heterogeneous sources far better than linear interpolation. This suggests a learned combiner when many weak signals must be merged.
• I5. Small hand-built gazetteers can pay off. A list assembled in a couple of hours gave measurable gains for German. Cheap lexical resources of this kind are worth trying for lower-resource languages such as Ukrainian.