📄 A Conditional Random Field Word Segmenter for Sighan Bakeoff 2005

Summary

The paper describes the Stanford/Colorado entry to the closed track of the SIGHAN Bakeoff 2005 Chinese word segmentation task. The task used four corpora: AS, HK, PK and MSR. Each follows its own definition of a "word" and reflects a different Mandarin variety. Earlier bakeoffs showed that systems tended to excel only on data resembling the variety they were built for. The authors therefore aimed for a segmenter that generalizes across standards. They frame segmentation as character tagging, where each character either starts a word or continues one, and model it with a conditional random field. The features cover:

• character identity n-grams,
• morphological cues,
• character reduplication.

The morphological cues are prefix, suffix and single-character-word tables plus bigram lists. Crucially, these tables are harvested automatically from each training corpus rather than hand-built, so the designers' own linguistic intuitions do not bias the model. The final system scored F = 0.947 (AS), 0.943 (HK), 0.950 (PK) and 0.964 (MSR), which is state-of-the-art level on all four corpora. This shows that one data-driven feature design can handle divergent segmentation conventions.

Key points

• The model is a CRF in the style of Lafferty et al. (2001), with a binary begin/continue labeling as in Peng et al. (2004). It uses Gaussian priors and quasi-Newton optimization.
• On SIGHAN 2003 data it beat Peng et al. (2004) on every corpus:
  • CTB: 0.863 vs 0.849
  • AS: 0.970 vs 0.956
  • HK: 0.947 vs 0.928
  • PK: 0.953 vs 0.941
• Feature counts were very large. For example, MSR had about 3.6M data features and about 12.5M lambda weights, and PK had about 1.7M and 5.4M.
• The unknown-word and reduplication features improved results only on AS and MSR, whose standards produce longer words. On HK they hurt, because the test set used punctuation absent from training. With that punctuation issue fixed, HK would reach 0.952 F, with OOV recall rising to 0.791.
• Error analysis identified several weaknesses:
  • over-splitting of strings that happen to end in frequent suffix characters,
  • confusion between ordinal numbers and number + measure-word sequences,
  • inconsistencies in the gold segmentation of non-hanzi tokens.

Interesting ideas

• I1. Corpus-derived affix tables. Prefix and suffix lists are built by taking the first and last characters of rare words in the training data, instead of relying on hand-curated affix lists. The same approach could bootstrap morphological features for Ukrainian or other languages where annotation conventions differ between corpora.
• I2. Single-character-word table. The authors list characters that only ever appear as standalone words in a given corpus and use them as features. This is a cheap, standard-specific cue that adapts automatically to each annotation scheme.
• I3. Reduplication features. Two simple features check whether the current character repeats the previous one, or whether the previous and next characters match. This shows that small structural pattern detectors can capture productive morphology at almost no cost, and the idea could transfer to other languages with reduplicative word formation.
• I4. Diagnosing a drop through train/test mismatch. When a feature appeared to hurt HK, the authors traced the loss to unseen punctuation rather than to the feature itself, and quantified the corrected score. This is a useful reminder to check preprocessing and normalization gaps before discarding a modeling idea.
• I5. Comparing against a MaxMatch baseline. The authors note that their model groups novel four-character idioms into single words, which a dictionary-based MaxMatch approach would break apart. Comparing a learned model's behavior on novel items against such a baseline is a qualitative evaluation angle worth reusing.