📄 The Second International Chinese Word Segmentation Bakeoff

Summary

This paper reports on the second SIGHAN Chinese word segmentation bakeoff, held in summer 2005. Because Chinese text has no spaces between words, and what counts as a "word" is itself contested, segmentation is a basic but difficult preprocessing step. The organizers wanted to measure how far the field had moved since the first bakeoff in 2003.

Four segmented corpora served as the data: two in Simplified Chinese (Beijing University/PKU, Microsoft Research/MSR) and two in Traditional Chinese (Academia Sinica/AS, City University of Hong Kong/CityU). Each provider followed its own segmentation standard. Systems competed in a closed track, limited to the training data, and an open track, where any external resources were allowed. Scoring was automatic and reported recall, precision, F, OOV rate, and recall on OOV and in-vocabulary words. Results were placed between a maximal-matching baseline and a topline.

In total, 23 groups submitted 130 runs. Segmentation quality improved over the two years, but handling unseen words remains the central weakness. The bakeoff thus works as a shared benchmark across several segmentation standards.

Key points

• Corpus sizes (training words / test words): AS 5.45M / 122K, PKU 1.1M / 104K, CityU 1.46M / 41K, MSR 2.37M / 107K. A fifth corpus from the University of Pennsylvania was not used for technical reasons, but it was published on the SIGHAN site.
• Each corpus used its own standard: PKU's derives from GB 13,715, AS follows a Taiwanese national standard, CityU uses the LIVAC standard, and MSR uses Microsoft's internal one.
• 36 sites from 10 countries registered. Of these, 23 submitted results and a paper, for a total of 130 runs.
• The tight schedule ran as follows:
  • Training data was released on July 11.
  • Test data was released on July 27.
  • Participants had two days to return their segmentations.
• Baseline F scores (maximal matching with the training lexicon) were AS 0.882, CityU 0.833, MSR 0.933 and PKU 0.869. The topline (test lexicon) reached roughly 0.98–0.99. Top reported systems scored about 0.95 F on AS closed and about 0.97 on PKU open.

Interesting ideas

• I1. Bracketing scores with a baseline and a topline. The organizers ran simple left-to-right maximal matching twice: once with the training vocabulary and once with the test vocabulary. This gives cheap reference points for interpreting any score, and the idea carries over to tokenization or lemmatization tasks in other languages.
• I2. Separate closed and open tracks. The closed track forbids even knowledge of punctuation or character sets. This isolates what can be learned from the training data alone, while the open track shows practical upper limits.
• I3. Reporting OOV recall and in-vocabulary recall separately. This exposes whether a system truly generalizes or merely memorizes the lexicon. It is useful for any low-resource or morphologically rich setting, such as Ukrainian.
• I4. Attaching confidence intervals to recall and precision. This lets readers judge whether small differences between systems are meaningful, a habit worth adopting in shared evaluations.
• I5. Distributing data in both Unicode and legacy encodings with careful transcoding. This kept results consistent across sites. The cases where it went wrong, such as Big Five Plus gaps and characters outside the BMP, show why encoding normalization belongs in the evaluation design.