📄 The First International Chinese Word Segmentation Bakeoff

Summary

Chinese word segmentation research has been hard to compare because each group tested on its own data and several incompatible segmentation standards exist. The authors ran the first international, SIGHAN-sponsored bakeoff in 2003 to fix this. They gave participants four corpora, each following a different standard: Academia Sinica (AS), the Penn Chinese Treebank (CTB), Hong Kong CityU (HK) and Beijing University (PK). Each corpus had a closed track (only its own training data allowed) and an open track (any extra resources, with their contribution disclosed). Twelve sites from six regions took part. Scoring was automatic and strict: only the exact segmentation of the given standard counted, with no credit for plausible alternatives. Systems were compared against a maximum-matching baseline built from the training lexicon and a "cheating" topline built from the test lexicon. Results came as recall, precision, F, and recall on OOV and in-vocabulary words, with confidence values. No single system won across all tracks. The bakeoff matters because it created a shared, reusable benchmark and evaluation protocol for the field.

Key points

• Corpus sizes in training/test words were AS 5.8M/12K, CTB 250K/40K, HK 240K/35K and PK 1.1M/17K. The corpora used different encodings: Big Five, Big Five with HKSCS characters, EUC-CN (GB 2312-80) and GBK (CP936).
• The 12 participating sites came from Mainland China (2), Hong Kong (3), Japan, Singapore, Taiwan and the US (4). Each site chose its own corpora and tracks and could not be tested on data from its own institution.
• The timetable was tight. Training data was released March 15, test data April 22, and segmented output was due April 25. Scoring used a simple Perl script built on diff -y.
• Example results:
  • PK closed was led by S01 with F = 0.951 (8 systems).
  • CTB open was led by S02 with F = 0.912.
  • OOV recall varied far more across systems than in-vocabulary recall did.
• Participants reported annotation inconsistencies during the contest. These were partly corrected in CTB, but the AS report came too late to fix.

Interesting ideas

• I1. Paired baseline and "oracle" topline: The same simple algorithm is run twice, once with a lexicon from the training data and once with a lexicon taken from the test data. This brackets realistic performance and can be reused for any lexicon-dependent task.
• I2. Closed vs. open tracks with disclosure: Separating "own training data only" from "anything goes" makes clear how much of a score comes from the method and how much from extra resources. Asking open-track teams to attribute their gains, for example to a dictionary, makes the comparison more honest.
• I3. Splitting recall into OOV and in-vocabulary parts: A single F score hides how a system handles unseen words. Reporting the two recalls separately shows which systems generalize, which helps any tokenization or tagging benchmark.
• I4. Strict single-standard scoring: The organizers refused to judge which alternative segmentations were acceptable. This keeps the evaluation objective and simple, and lets multiple standards be compared side by side instead of being merged.
• I5. Planned benchmark extensions: The authors recommend short query-like test strings and recording whether each system is commercial. Both would show gaps between lab benchmarks and real deployment, such as search queries or the commercial-versus-research quality lag.