Elamur Research⌕
☰
home / papers / 2004 / Chinese Segmentation and New Word Detection using Conditional Random Fields

Chinese Segmentation and New Word Detection using Conditional Random Fields

Summary

The paper addresses Chinese word segmentation, which is hard because text has no spaces between words and no single agreed definition of a word. The authors frame segmentation as character tagging: each character either starts a word or continues one. They label sequences with a linear-chain CRF, which gives exact Viterbi decoding and accepts arbitrary overlapping features. Domain knowledge enters as features drawn from many external lexicons of words and characters. They also add a probabilistic new word detection step built into segmentation. It uses the CRF's confidence in local decisions to find unfamiliar character strings, adds them to the lexicon and segments again. The system is evaluated on the four datasets of the Sproat and Emerson (2003) bake-off, in both closed and open conditions. It wins two of the eight runs and is competitive on average. The study matters because it gives a principled, knowledge-friendly segmentation model and a consistent baseline across all four benchmarks.

Key points

  • Segmentation uses two tags, START and NONSTART. Training uses BFGS with a Gaussian prior, and the Markov order (first-order up to third-order) is chosen on held-out data.
  • Open features come from a large word list, a character list and 24 extra lexicons (topic and part-of-speech classes such as last names, country names and numbers). A native speaker cleaned these lexicons. Closed features use only lexicon entries that intersect with characters seen in training.
  • On the two datasets where both systems competed, the CRF averaged 94.2% against 93.8% for the sliding-window maximum entropy system (site S12, Xue 2003).
  • CTB gave the worst results for every system because its training and test segmentation is inconsistent. A random 80/20 split of the CTB training data alone yields 97.13% F1. In cross-testing, models trained on other corpora even beat the CTB-trained model on CTB.
  • New word detection in closed tests raised F1 on CTB (0.792→0.849), HK (0.916→0.928) and PK (0.934→0.941). It lowered F1 on AS (0.956→0.946), and the gain grows with the dataset's OOV rate.

Interesting ideas

  • I1. Confidence-driven discovery of new words. The tagger's per-position confidence can mark uncertain spans that sit between confidently segmented regions. These spans can be treated as candidate new words and fed back into the lexicon, which could help any language or domain with many unseen terms.
  • I2. Splitting closed and open features. Separating features learned from training data from features taken from outside resources directly measures how much external lexicons add. The same design would work for any NLP system that leans on dictionaries or gazetteers.
  • I3. Feature-dependent regularization by frequency bins. Features are grouped by how often they occur, and each group gets its own prior variance, so rare and common features are smoothed differently. This is a cheap trick for log-linear models with very skewed feature counts.
  • I4. Using a corpus as a test of itself to diagnose annotation. Splitting a corpus's own training data and getting high scores, while the official test scores stay low, points to a mismatch between training and test guidelines rather than a weak model. Cross-corpus testing supports the same conclusion. Both checks are useful when auditing a treebank or benchmark.
  • I5. Applying a feature only where it is needed. Because new word detection helps only at high OOV rates and hurts at low ones, an adaptive component like this should be switched on according to the expected OOV rate of the target data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

126 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 121 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{peng2004chinese,
  title = {{Chinese Segmentation and New Word Detection using Conditional Random Fields}},
  author = {Peng, Fuchun and Feng, Fangfang and McCallum, Andrew},
  booktitle = {COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics},
  year = {2004},
  doi = {10.3115/1220355.1220436},
  url = {https://doi.org/10.3115/1220355.1220436}
}

References

20 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K.J. Chen and M.H. Bai. 1998. Unknown Word Detection for Chinese by a Corpus-based Learning Method. Computational Linguistics and Chinese Language Processing, 3(1):27–44, Feburary. find
  2. A. Culotta and A. McCallum. 2004. Confidence Estimation for Information Extraction. In Proceedings of Human Language Technology Conference and North American Chapter of the Association for Computational Linguistics(HLT-NAACL). find
  3. S. della Pietra, V. della Pietra, and J. Lafferty. 1995. Inducing Features Of Random Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(4). find
  4. J. Gao, M. Li, and C. Huang. 2003. Improved Source-Channel Models for Chinese Word Segmentation. In Proceedings of the 41th Annual Meeting of Association of Computaional Linguistics (ACL), Japan. find
  5. J. Lafferty, A. McCallum, and F. Pereira. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of the 18th International Conf. on Machine Learning, pages 282–289. find
  6. R. Malouf. 2002. A Comparison of Algorithms for Maximum Entropy Parameter Estimation. In Sixth Workshop on Computational Language Learning (CoNLL). find our card
  7. J. Nie, M. Hannan, and W. Jin. 1995. Unknown Word Detection and Segmentation of Chinese using Statistical and Heuristic Knowledge. Communications of the Chinese and Oriental Languages Information Processing Society, 5:47–57. find
  8. F. Peng and A. McCallum. 2004. Accurate Information Extraction from Research Papers using Conditional Random Fields. In Proceedings of Human Language Technology Conference and North American Chapter of the Association for Computational Linguistics(HLT-NAACL), pages 329–336. find
  9. F. Peng and D. Schuurmans. 2001. Self-Supervised Chinese Word Segmentation. In F. Hoffmann et al., editor, Proceedings of the 4th International Symposium of Intelligent Data Analysis, pages 238–247. Springer-Verlag Berlin Heidelberg. find
  10. L. Rabiner. 1990. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. In Alex Weibel and Kay-Fu Lee, editors, Readings in Speech Recognition, pages 267–296. find
  11. R. Schwartz and Y. Chow. 1990. The N-best Algorithm: An Efficient and Exact Procedure for Finding the N most Likely Sentence Hypotheses. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). find
  12. F. Sha and F. Pereira. 2003. Shallow Parsing with Conditional Random Fields. In Proceedings of Human Language Technology Conference and North American Chapter of the Association for Computational Linguistics(HLT-NAACL). find
  13. R. Sproat and T. Emerson. 2003. First International Chinese Word Segmentation Bakeoff. In Proceedings of the Second SIGHAN Workshop on Chinese Language Processing. find
  14. R. Sproat and C. Shih. 2002. Corpus-based Methods in Chinese Morphology and Phonology. In Proceedings of the 19th International Conference on Computational Linguistics (COLING). find
  15. W. J. Teahan, Y. Wen, R. McNab, and I. H. Witten. 2000. A Compression-based Algorithm for Chinese Word Segmentation. Computational Linguistics, 26(3):375–393. find
  16. A. Wu and Z. Jiang. 2000. Statistically-Enhanced New Word Identification in a Rule-Based Chinese System. In Proceedings of the Second Chinese Language Processing Workshop, pages 46–51, Hong Kong, China. find
  17. Z. Wu. 1999. LDC Chinese Segmenter. http://www.ldc.upenn.edu/ Projects/ Chinese/ segmenter/ mansegment.perl. ↗
  18. A. Wu. 2003. Chinese Word Segmentation in MSR-NLP. In Proceedings of the Second SIGHAN Workshop on Chinese Language Processing, Japan. find
  19. N. Xue. 2003. Chinese Word Segmentation as Character Tagging. International Journal of Computational Linguistics and Chinese Language Processing, 8(1). find our card
  20. H. Zhang, Q. Liu, X. Cheng, H. Zhang, and H. Yu. 2003. Chinese Lexical Analysis Using Hierarchical Hidden Markov Model. In Proceedings of the Second SIGHAN Workshop, pages 63–70, Japan. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2004-chinese-segmentation-new-word-detection/