Elamur Research
☰
home / papers / 2003 / Chinese Word Segmentation as Character Tagging

Chinese Word Segmentation as Character Tagging

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/O03-4002.pdf

Summary

Chinese text has no spaces between words, and most hanzi can appear in several positions inside a word. This makes segmentation ambiguous and leaves dictionary-driven methods weak on unseen words. Xue reframes segmentation as a character-level labelling task. A maximum entropy tagger assigns each hanzi one of four position-of-character tags: LL (word-initial), MM (word-internal), RR (word-final) and LR (single-character word). The tagger is trained on manually segmented newswire data following the Penn Chinese Treebank standard, and its tags are then turned back into word boundaries. Features cover the current character, neighbouring characters, character pairs and previously assigned tags.

With a training set of about 237K words, the system reaches roughly 95% precision and recall. This is competitive with earlier supervised segmenters, and it copes with unknown words far better than a maximum matching baseline. The approach works because the hanzi inventory is essentially closed even though the vocabulary is open. Framing segmentation as sequence tagging became a highly influential approach in Chinese NLP.

Key points

  • Main result: precision 95.01%, recall 94.94%, F-score 94.98%, and tagging accuracy 96.05%, with a 237K-word training set.
  • Personal names: 119 of 137 names in the test data were segmented correctly (86.86% recall). Long names and foreign transliterations caused most of the errors.
  • Feature ablation: every feature group helps. Removing the character-pair features (5d) hurts most, cutting F-score to about 91.05%.
  • Tag set comparison:
    • The 4-tag set gives the best F-score (94.98).
    • A 3-tag scheme in the style of Ramshaw & Marcus is close behind (94.96).
    • A 2-tag scheme has higher tagging accuracy (97.51%) but a lower F-score (94.38).
  • Planned future work: testing on non-newswire data and on segmentation standards other than the Penn Chinese Treebank, tagging morphemes instead of hanzi, and better conversion from tags to words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Topics: Part-of-speech tagging

Cited by

146 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@article{xue2003chinese,
  title = {{Chinese Word Segmentation as Character Tagging}},
  author = {Xue, Nianwen},
  journal = {International Journal of Computational Linguistics \& Chinese Language Processing, Volume 8, Number 1, February 2003: Special Issue on Word Formation and Chinese Language Processing},
  year = {2003},
  url = {https://aclanthology.org/O03-4002/}
}

References

29 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brill, Eric, 1993. A Corpus-based Approach to Language Learning. Ph.D. thesis, University of Pennsylvania. find
  2. Segmentation accuracy p(%) r(%) f(%) 94.37 94.40 94.38 95.09 94.83 94.96 95.01 94.94 94.98 This research CKIP, 1995. An Introduction to the Academia Sinica Balanced Corpus (in Chinese). Technical Report 95-02, Taipei: Academia Sinica. find
  3. Dai, Xiang-Ling, 1992. Chinese Morphology and its Interface with the Syntax. Ph.D. thesis, Ohio State University. find
  4. Fan, C. K. and W. H. Tsai, 1988. ‘‘Automatic word identification in Chinese sentences by the relaxation technique”. Computer Processing of Chinese and Oriental Languages, 4(1):33–56. find
  5. Gan, Kok-Wee, 1995. Integrating Word Boundary Disambiguation with Sentence Understanding . Ph.D. thesis, National University of Singapore. find
  6. Gan, Kok-Wee, Martha Palmer, and Kim-Teng Lua, 1996. ‘‘A statistically emergent approach for language processing: Application to modeling context effects in ambiguous Chinese word boundary perception”. Computational Linguistics, 22(4):531–53. find
  7. Ge, Xianping, Wanda Pratt, and Padhraic Smyth, 1999. ‘‘Dicovering Chinese words from unsegmented text”. In SIGIR’99. find
  8. Gu, Ping and Yuhang Mao, 1994. ‘‘Hanyu zidong fenci de jinlin pipei suanfa jiqi zai qhfy hanying jiqi fanyi xitong zhong de shixian”. [the adjacent matching algorithm of Chinese automatic word segmentation and its implementation in the qhfy Chinese-english system. In International Conference on Chin… find
  9. Guo, Jin, 1997. ‘‘Critical tokenization and its properties”. Computational Linguistics, 23(4):569–596. find
  10. Hockenmaier, Julia and Chris Brew, 1998. ‘‘Error-driven segmentation of Chinese”. Communications of COLIPS, 1(1):69–84. find
  11. Jin, Wangying and Lei Chen, 1998. ‘‘Identifying unknown words in Chinese corpora”. In The First Workshop on Chinese Language Processing. University of Pennsylvania, Philadelphia. find
  12. Liang, Nanyuan, 1993. ‘‘shumian hanyu zidong fenci xitong cdws”. Journal of Chinese Information Processing, 1(1):44–52. find
  13. Packard, Jerome, 2000. The Morphology of Chinese: A Linguistics and Cognitive Approach. Cambridge: Cambridge University Press. find
  14. Palmer, David, 1997. ‘‘A trainable rule-based algorithm to word segmentation”. In Proceedings of the 35th Annual Meeting of the Association of Computational Linguistics. Madrid, Spain. find
  15. Peng, Fuchun and Dale Schuurmans, 2001. “Self-supervised Chinese word segmentation”. In F. Hoffman et al (ed.), Advances in Intelligent Data Analysis, Proceedings of the Fourth International Conference (IDA-01). Heidelberg. Springer-Verlag. find
  16. Ramshaw, Lance and Mitchell P. Marcus, 1995. ‘‘Text chunking using transformation-based learning”. In Proceedings of the Third ACL Workshop on Very Large Corpora . find
  17. Nianwen Xue Chinese Word Segmentation as Character Tagging Ratnaparkhi, Adwait, 1996. ‘‘A maximum entropy part-of-speech tagger”. In Proceedings of the Empirical Methods in Natural Language Processing Conference . University of Pennsylvania. find
  18. Ratnaparkhi, Adwait, 1998. Maximum Entropy Models for Natural Language Ambiguity Resolution. Ph.D. thesis, University of Pennsylvania. find
  19. Sproat, R. and C. L. Shih, 1990. ‘‘A statistical method for finding word boundaries in Chinese text”. Computer Processing of Chinese and Oriental Languages, 4(4):336–351. find
  20. Sproat, R., Chilin Shih, William Gale, and Nancy Chang, 1996. ‘‘A stochastic finite-state word-segmentation algorithm for Chinese”. Computational Linguistics, 22(3):377–404. find
  21. Sun, Maosong, Dayang Shen, and Benjamin K. Tsou, 1998. ‘‘Chinese word segmentation without using lexicon and hand-crafted training data”. In Proceedings of COLING-ACL’98. find
  22. Wu, Andi, this issue. ‘‘Customizable segmentation of morphologically derived words in Chinese”. Computational Linguistics and Chinese Language Processing . find
  23. Wu, Andi and Zixin Jiang, 1998. ‘‘Word segmentation in sentence analysis”. In Proceedings of the 1998 International Conference on Chinese Information Processing. Beijing, China. find
  24. Wu, Dekai and Pascale Fung, 1994. ‘‘Improving Chinese tokenization with linguistic filters on statistical lexical acquisition”. In Proceedings of the Fourth Conference on Applied Natural Language Processing. find
  25. Xia, Fei, 2000. The Segmentation Guidelines for Chinese Treebank Project. Technical Report IRCS 00-06, University of Pennsylvania. find
  26. Xia, Fei, Martha Palmer, Nianwen Xue, Mary Ellen Okurowski, John Kovarik, Fu-Dong Chiou, Shizhe Huang, Tony Kroch, and Mitch Marcus, 2000. “Developing Guidelines and Ensuring Consistency for Chinese Text Annotation”. In Proc. of the 2nd International Conference on Language Resources and Evaluation (… find
  27. Xue, Nianwen, 2001. Defining and Automatically Identifying Words in Chinese. Ph.D. thesis, University of Delaware. find
  28. Xue, Nianwen, Fu-Dong Chiou, and Martha Palmer, 2002. ‘‘Building a large annotated Chinese corpus”. In The Proceedings of the 19th International Conference on Computational Linguistics. Taipei, Taiwan. find
  29. Yu, Shiwen, Xuefeng Zhu, Hui Wang, and Yunyun Zhang, 1998. The Grammatical Knowledge-base of Contemporary Chinese — A Complete Specification (in Chinese). Tsinghua University Press. find