Elamur Research⌕
☰
home / papers / 2013 / Deep Learning for Chinese Word Segmentation and POS Tagging

Deep Learning for Chinese Word Segmentation and POS Tagging

Summary

Zheng, Chen and Xu ask whether Chinese word segmentation (CWS) and POS tagging can work well without hand-crafted feature templates. Traditional joint systems rely on huge feature sets, which makes models large, slow to train, prone to overfitting and hard to decode. The authors instead treat both tasks as character-level labelling. Their network has a character lookup table, a sliding context window, two linear layers with a sigmoid between them, and learned tag-transition scores. A Viterbi search then finds the best tag sequence. The character embeddings are first trained on large unlabeled text and then reused in the supervised taggers. The authors also propose a perceptron-style training algorithm (PSA) as a cheaper and simpler alternative to sentence-level maximum likelihood. Its cost in accuracy is reported as negligible. The resulting systems come close to state-of-the-art accuracy, with far fewer parameters and much shorter run time than CRF-based segmenters. This shows that compact learned representations can replace much of the manual feature engineering for languages written without spaces.

Key points

  • Both tasks are framed as tagging each character; segmentation uses the B/I/E/S boundary labels shown in the architecture figure.

  • The architecture follows Bengio et al. (2003) and Collobert et al. (2011), but its input units are characters rather than words. It combines a window-based scorer with transition scores A_ij and Viterbi decoding.

  • In the authors' tests, a sigmoid non-linearity worked slightly better than the "hard" tanh used by Collobert. The sigmoid also keeps the cost function differentiable everywhere.

  • Computational cost compared with CRF segmenters:

    SystemParametersRun time
    Tsai et al. (2006)~3.1M1669 s
    Zhao et al. (2006)~3.8M2382 s
    Neural network~0.47M138 s

  • Future directions named by the authors:

    • adding linguistic features such as gazetteers;
    • ensembling, voting or cascading;
    • task-tailored network architectures.

Interesting ideas

  • I1. Pretrain embeddings on raw text, then transfer them into a supervised tagger. This could help any low-resource language or domain where unlabeled text is plentiful but annotated corpora are small.
  • I2. Replace sentence-level likelihood training with a perceptron-style update. The update compares the gold path with the predicted path. It avoids the expensive normalisation over all tag paths and is simpler to implement, which suits quick prototyping of structured taggers.
  • I3. Attach a separate lookup table to each optional feature and concatenate the outputs. Examples are a surname-list flag or unsupervised statistics such as accessor variety or boundary entropy. This injects prior knowledge into a neural model without redesigning it.
  • I4. Report parameter count and run time alongside accuracy. Doing so makes a model's practical deployability visible, which matters for constrained or production settings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Part-of-speech tagging Tokenization & segmentation

 Institutions

🎓Fudan UniversityShanghai · China45 works · 16,344 citations

Also on the byline: General Electric (United States).

Cite

Cite as Xiaoqing Zheng, Hanyang Chen and Tianyu Xu. 2013. Deep Learning for Chinese Word Segmentation and POS Tagging. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zheng2013deep,
  title = {{Deep Learning for Chinese Word Segmentation and POS Tagging}},
  author = {Xiaoqing Zheng, Hanyang Chen and Tianyu Xu},
  booktitle = {Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  doi = {10.18653/v1/d13-1061},
  url = {https://doi.org/10.18653/v1/d13-1061}
}

References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic langugage model. Journal of Machine Learning Research, 3: 1137–1155. find
  2. Léon Bottou. 1991. Stochastic gradient learning in neural networks. In Proceedings of the Neuro-Nı̂mes. find
  3. Michael Collins. 2002. Discriminative training methods for hidden Markov models: Theory and experiments with perceptron algorithms. In Proceedings of the International Conference on Empirical Methods in Natural Language Processing (EMNLP’02). our card
  4. Ronan Collobert. 2011. Deep learning for efficient discriminative parsing. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS’11). find
  5. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12: 2493– 2537. our card
  6. Haodi Feng, Kang Chen, Xiaotie Deng, and Weimin Zheng. 2004. Accessor variety criteria for Chinese word extraction. Computational Linguistics, 30(1): 75–93. find
  7. Yuanyong Feng, Sun Le, and Yuanhua Lv. 2006. Chinese word segmentation and name entity recognition based on conditional random fields models. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN’06). find
  8. Wenbin Jiang, Liang Huang, Qun Liu, Yajuan Lu. 2008. A cascaded linear model for joint Chinese word segmentation and part-of-speech tagging. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL’08). find
  9. Zhihui Jin, and Kumiko Tanaka-Ishii. 2006. Unsupervised segmentation of Chinese text by use of branching entropy. In Proceedings of the International Conference on Computational Linguistics and the Annual Meeting of the Association for Computational Linguistics (COLING/ACL’06). find
  10. Canasai Kruengkrai, Kiyotaka Uchimoto, Jun’ichi Kazama, Yiou Wang, Kentaro Torisawa, and Hitoshi Isahara. 2009. An error-driven word-character hybrid model for joint Chinese word segmentation and pos tagging. In Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics… find
  11. John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the International Conference on Machine learning (ICML’01). find
  12. Gina A. Levow. 2006. The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. In Proceedings of the SIGHAN Workshop on Chinese Language Processing (SIGHAN’06). our card
  13. Hwee T. Ng and Jin K. Lou. 2004. Chinese part-of-speech tagging: one-at-atime or all-at-once? word-based or character-based? In Proceedings of the International Conference on Empirical Methods in Natural Language Processing (EMNLP’04). find
  14. Fuchun Peng, Fangfang Feng, and Andrew McCallum. 2004. Chinese segmentation and new word detection using conditional random fields. In Proceedings of the 20th International Conference on Computational Linguistics (COLING’04). our card
  15. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1986. Learning internal representations by backpropagating errors. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, 1: 318–362. MIT Press. find
  16. Richard Socher, Cliff C-Y. Lin, Andrew Y. Ng, and Christopher D. Manning 2011. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In Proceedings of the International Conference on Machine learning (ICML’11). find
  17. Weiwei Sun. 2011. A stacked sub-word model for joint Chinese word segmentation and part-of-speech tagging. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL’11). find
  18. Weiwei Sun and Jia Xu. 2011. Enhancing Chinese word segmentation using unlabeled data. In Proceedings of the International Conference on Empirical Methods in Natural Language Processing (EMNLP’11). find
  19. Richard T.-H. Tsai, Hsieh C. Hung, Chenglung Sung, Hongjie Dai, and Wenlian Hsu. 2006. On closed task of Chinese word segmentation: An improved CRF model coupled with character clustering and automatically generated template matching. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language P… find
  20. Xinhao Wang, Xiaojun Lin, Dianhai Yu, Hao Tian, and Xihong Wu. 2006. Chinese word segmentation with maximum entropy and n-gram language model. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN’06). find
  21. Nianwen Xue. 2003. Chinese word segmentation as character tagging. Computational Linguistics and Chinese Language Processing, 8(1): 29–48. our card
  22. Min Zhang, GuoDong Zhou, LingPeng Yang, and DongHong Ji. 2006. Chinese word segmentation and named entity recognition based on a context-dependent mutual information independence Model. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN’06). find
  23. Yue Zhang and Stephen Clark. 2007. Chinese segmentation with a word-base perceptron algorithm. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL’07). find
  24. Yue Zhang and Stephen Clark. 2008. Joint word segmentation and pos tagging using a single perceptron. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL’08). find
  25. Hai Zhao, Chang N. Huang, and Mu Li. 2006. An improved Chinese word segmentation system with conditional random field. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN’06). find
  26. Hai Zhao and Chunyu Kit. 2011. Integrating unsupervised and supervised word segmentation: The role of goodness measures. Information Sciences, 181(1): 163–183. find
  27. Muhua Zhu, Yilin Wang, Zhenxing Wang, Huizhen Wang, and Jingbo Zhu. 2006. Designing special post-processing rules for SVM-based Chinese word segmentation. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN’06). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2013-deep-learning-chinese-word-segmentation/