Elamur Research⌕
☰
home / papers / 2015 / Long Short-Term Memory Neural Networks for Chinese Word Segmentation

Long Short-Term Memory Neural Networks for Chinese Word Segmentation

Paper · 2015 · Xinchi Chen, Xipeng Qiu, Chenxi Zhu et al.

Summary

Chinese word segmentation (CWS) is usually treated as character-level tagging with the B/M/E/S scheme. Prior neural taggers, including the models of Zheng et al. (2013) and Pei et al. (2014), consider only a fixed span of surrounding characters, and widening this span usually leads to overfitting instead of improvement. Chen et al. replace the feed-forward hidden layer of such a tagger with an LSTM. Its memory cell can carry useful signals from earlier in the sentence while the explicit input window stays tiny: the current character plus two characters to its right, written (0,2). Tag-to-tag transition scores are kept, following Collobert et al. (2011).

The authors compare several LSTM architectures and examine dropout as a regularizer for the LSTM. They test random, word2vec-pretrained (Chinese Wikipedia) and bigram character embeddings on PKU, MSRA and CTB6. A single-layer model, LSTM-1, beats the earlier neural segmenters in every setting and matches or exceeds feature-heavy state-of-the-art systems. This was an early demonstration that recurrent memory can stand in for wide windows and hand-built features in sequence labeling.

Key points

  • Random embeddings: LSTM-1 reaches F1 of 95.7 (PKU), 96.4 (MSRA) and 94.9 (CTB6). Pei et al. (2014) scores 93.5, 94.4 and 93.9, and it uses a wider (2,2) window. The CTB6 baseline numbers come from the authors' own re-implementations.
  • Pretrained plus bigram embeddings: the best configuration scores 96.5 (PKU), 97.4 (MSRA) and 96.0 (CTB6). This surpasses Zhang et al. (2013) on PKU (96.1), ties it on MSRA, and beats Sun and Xu (2011) on CTB6 (95.7).
  • Closed setting: with bigram embeddings and no external pre-training corpus, the model still scores 96.1, 97.1 and 95.5.
  • Depth: the shallow LSTM-1 performed best. Two stacked layers (LSTM-2) did worse, and the more complex LSTM-3 and LSTM-4 variants did not converge.
  • Cost: training the strongest PKU model took about 3 days on a CPU, using an unoptimized Python implementation.

Interesting ideas

  • I1. Asymmetric window plus recurrence: a left-to-right LSTM supplies the past, while a short right-hand input window supplies the immediate future. This keeps the parameter count low without discarding lookahead, which makes it a cheap pattern for any tokenization or tagging task where wide windows overfit.
  • I2. Bigram vectors from character averages: each bigram embedding is initialised as the mean of its two character embeddings. This gives n-gram representations "for free," with no extra corpus, and could help segmentation or morphology tasks for languages with limited pretraining data.
  • I3. Separate closed and open evaluation: results are reported both with and without external pretrained embeddings. This shows how much of the gain comes from the architecture itself and how much from outside data, which is a fairer basis for comparing systems.
  • I4. Simple before deep: the authors compared shallow and stacked recurrent variants before committing to one, and found the simplest worked best. This is a useful reminder to benchmark minimal recurrent setups before adding depth on small labeled corpora.
  • I5. Long-distance motivating example: the paper uses a sentence whose correct split depends on a distant word ("winter" vs "summer"). Collecting such cases is a concise way to justify, and later probe, a model's use of non-local context.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China65 works · 19,402 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Tokenization & segmentation · 77 works

Cite

Cite as Xinchi Chen, Xipeng Qiu, Chenxi Zhu, Pengfei Liu and Xuanjing Huang. 2015. Long Short-Term Memory Neural Networks for Chinese Word Segmentation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1141

@inproceedings{chen2015long,
  title = {{Long Short-Term Memory Neural Networks for Chinese Word Segmentation}},
  author = {Xinchi Chen, Xipeng Qiu, Chenxi Zhu, Pengfei Liu and Xuanjing Huang},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1141},
  url = {https://doi.org/10.18653/v1/d15-1141}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. The Journal of Machine Learning Research, 3:1137–1155.
  2. A.L. Berger, V.J. Della Pietra, and S.A. Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39–71.
  3. Xinchi Chen, Xipeng Qiu, Chenxi Zhu, and Xuanjing Huang. 2015. Gated recursive neural network for Chinese word segmentation. In Proceedings of Annual Meeting of the Association for Computational Linguistics. find
  4. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of ICML. find
  5. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537.
  6. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. find
  7. T. Emerson. 2005. The second international Chinese word segmentation bakeoff. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, pages 123–133. Jeju Island, Korea.
  8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  9. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning. find
  10. PengFei Liu, Xipeng Qiu, Xinchi Chen, Shiyu Wu, and Xuanjing Huang. 2015. Multi-timescale long short-term memory neural network for modelling sentences and documents. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  11. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH. find
  12. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  13. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111–3119.
  14. Wenzhe Pei, Tao Ge, and Chang Baobao. 2014. Max-margin tensor neural network for chinese word segmentation. In Proceedings of ACL. find
  15. F. Peng, F. Feng, and A. McCallum. 2004. Chinese segmentation and new word detection using conditional random fields. Proceedings of the 20th international conference on Computational Linguistics.
  16. Yanjun Qi, Sujatha G Das, Ronan Collobert, and Jason Weston. 2014. Deep learning for character-based information extraction. In Advances in Information Retrieval, pages 668–674. Springer. find
  17. Nathan D Ratliff, J Andrew Bagnell, and Martin A Zinkevich. 2007. (online) subgradient methods for structured prediction. In Eleventh International Conference on Artificial Intelligence and Statistics (AIStats). find
  18. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45(11):2673–2681. find
  19. Richard Socher, John Bauer, Christopher D Manning, and Andrew Y Ng. 2013. Parsing with compositional vector grammars. In In Proceedings of the ACL conference. Citeseer. find
  20. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  21. Weiwei Sun and Jia Xu. 2011. Enhancing Chinese word segmentation using unlabeled data. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 970–979. find
  22. Ilya Sutskever, James Martens, and Geoffrey E Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 1017–1024. find
  23. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. find
  24. Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky, and Christopher Manning. 2005. A conditional random field word segmenter for sighan bakeoff 2005. In Proceedings of the fourth SIGHAN workshop on Chinese language Processing, volume 171.
  25. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 384–394. Association for Computational Linguistics.
  26. Oriol Vinyals, Suman V Ravuri, and Daniel Povey. 2012. Revisiting recurrent neural networks for robust asr. In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on, pages 4085–4088. IEEE. find
  27. Naiwen Xue, Fei Xia, Fu-Dong Chiou, and Martha Palmer. 2005. The Penn Chinese TreeBank: Phrase structure annotation of a large corpus. Natural language engineering, 11(2):207–238. find
  28. N. Xue. 2003. Chinese word segmentation as character tagging. Computational Linguistics and Chinese Language Processing, 8(1):29–48.
  29. Yaqin Yang and Nianwen Xue. 2012. Chinese comma disambiguation for discourse analysis. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 786–794. Association for Computational Linguistics. find
  30. Yue Zhang and Stephen Clark. 2007. Chinese segmentation with a word-based perceptron algorithm. In ACL. find
  31. Longkai Zhang, Houfeng Wang, Xu Sun, and Mairgup Mansur. 2013. Exploring representations from unlabeled data with co-training for Chinese word segmentation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. find
  32. Xiaoqing Zheng, Hanyang Chen, and Tianyu Xu. 2013. Deep learning for chinese word segmentation and pos tagging. In EMNLP, pages 647–657.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2015-long-short-term-memory-neural/