Optimizing Chinese Word Segmentation for Machine Translation Performance
Summary
The paper asks how Chinese word segmentation should be designed when its output feeds a phrase-based Chinese-to-English MT system rather than a segmentation benchmark. The authors used Moses, GIZA++ alignments and MERT tuning on NIST MT03, with testing on MT03 and MT05. They compared character-level tokenization, a greedy dictionary matcher (MaxMatch) and CRF segmenters trained on the SIGHAN Bakeoff 2006 UPUC data.
Their central finding is that a higher segmentation F measure does not reliably produce better translations. Two other properties matter more:
- Consistency: the same string should be split the same way across contexts.
- Granularity: the average length of the produced "words" should suit the MT system.
Based on this, they made three changes to the CRF segmenter:
- added features drawn from large external lexicons,
- jointly tagged proper nouns,
- added a bias parameter that controls how often word boundaries are placed.
Tuning granularity for MT gave 0.73 BLEU, and the consistency features gave further gains. MT performed best when the segmenter produced units a little shorter on average than those defined by the CTB standard. The work matters because it shows that upstream preprocessing should be tuned for the downstream task, not for an intrinsic metric.
Key points
- Characters vs. a simple segmenter: CharBased scored F 0.334 and 29.36 BLEU on MT05. MaxMatch scored F 0.828 and 30.73 BLEU, a gain of about 1.37 BLEU. MaxMatch still won on the MT05 subset where every token is in-vocabulary (32.09 vs. 30.28). This means the phrase table cannot fully replace segmentation.
- Lexicon features (CRF-Lex): these drew on external lexicons with 423,224 entries, including Wikipedia/UN named entities, ADSO, HowNet and others. They shrank the MT training lexicon by 29.5%, cut the MT test OOV rate by 34.1%, and added 0.38 BLEU on MT05.
- Proper noun labels (CRF-Lex-NR): extending the CRF labels from 2 to 4 ({begin, continue} × {NR, not NR}) raised OOV recall from 0.729 to 0.753. It also improved MT05 BLEU from 30.95 to 31.27.
- Granularity bias: pushing the CRF toward more boundaries (λ0 > 0) lowered segmentation precision, recall and F. Despite this, it raised BLEU, with a total gain of 0.73 BLEU on MT05.
- MT data: about 1.14M sentence pairs were subsampled from GALE Year 2 using character n-grams, so the sample does not favor any segmenter. The language model was a 5-gram trained mainly on a 25M-sentence subsample of English Gigaword.
Interesting ideas
- I1. Evaluate preprocessing by its downstream effect. Judge a tokenizer or segmenter by the end-task score, not only by agreement with a gold standard. This applies to any pipeline component, such as morphological splitting for Ukrainian MT or tokenization for retrieval.
- I2. Add a single bias knob on the boundary label. One weight in the sequence model can shift average token length, so granularity can be tuned directly against the end metric. This is cheaper than redesigning annotation standards.
- I3. Use external lexicon features to stabilize decisions. Encode, for each character, the maximum length of dictionary words that start, contain or end with it. This reduces inconsistent splits and spurious OOVs, and it is a practical way to inject gazetteers into a CRF or neural tagger.
- I4. Combine entity labels with boundary labels. Folding a proper-noun tag into the boundary labels keeps names intact, and those names are often unsplittable OOVs. It also gives entity information to later modules such as named-entity transliteration.
- I5. Sample MT training data with a segmentation-neutral method. Subsampling parallel data with character n-grams avoids biasing it toward any one segmenter. This is a useful precaution whenever compared systems tokenize differently.
Cited by
- Findings of the WMT 2019 Biomedical Translation Shared Task: Evaluation for MEDLINE Abstracts and Biomedical Terminologies
- Learning Word Vectors for 157 Languages
- The Johns Hopkins University Bible Corpus: 1600+ Tongues for Typological Exploration
Cite
@inproceedings{chang2008optimizing,
title = {{Optimizing Chinese Word Segmentation for Machine Translation Performance}},
author = {Chang, Pi-Chuan and Galley, Michel and Manning, Christopher D.},
booktitle = {Proceedings of the Third Workshop on Statistical Machine Translation},
year = {2008},
doi = {10.3115/1626394.1626430},
url = {https://doi.org/10.3115/1626394.1626430}
}References
- Jianfeng Gao, Mu Li, Andi Wu, and Chang-Ning Huang. 2005. Chinese word segmentation and named entity recognition: A pragmatic approach. Computational Linguistics. find
- Nizar Habash and Fatiha Sadat. 2006. Arabic preprocessing schemes for statistical machine translation. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 49–52, New York City, USA, June. Association for Computational Linguistics. find
- Vera Hollink, Jaap Kamps, Christof Monz, and Maarten de Rijke. 2004. Monolingual document retrieval for European languages. Information Retrieval, 7(1). find
- Philipp Koehn and Kevin Knight. 2003. Empirical methods for compound splitting. In EACL ’03: Proceedings of the tenth conference on European chapter of the Association for Computational Linguistics, pages 187–193. Association for Computational Linguistics. find
- Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proc. of NAACL-HLT. find
- John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proc. 18th International Conf. on Machine Learning. find
- Gina-Anne Levow. 2006. The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. In Proc. of the Fifth SIGHAN Workshop on Chinese Language Processing, July. find
- Einat Minkov, Richard Wang, Anthony Tomasic, and William Cohen. 2006. NER systems that suit user’s preferences: Adjusting the recall-precision trade-off for entity extraction. In Proc. of NAACL-HLT, Companion Volume: Short Papers, New York City, USA, June. find
- Hwee Tou Ng and Jin Kiat Low. 2004. Chinese part-of-speech tagging: One-at-a-time or all-at-once? Word-based or character-based? In Proc. of EMNLP. find
- Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1). find
- Franz Josef Och. 2003. Minimum error rate training for statistical machine translation. In ACL. find
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. BLEU: a method for automatic evaluation of machine translation. In ACL. find our card
- Fuchun Peng, Xiangji Huang, Dale Schuurmans, and Nick Cercone. 2002. Investigating the relationship between word segmentation performance and retrieval performance in Chinese IR. In Proc. of the 19th International Conference on Computational Linguistics. find
- Fuchun Peng, Fangfang Feng, and Andrew McCallum. 2004. Chinese segmentation and new word detection using conditional random fields. In Proc. of COLING. find
- Yanxin Shi and Mengqiu Wang. 2007. A dual-layer CRFs based joint decoding method for cascaded segmentation and labeling tasks. In IJCAI. find
- Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky, and Christopher Manning. 2005. A conditional random field word segmenter for Sighan bakeoff 2005. In Proc. of the Fourth SIGHAN Workshop on Chinese Language Processing. find
- Jia Xu, Richard Zens, and Hermann Ney. 2004. Do we need Chinese word segmentation for statistical machine translation. In Proc. of the Third SIGHAN Workshop on Chinese Language Learning. find
- Nianwen Xue, Fei Xia, Fu dong Chiou, and Martha Palmer. 2005. Building a large annotated Chinese corpus: the Penn Chinese treebank. Journal of Natural Language Engineering, 11(2). find
This page in Markdown → https://md.research.elamur.ai/free/papers/chang-2008-optimizing-chinese-word-segmentation-machine/