📄 HHMM-based Chinese Lexical Analyzer ICTCLAS

Note before use: The PDF extraction is corrupted. The text is mostly stray symbols from a font-encoding failure, and none of the paper's sentences, tables or numbers can be read. The draft below is therefore not based on the supplied text. It reflects the generally known content of this paper. Every item marked [verify] must be checked against a clean copy (e.g., the ACL Anthology version) before it goes into the library.

Summary

Chinese text has no spaces between words. Building a lexical analyzer therefore means solving several linked tasks: word segmentation, part-of-speech (POS) tagging, recognition of unknown words such as person, place and organization names, and ambiguity resolution. Pipelines that handle these separately tend to pass errors from one stage to the next. Zhang et al. present ICTCLAS, a lexical analyzer built on a Hierarchical Hidden Markov Model (HHMM). Each layer of the hierarchy is an HMM handling one level of analysis:

• splitting text into atoms,
• simple and recursive unknown-word recognition through role tagging,
• class-based segmentation,
• POS tagging.

To keep the search tractable while still allowing ambiguity to be resolved, the system first generates several candidate segmentations with an N-shortest-paths method rather than a single best guess. ICTCLAS took part in the First SIGHAN Chinese Word Segmentation Bakeoff and was among the top systems on some tracks [verify ranks and tracks]. It became one of the most widely used Chinese segmenters, which makes it a key reference for hierarchical, statistically grounded lexical analysis.

Key points

• The system is ICTCLAS, from the Institute of Computing Technology, Chinese Academy of Sciences, and the underlying model is a Hierarchical Hidden Markov Model (HHMM).
• Unknown words (person names, transliterated names, locations, organizations) are found by tagging characters or words with roles using an HMM. Recognition can be applied recursively to handle nested names [verify details].
• N-shortest-paths rough segmentation keeps several candidate segmentations for later layers to rescore.
• A class-based HMM performs the final segmentation, followed by HMM-based POS tagging.
• It was evaluated in the First SIGHAN Bakeoff (2003) on corpora such as PK and CTB. Exact precision, recall, F-scores and speed figures are [verify; not recoverable from the supplied text].

Interesting ideas

• I1. Layered HMMs instead of one large model: Assigning each linguistic level its own HMM keeps every component simple and trainable, while the levels still share probabilistic scores. This design could carry over to morphologically rich languages like Ukrainian, where tokenization, morphology and NER interact.
• I2. Keep N candidates, not one: Passing several rough segmentations forward delays hard decisions until more context is available. It is a cheap way to reduce error propagation in any NLP pipeline.
• I3. Role tagging for unknown words: Recasting name detection as tagging tokens with roles (e.g., surname, given name, context) lets one sequence-labeling machinery cover many entity types. The same trick could help with transliterated foreign names in Ukrainian trade or geography data.
• I4. Recursive recognition: Running recognition again on its own output allows nested entities to be found, such as an organization name that contains a place name. Nested entities are also common in geographic and commercial texts.