Elamur Research⌕
☰
home / papers / 2004 / Accessor Variety Criteria for Chinese Word Extraction

Accessor Variety Criteria for Chinese Word Extraction

Paper · 2004 · Haodi Feng, Kang Chen, Xiaotie Deng et al.

Summary

Chinese text has no spaces between words, and many useful terms (names, compounds, derived forms) are absent from dictionaries. Extracting such strings from raw text is therefore an important preprocessing step for retrieval, lexicon building and other NLP applications. Feng et al. use a different signal from the usual frequency or mutual-information statistics. They measure how varied a string's immediate contexts are. Their criterion, accessor variety (AV), counts how many distinct characters appear directly before (predecessors) and directly after (successors) a candidate string in a large corpus. A string that occurs in many different surroundings is likely an independent, meaningful unit. Candidates whose AV falls below a threshold are discarded. Rules then remove strings that are only a word with "adhesive" characters attached. Numeric-type compounds are handled separately. Experiments on TREC 5 and TREC 6 Mandarin documents show that this simple, non-iterative rule performs on par with iterative statistical methods. It does better on long words and finds low-frequency words without a large loss of precision.

Key points

  • The core idea is that context diversity, not repetition count, indicates wordhood. It adapts Harris's (1970) criterion for English morpheme boundaries to Chinese.
  • The data are the TREC 5 and TREC 6 Mandarin newswire documents (LDC 2000T52).
  • The pipeline has two stages: an AV threshold filter, followed by rules that remove adhesive characters. The authors report that even small thresholds give quite good results.
  • As background, the authors cite the Frequency Dictionary of Modern Chinese: among its 9,000 most frequent words, 69.8% are bigrams, 26.7% unigrams and 2.7% trigrams. Longer words are very rare.
  • The method is simpler than PAT-tree, mutual-information and Viterbi-based approaches, and it matches them in quality. It is strongest on long words. Detailed precision figures are in the experimental sections, which were omitted from the provided text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🎓Peking UniversityBeijing · China100 works · 35,618 citations🎓City University of Hong KongHong Kong · China15 works · 2,314 citations🎓Shandong UniversityJinan · China3 works · 692 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Haodi Feng, Kang Chen, Xiaotie Deng and Weimin Zheng. 2004. Accessor Variety Criteria for Chinese Word Extraction. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/089120104773633394

@article{feng2004accessor,
  title = {{Accessor Variety Criteria for Chinese Word Extraction}},
  author = {Haodi Feng, Kang Chen, Xiaotie Deng and Weimin Zheng},
  journal = {Computational Linguistics},
  year = {2004},
  doi = {10.1162/089120104773633394},
  url = {https://doi.org/10.1162/089120104773633394}
}

References

34 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Beeferman, Doug, Adam Berger, and John D. Lafferty. 1999. Statistical models for text segmentation. Machine Learning, 34(1–3):177–210. find
  2. Beijing Language Institute. 1986. Xian dai han yu pin lu ci dian [Word Frequency Dictionary of Modern Chinese]. Beijing Language Institute Press. find
  3. Chang, Jing-Shin and Keh-Yih Su. 1997. An unsupervised iterative method for Chinese new lexicon extraction. International Journal of Computational Linguistics and Chinese Language Processing, 2(2):97–148. find
  4. Chao, Yuen-Ren. 1968. A Grammar of Spoken Chinese. University of California Press, Berkeley and Los Angeles. find
  5. Chen, Ching-Yu, Shu-Fen Tseng, Chu-Ren Huang, and Keh-Jiann Chen. 1993. Some distributional properties of Mandarin Chinese—A study based on the Academia Sinica corpus. Proceedings of Pacific Asia Conference on Formal and Computational Linguistics I, pages 81–95, Taipei. find
  6. Chen, Keh-Jiann and Ming-Hong Bai. 1998. Unknown word detection for Chinese by a corpus-based learning method. International Journal of Computational Linguistics and Chinese Language Processing, 3(1):27–44. find
  7. Chien, Lee-Feng. 1995. Csmart—A high-performance Chinese document retrieval system. In Proceedings of the 1995 International Conference of Computer Processing of Oriental Languages (ICCPOL), pages 176–183, Hawaii. find
  8. Chien, Lee-Feng. 1997. PAT-tree-based keyword extraction for Chinese information retrieval. In Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 50–58, Philadelphia. find
  9. Dai, Yubin, Teck Ee Loh, and Christopher Khoo. 1999. A new statistical formula for Chinese text segmentation incorporating contextual information. In SIGIR ’99: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 82–89, Berkel… find
  10. Deng, Xiaotie, Guojun Li, and Lusheng Wang. 2002. Center and distinguisher for strings with unbounded alphabet. Journal of Combinatorial Optimization, 6(4):383–400. find
  11. Garside, Roger, Geoffrey Leech, and Geoffrey Sampson. 1987. The Computational Analysis of English: A Corpus-Based Approach. London, Longman. find
  12. Ge, Xian-Ping, Wanda Pratt, and Padhraic Smyth. 1999. Discovering Chinese words from unsegmented text. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 271–272, Berkeley, CA. find
  13. Guo, Jin. 1997. Critical tokenization and its properties. Computational Linguistics, 23(4):569–596. find
  14. Harris, Zellig S. 1970. Morpheme boundaries within words. In Papers in Structural and Transformational Linguistics. D. Reidel, Dordrecht, pages 68–77. find
  15. Huang, Chu-Ren, Keh-Jiann Chen, and Ben-Jamin K. Tsou. 1996. Readings in Chinese Natural Language Processing. In Journal of Chinese Linguistics Monograph Series no. 9, edited by Chu-ren Huang et al., pages 1–22. find
  16. Kobayasi, Yosiyuki, Takenobu Tokumaga, and Hozumi Tanaka. 1994. Analysis of Japanese compound nouns using collocational information. In Proceedings of the 15th International Conference on Computational Linguistics (COLING’94), vol. 2, pages 865–869, Kyoto, Japan. find
  17. Kwok, Kui-Lam. 1997. Comparing representations in Chinese information retrieval. In Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 34–41, Philadelphia. find
  18. Linguistic Data Consortium. n.d. TREC Mandarin-Text Retrieval Conference Mandarin Newswire, LDC 2000T52. find
  19. Majoros, William H., G. Mani Subramanian, and Mark Yandell. 2003. Identification of key concepts in biomedical literature using a modified Markov heuristic. Bioinformatics, 19(3):402–407. find
  20. Mo, Ruo-Ping J., Yao-Jung Yang, Keh-Jiann Chen, and Chu-Ren Huang. 1996. Determinative-measure compounds in Mandarin Chinese: Formation rules and parser implementation. In Readings in Chinese Natural Language Processing, Journal of Chinese Linguistics Monograph Series no. 9, edited by Chu-ren Huang… find
  21. Narasimhan, Giri, Changsong Bu, Yuan Gao, Xuning Wang, Ning Xu, and Kalai Mathee. 2002. Mining protein sequences for motifs. Journal of Computational Biology, 9(5):707–720. find
  22. Pachunke, Thomas, Oliver Mertineit, Klaus Wothke, and Rudolf Schmidt. 1992. Broad coverage automatic morphological segmentation of German words. In Proceedings of the 14th International Conference on Computational Linguistics (COLING’92), vol. 4, pages 1218–1222, Nantes, France. find
  23. Paola, Merlo and Suzanne Stevenson. 2001. Automatic verb classification based on statistical distribution of argument structure. Computational Linguistics, 27(3):373–408. find
  24. Ricardo, Baeza-Yates and Ribeiro-Neto Berthier. 1999. Modern Information Retrieval. ACM Press, Addison Wesley Longman. find
  25. Sproat, Richard and Chilin Shih. 1990. A statistical method for finding word boundaries in Chinese text. Computer Processing of Chinese and Oriental Languages, 4:336–351. find
  26. Sproat, Richard, Chilin Shih, William Gale, and Nancy Chang. 1996. A stochastic finite-state word-segmentation algorithm for Chinese. Computational Linguistics, 22(3):377–404.
  27. Teahan, William J., Yingying Wen, Rodger J. McNab, and Ian H. Witten. 2000. A compression-based algorithm for Chinese word segmentation. Computational Linguistics, 26(3):375–393. find
  28. Thijs, Gert, Kathleen Marchal, Magali Lescot, Stephane Rombauts, Bart De Moor, Pierre Rouze, and Yves Moreau. 2002. A Gibbs sampling method to detect overrepresented motifs in the upstream regions of coexpressed genes. Journal of Computational Biology, 9(2):447–464. find
  29. Wang, Yong-Heng, Hai-Ju Su, and Yan Mo. 1990. Automatic processing of Chinese words. Journal of Chinese Information Processing, 4(4):1–11. find
  30. Weeber, Marc, Rein Vos, and R. Harald Baayen. 2000. Extracting the lowest frequency words: Pitfalls and possibilities. Computational Linguistics, 26(3):301–317. find
  31. Wu, Zimin and Gwyneth Tseng. 1993. Chinese text segmentation for text retrieval: Achievements and problems. Journal of the American Society for Information Science, 44(9):532–542. find
  32. Yamamoto, Mikio and Kenneth W. Church. 2001. Using suffix arrays to compute term frequency and document frequency for all substrings in a corpus. Computational Linguistics, 27(1):1–30. find
  33. Yun, Bo-Hyun, Ho Lee, and Hae-Chang Rim. 1995. Analysis of Korean compound nouns using statistical information. In Proceedings of the 1995 International Conference on Computer Processing of Oriental Languages (ICCPOL-95), Hawaii. find
  34. Zhang, Jian, Jianfeng Gao, and Ming Zhou. 2000. Extraction of Chinese compound words—An experimental study on a very large corpus. Proceedings of the Second Chinese Language Processing Workshop, pages 132–139, Hong Kong. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/feng-2004-accessor-variety-criteria-chinese-word/