Elamur Research⌕
☰
home / papers / 2003 / Combining Distributional and Morphological Information for Part of Speech Induction

Combining Distributional and Morphological Information for Part of Speech Induction

Summary

The paper addresses unsupervised induction of word classes that approximate parts of speech, with particular attention to rare and unseen words. Clark starts from the Ney-Essen style exchange clustering algorithm, which groups words to maximise class-bigram likelihood. He extends it in a Bayesian way: each cluster gets its own small letter-level HMM, and the prior over partitions rewards clusters whose words look alike orthographically. A second extension adds per-class prior weights, so rare words lean toward open-class clusters. Clusters are scored by conditional entropy of gold tags given induced tags. Experiments use the WSJ (about 500,000 words) and the seven-language MULTEXT-East edition of Orwell's 1984. Adding spelling information substantially improves clustering of infrequent words, frequency gives a further gain, and the effect holds across typologically diverse languages. Fully probabilistic HMM taggers that allow ambiguity performed poorly. This matters for morphologically rich languages, where many word types are rare.

Key points

  • Data. English data: WSJ sections 0–19, about 500k tokens. Cross-lingual data: MULTEXT-East 1984 in English, Romanian, Czech, Slovene, Bulgarian (Cyrillic), Estonian and Hungarian.
  • Token-type ratios run from 12.1 in English down to 4.84 in Hungarian, so rare words are far more common in the latter. Hungarian tags are very fine-grained, with around 400 tags.
  • Baselines on WSJ use H(G|T):
    • "Frequent word" baselines score 2.00, 1.75, 1.46 and 1.25 for 16, 32, 64 and 128 classes.
    • Giving every word its own class leaves only 0.12, which reflects lexical ambiguity.
    • This low residual is the author's argument for using hard (deterministic) clustering.
  • Plain HMM. A fully connected, randomly initialised HMM trained with EM does not improve H(G|T) as states are added (2.18 → 1.72). Instead, lexical ambiguity H(T|W) grows.
  • Two-level HMM. A word-state HMM whose emissions are letter HMMs keeps H(T|W) low (about 0.1–0.17), but it still scores worse than the frequency baselines.

Interesting ideas

  • I1. Morphology as a prior. Spelling is added as a prior over the clustering rather than as an extra feature. This keeps the distributional objective intact while still steering ambiguous decisions, and it would be easy to graft onto other clustering objectives.
  • I2. Deliberately weak string models. Small letter HMMs are chosen over character n-grams so they can capture whole-word properties without memorising trivial patterns, such as words that begin with "st". Limiting model capacity in this way is a reusable lesson for any auxiliary model.
  • I3. Frequency as evidence. The fact that a word is rare is itself treated as a signal that it belongs to an open class, encoded through class prior weights. This is a cheap trick for handling OOV words in taggers or embedding models, including for Ukrainian.
  • I4. Baselines expose the metric. Giving frequent words singleton classes scores well on entropy measures, which shows how Zipfian distributions can inflate results. The author also compares gold tagsets against each other (0.22–1.3 between annotation schemes) to calibrate the scores. Both are good evaluation habits for unsupervised tagging.
  • I5. Diagnosing EM failure. When unconstrained emission distributions are too expressive, HMM states split by position rather than by category. Constraining the emissions is the remedy, which is worth remembering when latent states drift away from the intended semantics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Part-of-speech tagging

Cited by

88 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 83 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{clark2003combining,
  title = {{Combining Distributional and Morphological Information for Part of Speech Induction}},
  author = {Clark, Alexander},
  booktitle = {10th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2003},
  doi = {10.3115/1067807.1067817},
  url = {https://doi.org/10.3115/1067807.1067817}
}

References

17 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Murakami, H. Yamatomo, and S. Sagayama. 1993. The possibility for acquisition of statistical network grammar using ergodic HMM. In Proceedings of Eurospeech 93, pages 1327-1330. find
  2. E. Atwell, G. Demetriou, J. Hughes, A. Schiffrin, C. Souter, and S. Wilcock. 2000. A comparative evaluation of modern English corpus grammatical annotation schemes. ICAME Journal, 24:7-23. find
  3. Peter F. Brown, Vincent J. Della Pietra, Peter V. de Souza, Jenifer C. Lai, and Robert Mercer. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18:467-479. find
  4. Martin Redington, Nick Chater, and Steven Finch. 1998. Distributional information: A powerful cue for acquiring syntactic categories. Cognitive Science, 22(4):425-469. find
  5. Alexander Clark. 2000. Inducing syntactic categories by context distribution clustering. In Proc. of CoNLL-2000 and LLL-2000, pages 91-94, Lisbon, Portugal. find
  6. Alexander Clark. 2001. Partially supervised learning of morphology with stochastic transducers. In Proc. of Natural Language Processing Pacific Rim Symposium, NLPRS 2001, pages 341-348, Tokyo, Japan, November. find
  7. Toma'Z Erjavec and Nancy Ide. 1998. The MULTEXT-East corpus. In First International Conference on Language Resources and Evaluation, LREC'98, pages 971-974, Granada. ELRA. find
  8. S. Finch and N. Chater. 1992a. Bootstrapping syntactic categories. In Proceedings of the 14th Annual Meeting of the Cognitive Science Society, pages 820-825. find
  9. S. Finch and N. Chater. 1992b. Bootstrapping syntactic categories using statistical methods. In W. Daelemans and D. Powers, editors, Background and Experiments in Machine Learning of Natural Language, pages 229-235. Tilburg University: Institute for Language Technology and Al. find
  10. Shai Fine, Yoram Singer, and Naftali Tishby. 1998. The hierarchical Hidden Markov Model: Analysis and applications. Machine Learning, 32:41. find
  11. Sydney M. Lamb. 1961. On the mechanisation of syntactic analysis. In 1961 Conference on Machine Translation of Languages and Applied Language Analysis, volume 2, pages 674-685. HMSO, London. find
  12. Sven Martin, JOrg Liermann, and Hermann Ney. 1998. Algorithms for bigram and trigram word clustering. Speech Communication, 24:19-37. find
  13. Hermann Ney, Ute Essen, and Reinhard Kneser. 1994. On structuring probabilistic dependencies in stochastic language modelling. Computer Speech and Language, 8:1-38. find
  14. Ronald Rosenfeld. 2000. Two decades of statistical language modeling: Where do we go from here? Proceedings of the IEEE, 88(8). find
  15. Hinrich Schtitze. 1997. Ambiguity Resolution in Language Learning. CSLI Publications. find
  16. K. Weber, S. Bengio, and H. Bourlard. 2001. Speech recognition using advanced hmm2 features. IDIAP-RR 24, IDIAP, Martigny, Switzerland. find
  17. Published: ASRU 2001, Madonna di Campiglio, Italy, December 2001. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/clark-2003-combining-distributional-morphological-information-part/