Elamur Research⌕
☰
home / papers / 2010 / Two Decades of Unsupervised POS Induction: How Far Have We Come?

Two Decades of Unsupervised POS Induction: How Far Have We Come?

Paper · 2010 · Christos Christodoulopoulos, Sharon Goldwater and Mark Steedman

Summary

Unsupervised part-of-speech induction has produced many competing methods. They are hard to rank because papers use different metrics and usually compare against only one or two rivals. The authors, motivated by plans to process more than 50 low-resource languages without annotators, run seven induction systems out of the box with default settings. The systems are brown, clark, cw, bhmm, vbhmm, pr and feat, and together they span roughly twenty years. Each is evaluated with several metrics on WSJ and on the parallel, eight-language Multext-East corpus. The authors argue for V-Measure because it is less sensitive to the number of clusters than mapping-based accuracies. The main finding is that the old, simple, fast class-based models of Brown et al. (1992) and Clark (2003) match or beat newer Bayesian and feature-based HMMs, and clark is strongest on non-English data. The paper also proposes judging systems by how useful their automatically extracted prototypes are to the Haghighi and Klein prototype-driven learner. Seeding that learner with brown's prototypes gives state-of-the-art WSJ results. This shows that methodological novelty does not guarantee better performance.

Key points

  • Seven systems were compared: brown, clark, cw (Chinese Whispers, which chooses |C| itself), bhmm (Gibbs), vbhmm (variational Bayes), pr (posterior regularization) and feat (feature-based logistic HMM). Each ran with default parameters, and only |C| was varied.
  • The metrics were many-to-1, crossval, 1-to-1, VI and V-Measure. The authors recommend V-Measure as more stable when the numbers of found and gold clusters differ. Many-to-1 tends to rise with |C|, and 1-to-1 tends to fall.
  • With 45 clusters on WSJ, brown+proto reached 76.1 many-to-1 and 68.8 VM, a gain of 5.8 VM over plain brown. The authors report this as the best unsupervised WSJ result they knew of. Haghighi and Klein's learner with gold prototypes scores 80.2 many-to-1 and 75.2 VM.
  • On Multext-East, brown+proto also beat brown, though by smaller margins (for example +0.1 VM on Hungarian and +5.0 on Czech). In contrast, clark+proto scored lower than clark on every non-English language.
  • The Infinite HMM could not be run in time. Its reported score of about 59% VM on WSJ, with 194 clusters, is below the best systems.

Interesting ideas

  • I1. Test the defaults on unseen languages. Benchmarking tools with default settings, on data they were not tuned on, measures the portability that actually matters for low-resource languages. This is directly relevant to Ukrainian or other under-resourced projects.
  • I2. Pick metrics that survive a changing cluster count. Accuracy-style scores reward or punish the number of clusters itself. An entropy-based score such as V-Measure, ideally reported alongside them, gives fairer comparisons for any clustering task where cluster counts vary between systems.
  • I3. Score a model by its downstream usefulness. Instead of grading an unsupervised clustering only directly, extract a few representative items from each cluster and check whether they help a weakly supervised learner. This "usefulness as seeds" view could also suit topic models or embedding clusters.
  • I4. Choose prototypes with a simple heuristic. Start from the frequent words in each cluster. Rank them by the average of their similarity to cluster-mates and their dissimilarity to other clusters, using SVD context vectors and cosine. Keep up to ten that pass a relative cutoff. This is a cheap, language-agnostic way to turn noisy clusters into seed lists, and it naturally allows small closed classes to have fewer prototypes.
  • I5. Do not assume newer is better. Re-running simple baselines alongside sophisticated methods can reveal that older, faster methods are still competitive. This is a useful sanity check before adopting complex models in any NLP pipeline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Christos Christodoulopoulos, Sharon Goldwater and Mark Steedman. 2010. Two Decades of Unsupervised POS Induction: How Far Have We Come?. Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{christodoulopoulos2010two,
  title = {{Two Decades of Unsupervised POS Induction: How Far Have We Come?}},
  author = {Christos Christodoulopoulos, Sharon Goldwater and Mark Steedman},
  booktitle = {Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://aclanthology.org/D10-1056/}
}

References

22 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Taylor Berg-Kirkpatrick, Alexandre B. Côté, John DeNero, and Dan Klein. 2010. Painless unsupervised learning with features. In Proceedings of NAACL 2010, pages 582–590, Los Angeles, California, June.
  2. Chris Biemann. 2006. Unsupervised part-of-speech tagging employing efficient graph clustering. In Proceedings of COLING ACL 2006, pages 7–12, Morristown, NJ, USA. find
  3. Peter F. Brown, Vincent J. Della Pietra, Peter V. Desouza, Jennifer C. Lai, and Robert L. Mercer. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467–479. find
  4. Alexander Clark. 2003. Combining distributional and morphological information for part of speech induction. In Proceedings of EACL 2003, pages 59–66, Morristown, NJ, USA.
  5. Tomaž Erjavec. 2004. MULTEXT-East Version 3: Multilingual Morphosyntactic Specifications, Lexicons and Corpora. In Fourth International Conference on Language Resources and Evaluation, LREC’04, page In print, Paris. ELRA. find
  6. Stella Frank, Sharon Goldwater, and Frank Keller. 2009. Evaluating models of syntactic category acquisition without using a gold standard. In Proceedings of CogSci09, July. find
  7. Kuzman Ganchev, João Graça, Jennifer Gillenwater, and Ben Taskar. 2009. Posterior regularization for structured latent variable models. Technical report, University of Pennsylvania. find
  8. Jianfeng Gao and Mark Johnson. 2008. A comparison of bayesian estimators for unsupervised hidden markov model pos taggers. In Proceedings of EMNLP 2008, pages 344–352, Morristown, NJ, USA. find
  9. Sharon Goldwater and Tom Griffiths. 2007. A fully bayesian approach to unsupervised part-of-speech tagging. In Proceedings of ACL 2007, pages 744–751, Prague, Czech Republic, June.
  10. Joao Graca, Kuzman Ganchev, Ben Taskar, and Fernando Pereira. 2009. Posterior vs parameter sparsity in latent variable models. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 664–672. find
  11. Aria Haghighi and Dan Klein. 2006. Prototype-driven learning for sequence models. In Proceedings of NAACL 2006, pages 320–327, Morristown, NJ, USA.
  12. Mark Johnson. 2007. Why doesn’t EM find good HMM POS-taggers? In Proceedings of EMNLP-CoNLL 2007, pages 296–305, Prague, Czech Republic, June.
  13. M. Marcus, B. Santorini, and M. A. Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):331–330.
  14. Marina Meilǎ. 2003. Comparing clusterings by the variation of information. In Learning Theory and Kernel Machines, pages 173–187. find
  15. B. Merialdo. 1994. Tagging English text with a probabilistic model. Computational Linguistics, 20(2):155– 172.
  16. Tahira Naseem, Benjamin Snyder, Jacob Eisenstein, and Regina Barzilay. 2009. Multilingual part-of-speech tagging: Two unsupervised approaches. Journal of Artificial Intelligence Research, 36:341–385. find
  17. Sujith Ravi and Kevin Knight. 2009. Minimized models for unsupervised part-of-speech tagging. In Proceedings of ACL-IJCNLP 2009, pages 504–512, Suntec, Singapore, August. find
  18. Andrew Rosenberg and Julia Hirschberg. 2007. Vmeasure: A conditional entropy-based external cluster evaluation measure. In Proceedings of EMNLP-CoNLL 2007, pages 410–420. find
  19. Noah A. Smith and Jason Eisner. 2005. Contrastive estimation: training log-linear models on unlabeled data. In Proceedings of ACL 2005, pages 354–362, Morristown, NJ, USA.
  20. K. Toutanova and M. Johnson. 2007. A Bayesian LDA-based model for semi-supervised part-of-speech tagging. In Proceedings of NIPS 2007. find
  21. Jurgen Van Gael, Andreas Vlachos, and Zoubin Ghahramani. 2009. The infinite HMM for unsupervised PoS tagging. In Proceedings of EMLNP 2009, pages 678– 687, Singapore, August. find
  22. Andreas Vlachos, Anna Korhonen, and Zoubin Ghahramani. 2009. Unsupervised and constrained dirichlet process mixture models for verb clustering. In Proceedings of GEMS 2009, pages 74–82, Morristown, NJ, USA. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/christodoulopoulos-2010-two-decades-unsupervised-pos-induction/