📄 A fully Bayesian approach to unsupervised part-of-speech tagging

Summary

The paper addresses unsupervised part-of-speech tagging. Standard methods fit one "best" parameter setting by maximum-likelihood estimation (MLE) and then pick tags under that setting. Goldwater and Griffiths instead use a fully Bayesian approach. They keep the familiar trigram HMM structure but place symmetric Dirichlet priors on its transition and emission distributions. They then integrate those parameters out, so tags are inferred directly from the observed words. This has two benefits. Tag sequences are chosen because they remain probable across many parameter values. The approach also allows sparsity-favoring priors (β < 1), which make MAP estimation degenerate. Hyperparameters are learned from unlabeled data. The authors evaluate conditions that vary from having a complete tag dictionary to having none at all, using data sets provided by Noah Smith. Their model beats an MLE-trained HMM on every condition tested, by up to 14 accuracy points, and approaches the discriminative CRF/CE model of Smith and Eisner (2005). The gains are largest when supervision is weak, for example with small corpora or thin dictionaries. This suggests Bayesian integration as a general remedy for underconstrained structure learning.

Key points

• The model, called BHMM, is a trigram HMM with Dirichlet(α) priors on transitions and Dirichlet(β) priors on outputs. With the parameters integrated out, each prediction reduces to smoothed counts of the form (n + α)/(n_context + Tα). Exchangeability of trigrams and outputs is what makes inference workable.

• There are two variants. BHMM1 learns one shared β for all tags. BHMM2 learns a separate output hyperparameter for each tag/cluster.

• With the full dictionary (d=1), accuracy was:

  Model	Accuracy
  MLHMM	83.2%
  BHMM1	86.0%
  BHMM2	87.3%
  CRF/CE	90.4%

• Without a dictionary, variation of information (VI) dropped from 6.50 for MLHMM to 4.30 for BHMM1 and 4.04 for BHMM2. Random tagging scored 7.29.

• MLHMM tended to produce equally sized clusters, averaging 542 word types per cluster with a standard deviation of 174. BHMM2 clusters were much more uneven, with standard deviations of roughly 460–674. They were also linguistically more coherent: determiners, prepositions and punctuation were grouped cleanly, and common and proper nouns were split.

Interesting ideas

• I1. Integrate out parameters instead of point-estimating them. A decision that holds across many plausible parameter values is more trustworthy than one tuned to a single estimate. The paper's toy example about deciding whether a coin is biased shows this clearly. The same idea could apply to any latent-variable model trained on scarce data, such as low-resource Ukrainian corpora.
• I2. Use sparse priors (β < 1) where natural-language distributions are skewed. Under MAP estimation such priors break down, but they work once parameters are marginalized. This suits tag transitions, word emissions, and other heavily long-tailed linguistic distributions.
• I3. Evaluate clusterings with variation of information. VI compares two clusterings independently of their labels. This makes dictionary-free induction measurable, and it can separate two systems with equal accuracy whose errors differ in consistency.
• I4. Sweep the amount of supervision. Running one framework from full dictionary to no dictionary links disambiguation and pure clustering in a single study. It also shows where a method's advantage is largest, which is a useful evaluation design for other weakly supervised tasks.
• I5. Watch for junk clusters when hyperparameters are learned per cluster. In BHMM2, separate hyperparameters led to "junk" clusters that collected rare words. A shared hyperparameter, or a prior over hyperparameters, can steer rare items toward clusters with similar contexts instead.