Elamur Research⌕
☰
home / papers / 2009 / Shared Logistic Normal Distributions for Soft Parameter Tying in Unsupervised Grammar Induction

Shared Logistic Normal Distributions for Soft Parameter Tying in Unsupervised Grammar Induction

Paper · 2009 · Shay B. Cohen and Noah A. Smith

Summary

Unsupervised induction of dependency grammars usually puts Dirichlet priors on grammar weights. Dirichlets cannot express beliefs about which rule probabilities should rise or fall together. An earlier logistic normal (LN) prior (Cohen et al., 2008) allowed such correlations, but only inside a single multinomial. Cohen and Smith generalize this into the shared logistic normal (SLN) distribution. In the SLN, several Gaussian "experts" are sampled and their components are averaged before the softmax, so weights in different multinomials can be softly tied. They derive a variational EM algorithm within an empirical Bayes framework and apply it to the DMV model of Klein and Manning (2004). In experiments on the Penn Treebank and the Chinese Treebank, tying distributions for heads of the same coarse category (verbs, nouns, adjectives) beats the LN, Dirichlet and EM baselines. They also train English and Chinese grammars jointly from non-parallel treebanks by sharing experts across languages, which gives the best accuracy for both. This shows that cross-lingual syntactic knowledge can be exploited without parallel text.

Key points

  • The model is the unlexicalized DMV for projective, unlabeled dependencies. It is trained fully unsupervised on POS-tag sequences, and evaluation uses attachment accuracy on sentences of length ≤10, ≤20 and all lengths.
  • On English (Penn Treebank, all lengths), accuracy rises from 40.5% (LN baseline) to 41.4% with monolingual SLN tying of verbs and nouns, and to 42.2% with bilingual tying. The supervised MLE ceiling is 68.8%.
  • For Chinese (Chinese Treebank), the best results also come from the bilingual setting, with the TIE A and TIE V&N variants.
  • Tag sets are mapped onto twelve coarse families (noun, verb, adjective, etc.). Covariance matrices start at 1 on the diagonal, 0.5 for tags in the same family, and 0 elsewhere.
  • Baselines are attach-right, EM (Klein & Manning, 2004), and Dirichlet and LN empirical Bayes (Cohen et al., 2008).

Interesting ideas

  • I1. Soft tying via shared Gaussian experts. Several normal vectors are drawn and their entries are averaged before normalizing. Any parameters across separate distributions can then share statistical strength without being forced to be identical. This could serve any model built from many multinomials, such as HMMs, PCFGs or topic models.
  • I2. Linguistic grouping as the prior's structure. Coarse POS categories decide which distributions share an expert. This is a cheap way to inject typological or category knowledge into unsupervised learning, and it could suit low-resource languages like Ukrainian with UD-style universal tags.
  • I3. Bilingual learning without parallel data. Two grammars are merged into one model and tied only through a shared coarse tagset. Each language benefits from the other's patterns, which is attractive when parallel corpora are scarce.
  • I4. Staged inference. Inference first runs without tying and switches tying on only later. This saves computation and gives the expensive coupled model a sensible starting point, a trick that works for many non-concave objectives.
  • I5. Informed covariance initialization. Covariances begin with positive values within tag families instead of the identity matrix. This simple initialization helped, and it could be reused wherever correlated priors are learned.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Shay B. Cohen and Noah A. Smith. 2009. Shared Logistic Normal Distributions for Soft Parameter Tying in Unsupervised Grammar Induction. Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{cohen2009shared,
  title = {{Shared Logistic Normal Distributions for Soft Parameter Tying in Unsupervised Grammar Induction}},
  author = {Shay B. Cohen and Noah A. Smith},
  booktitle = {Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
  doi = {10.3115/1620754.1620766},
  url = {https://doi.org/10.3115/1620754.1620766}
}

References

30 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Aitchison. 1986. The Statistical Analysis of Compositional Data. Chapman and Hall, London. find
  2. D. M. Blei and J. D. Lafferty. 2006. Correlated topic models. In Proc. of NIPS. find
  3. D. M. Blei, A. Ng, and M. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022.
  4. D. Burkett and D. Klein. 2008. Two languages are better than one (for syntactic parsing). In Proc. of EMNLP. find
  5. E. Charniak and M. Johnson. 2005. Coarse-to-fine nbest parsing and maxent discriminative reranking. In Proc. of ACL. find
  6. S. B. Cohen and N. A. Smith. 2009. Inference for probabilistic grammars with shared logistic normal distributions. Technical report, Carnegie Mellon University. find
  7. S. B. Cohen, K. Gimpel, and N. A. Smith. 2008. Logistic normal priors for unsupervised probabilistic grammar induction. In NIPS. find
  8. M. Collins. 2003. Head-driven statistical models for natural language processing. Computational Linguistics, 29:589–637. find
  9. I. Dagan. 1991. Two languages are more informative than one. In Proc. of ACL.
  10. J. Eisner. 2002. Transformational priors over grammars. In Proc. of EMNLP. find
  11. J. R. Finkel, T. Grenager, and C. D. Manning. 2007. The infinite tree. In Proc. of ACL. find
  12. J. Goodman. 1996. Parsing algorithms and metrics. In Proc. of ACL. find
  13. A. Haghighi, P. Liang, T. Berg-Kirkpatrick, and D. Klein. 2008. Learning bilingual lexicons from monolingual corpora. In Proc. of ACL.
  14. W. P. Headden, M. Johnson, and D. McClosky. 2009. Improving unsupervised dependency parsing with richer contexts and smoothing. In Proc. of NAACL-HLT. find
  15. G. E. Hinton. 1999. Products of experts. In Proc. of ICANN. find
  16. M. Johnson, T. L. Griffiths, and S. Goldwater. 2006. Adaptor grammars: A framework for specifying compositional nonparameteric Bayesian models. In NIPS. find
  17. M. Johnson, T. L. Griffiths, and S. Goldwater. 2007. Bayesian inference for PCFGs via Markov chain Monte Carlo. In Proc. of NAACL.
  18. M. Johnson. 2007. Why doesn’t EM find good HMM POS-taggers? In Proc. EMNLP-CoNLL.
  19. M. I. Jordan, Z. Ghahramani, T. S. Jaakola, and L. K. Saul. 1999. An introduction to variational methods for graphical models. Machine Learning, 37(2):183– 233. find
  20. D. Klein and C. D. Manning. 2004. Corpus-based induction of syntactic structure: Models of dependency and constituency. In Proc. of ACL. find
  21. K. Kurihara and T. Sato. 2006. Variational Bayesian grammar induction for natural language. In Proc. of ICGI. find
  22. P. Liang, S. Petrov, M. Jordan, and D. Klein. 2007. The infinite PCFG using hierarchical Dirichlet processes. In Proc. of EMNLP. find
  23. M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn treebank. Computational Linguistics, 19:313–330.
  24. D. A. Smith and N. A. Smith. 2004. Bilingual parsing with factored estimation: Using English to parse Korean. In Proc. of EMNLP, pages 49–56. find
  25. N. A. Smith. 2006. Novel Estimation Methods for Unsupervised Discovery of Latent Structure in Natural Language Text. Ph.D. thesis, Johns Hopkins University. find
  26. B. Snyder and R. Barzilay. 2008. Unsupervised multilingual learning for morphological segmentation. In Proc. of ACL.
  27. Y. W. Teh. 2006. A hierarchical Bayesian language model based on Pitman-Yor processes. In Proc. of COLING-ACL.
  28. M. Wang, N. A. Smith, and T. Mitamura. 2007. What is the Jeopardy model? a quasi-synchronous grammar for question answering. In Proc. of EMNLP. find
  29. D. Wu. 1997. Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Comp. Ling., 23(3):377–404.
  30. N. Xue, F. Xia, F.-D. Chiou, and M. Palmer. 2004. The Penn Chinese Treebank: Phrase structure annotation of a large corpus. Natural Language Engineering, 10(4):1–30. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cohen-2009-shared-logistic-normal-distributions-soft/