Elamur Research⌕
☰
home / papers / 2011 / Unsupervised Structure Prediction with Non-Parallel Multilingual Guidance

Unsupervised Structure Prediction with Non-Parallel Multilingual Guidance

Paper · 2011 · Shay B. Cohen, Dipanjan Das and Noah A. Smith

Summary

The paper tackles building a part-of-speech tagger and a dependency parser for a language that has only raw, unannotated text. It borrows knowledge from other languages that do have treebanks, and it does this without any parallel corpora, which exist only in a few domains. The authors first train supervised generative models (an HMM for tagging and the DMV of Klein and Manning for dependencies) on helper-language treebanks whose tags are collapsed to a shared coarse, near-universal tagset. For the target language, each multinomial is restricted to a convex combination of the helper distributions. Only the per-distribution mixing weights are learned from unlabeled target text, using EM over an equivalent extended PCFG. The resulting model is then expanded to fine-grained tags and serves as the initializer for ordinary unsupervised training. The approach beats strong unsupervised baselines on both tasks. It also shows that joint tag-and-tree decoding can sometimes outperform parsing from gold tags. This matters for low-resource languages that have neither annotation nor translated text.

Key points

  • The authors evaluate on ten target languages: Portuguese, Turkish, Bulgarian, Japanese, Greek, Swedish, Spanish, Slovene, Dutch and Danish.
  • Baselines include posterior regularization (PR) and the phylogenetic model of Berg-Kirkpatrick and Klein (2010), "Phylo". Supervised MLE serves as an oracle upper bound.
  • Setting all mixing weights to uniform values already beats most unsupervised baselines.
  • Multilingual guidance helps dependency induction more than POS tagging. The authors' explanation is that HMM transition matrices encode word order, which differs widely across helper languages, while some dependency patterns are shared broadly.
  • In parsing directly from words, joint decoding over tag lattices reached a macro-average of 57.9 (Joint DG), compared with 56.9 for the comparable pipeline. The best gold-tag multilingual result averaged 64.5.
  • For Turkish, Japanese, Slovene and Dutch, the word-based joint system scored higher than parsing with gold-standard POS tags. For example, Turkish reached 52.4 versus 45.3.

Interesting ideas

  • I1. Learn only the mixing weights over foreign models. Helper-language distributions stay frozen, and only interpolation coefficients are fit on raw target data. This drastically shrinks what must be learned unsupervised, and it could transfer to any task where per-language component models share a parameter space.
  • I2. Use the transferred model as an initializer, not a final answer. The mixture model is coarse, so it is expanded to fine-grained tags and then refined by standard EM. This two-stage approach could stabilize other unsupervised learners that are sensitive to initialization.
  • I3. Express the mixture as a bigger grammar. Each mixture choice becomes an extra nonterminal, giving an "extended" PCFG. Existing inside-outside or forward-backward code can then estimate the weights with no new inference machinery.
  • I4. Inspect the learned weights as a diagnostic. Projecting the coefficient vectors with PCA showed related languages clustering together. This offers a cheap sanity check that a transfer model is picking up meaningful similarity, and it could help when choosing source languages, for example for Ukrainian.
  • I5. Pass tag uncertainty to the parser. Feeding the parser a weighted "sausage" lattice of tag posteriors, instead of a single best tag sequence, usually helped. This is a general way to reduce error propagation in pipelines.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksPart-of-speech tagging · 126 works

Cite

Cite as Shay B. Cohen, Dipanjan Das and Noah A. Smith. 2011. Unsupervised Structure Prediction with Non-Parallel Multilingual Guidance. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{cohen2011unsupervised,
  title = {{Unsupervised Structure Prediction with Non-Parallel Multilingual Guidance}},
  author = {Shay B. Cohen, Dipanjan Das and Noah A. Smith},
  booktitle = {Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing},
  year = {2011},
  url = {https://aclanthology.org/D11-1005/}
}

References

37 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. Berg-Kirkpatrick and D. Klein. 2010. Phylogenetic grammar induction. In Proceedings of ACL. find
  2. T. Berg-Kirkpatrick, A. B. Côté, J. DeNero, and D. Klein. 2010. Painless unsupervised learning with features. In Proceedings of NAACL-HLT.
  3. S. Brants, S. Dipper, S. Hansen, W. Lezius, and G. Smith. 2002. The TIGER treebank. In Proceedings of the Workshop on Treebanks and Linguistic Theories. find
  4. S. Buchholz and E. Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proceedings of CoNLL.
  5. D. Burkett and D. Klein. 2008. Two languages are better than one (for syntactic parsing). In Proceedings of EMNLP. find
  6. S. B. Cohen and N. A. Smith. 2007. Joint morphological and syntactic disambiguation. In Proceedings of EMNLP-CoNLL. find
  7. S. B. Cohen and N. A. Smith. 2009. Shared logistic normal distributions for soft parameter tying in unsupervised grammar induction. In Proceedings of HLT-NAACL.
  8. S. B. Cohen and N. A. Smith. 2010. Covariance in unsupervised learning of probabilistic grammars. Journal of Machine Learning Research, 11:3017–3051. find
  9. D. Das and S. Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proceedings of ACL-HLT.
  10. D. Elworthy. 1994. Does Baum-Welch re-estimation help taggers? In Proceedings of ACL. find
  11. K. Ganchev, J. Graça, J. Gillenwater, and B. Taskar. 2010. Posterior regularization for structured latent variable models. Journal of Machine Learning Research, 11:2001–2049. find
  12. J. Gillenwater, K. Ganchev, J. Graça, F. Pereira, and B. Taskar. 2010. Sparsity in dependency grammar induction. In Proceedings of ACL. find
  13. A. Haghighi and D. Klein. 2006. Prototype driven learning for sequence models. In Proceedings of HLT-NAACL.
  14. J. Hajič. 1998. Building a syntactically annotated corpus: The Prague Dependency Treebank. In Issues of Valency and Meaning. Studies in Honor of Jarmila Panevová. Prague Karolinum, Charles University Press. find
  15. W. P. Headden, M. Johnson, and D. McClosky. 2009. Improving unsupervised dependency parsing with richer contexts and smoothing. In Proceedings of NAACL-HLT. find
  16. M. Johnson, T. L. Griffiths, and S. Goldwater. 2007. Bayesian inference for PCFGs via Markov chain Monte Carlo. In Proceedings of NAACL.
  17. D. Klein and C. D. Manning. 2004. Corpus-based induction of syntactic structure: Models of dependency and constituency. In Proceedings of ACL. find
  18. D. C. Liu and J. Nocedal. 1989. On the limited memory BFGS method for large scale optimization. Math. Programming, 45:503–528. find
  19. M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. 1993. Building a large annotated corpus of English: the Penn treebank. Computational Linguistics, 19.
  20. R. McDonald, S. Petrov, and K. Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proceedings of EMNLP.
  21. B. Merialdo. 1994. Tagging English text with a probabilistic model. Compulational Lingustics, 20(2):155– 72.
  22. S. Montemagni, F. Barsotti, M. Battista, N. Calzolari, O. Corazzari, A. Zampolli, F. Fanciulli, M. Massetani, R. Raffaelli, R. Basili, M. T. Pazienza, D. Saracino, F. Zanzotto, N. Mana, F. Pianesi, and R. Delmonte. 2003. Building the Italian Syntactic-Semantic Treebank. In Building and using Parsed… find
  23. T. Naseem, B. Snyder, J. Eisenstein, and R. Barzilay. 2009. Multilingual part-of-speech tagging: Two unsupervised approaches. JAIR, 36. find
  24. T. Naseem, H. Chen, R. Barzilay, and M. Johnson. 2010. Using universal linguistic knowledge to guide grammar induction. In Proceedings of EMNLP.
  25. J. Nivre, J. Hall, S. Kübler, R. McDonald, J. Nilsson, S. Riedel, and D. Yuret. 2007. find
  26. The CoNLL 2007 shared task on dependency parsing. In Proceedings of CoNLL.
  27. S. Petrov, D. Das, and R. McDonald. 2011. A universal part-of-speech tagset. ArXiv:1104.2086.
  28. Y. Seginer. 2007. Fast unsupervised incremental parsing. In Proceedings of ACL. find
  29. N. A. Smith and J. Eisner. 2005. Contrastive estimation: Training log-linear models on unlabeled data. In Proceedings of ACL.
  30. D. A. Smith and J. Eisner. 2009. Parser adaptation and projection with quasi-synchronous grammar features. In Proceedings of EMNLP. find
  31. D. A. Smith and N. A. Smith. 2004. Bilingual parsing with factored estimation: Using English to parse Korean. In Proceedings of EMNLP. find
  32. N. A. Smith. 2006. Novel Estimation Methods for Unsupervised Discovery of Latent Structure in Natural Language Text. Ph.D. thesis, Johns Hopkins University. find
  33. B. Snyder and R. Barzilay. 2008. Unsupervised multilingual learning for morphological segmentation. In Proceedings of ACL.
  34. B. Snyder, T. Naseem, and R. Barzilay. 2009. Unsupervised multilingual grammar induction. In Proceedings of ACL-IJCNLP. find
  35. V. Spitkovsky, H. Alshawi, and D. Jurafsky. 2010. From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing. In Proceedings of NAACL. find
  36. C. Xi and R. Hwa. 2005. A backoff model for bootstrapping resources for non-English languages. In Proceedings of HLT-EMNLP. find
  37. D. Yarowsky and G. Ngai. 2001. Inducing multilingual POS taggers and NP bracketers via robust projection across aligned corpora. In Proceedings of NAACL. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cohen-2011-unsupervised-structure-prediction-non-parallel/