Elamur Research⌕
☰
home / papers / 2011 / Multi-Source Transfer of Delexicalized Dependency Parsers

Multi-Source Transfer of Delexicalized Dependency Parsers

Summary

The paper tackles dependency parsing for languages that have no annotated treebank. The authors' starting point is that unsupervised grammar induction still trails supervised parsing by a wide margin. Their parser is delexicalized: it is trained on English CoNLL treebank data using only part-of-speech features, with every tagset mapped to the shared universal tagset of Petrov et al. (2011). Because the features are language-neutral, this parser can be run on another language's tag sequences without changes, and that alone beats the unsupervised DMV baseline by a clear margin. In a second step, the transferred model seeds a perceptron for the target language. That perceptron is refined with augmented-loss (constraint-driven) training, which rewards parses that agree with English parses carried over through word alignments in Europarl. Training on treebanks from several source languages at once improves results further. Across eight Indo-European languages, the final parsers beat earlier unsupervised and projection-based systems. A practical consequence is that the score of a delexicalized transferred parser becomes the baseline that new grammar induction work should have to beat.

Key points

  • Setup: CoNLL 2006/2007 treebanks for Danish, Dutch, German, Greek, Italian, Portuguese, Spanish and Swedish, with English used only as a source. Alignments come from Europarl v5 aligned with IBM Model 1 and the HMM model. The parser is an arc-eager transition-based parser trained with the averaged perceptron (beam 8). The metric is unlabeled attachment score (UAS), not LAS, and no treebanks were modified.
  • On English, removing lexical features costs about 7 points: 89.3 UAS with them, 82.5 without. Part-of-speech tags alone therefore carry most of the information needed for unlabeled attachment.
  • Projection through parallel data cuts relative error by about 8% on average compared with direct English transfer, and by up to 18% for some languages. Multi-source training cuts error by about 16% compared with the English direct-transfer baseline.
  • On sentences of 10 words or fewer with gold POS tags, average UAS rises from 66.6 (English direct) to 69.4 (English projected), 71.7 (multi-source direct) and 74.1 (multi-source projected). The unsupervised baseline averages 57.4.
  • The multi-source gain holds even when the training data is subsampled down to the size of the English set. Early tests on Arabic, Chinese and Japanese suggest direct transfer is still a reasonable baseline across language families.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Dependency parsing · Treebanks · Part-of-speech tagging

Cited by

190 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 172 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mcdonald2011multi,
  title = {{Multi-Source Transfer of Delexicalized Dependency Parsers}},
  author = {McDonald, Ryan and Petrov, Slav and Hall, Keith},
  booktitle = {Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing},
  year = {2011},
  url = {https://aclanthology.org/D11-1006/}
}

References

47 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. Berg-Kirkpatrick and D. Klein. 2010. Phylogenetic grammar induction. In Proc. of ACL. find
  2. P. Blunsom and T. Cohn. 2010. Unsupervised induction of tree substitution grammars for dependency parsing. Proc. of EMNLP. find
  3. P. F. Brown, V. J. Della Pietra, S. A. Della Pietra, and R. L. Mercer. 1993. The mathematics of statistical machine translation: parameter estimation. Computational Linguistics, 19. find
  4. S. Buchholz and E. Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proc. of CoNLL. find our card
  5. D. Burkett, S. Petrov, J. Blitzer, and D. Klein. 2010. Learning better monolingual models with unannotated bilingual text. In Proc. of CoNLL. find
  6. G. Carroll and E. Charniak. 1992. Two experiments on learning probabilistic dependency grammars from corpora. In Proc. of the Working Notes of the Workshop Statistically-Based NLP Techniques. find
  7. M.W. Chang, L. Ratinov, and D. Roth. 2007. Guiding semi-supervision with constraint-driven learning. In Proc. of ACL. find
  8. M. Chang, D. Goldwasser, D. Roth, and V. Srikumar. 2010. Structured output learning with indirect supervision. In Proc. of ICML. find
  9. E. Charniak. 2000. A maximum-entropy-inspired parser. In Proc. of NAACL. find our card
  10. S. Clark and J. R. Curran. 2004. Parsing the WSJ using CCG and log-linear models. In Proc. of ACL. find
  11. S.B. Cohen and N.A. Smith. 2009. Shared logistic normal distributions for soft parameter tying in unsupervised grammar induction. In Proc. of NAACL. find
  12. S.B. Cohen, D. Das, and N.A. Smith. 2011. Unsupervised structure prediction with non-parallel multilingual guidance. In Proc. of EMNLP. find
  13. M. Collins, J. Hajič, L. Ramshaw, and C. Tillmann. 1999. A statistical parser for Czech. In Proc. of ACL. find
  14. M. Collins. 1997. Three generative, lexicalised models for statistical parsing. In Proc. of ACL. find
  15. M. Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proc. of ACL. find our card
  16. D. Das and S. Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proc. of ACL-HLT. find
  17. K. Ganchev, J. Gillenwater, and B. Taskar. 2009. Dependency grammar induction via bitext projection constraints. In Proc. of ACL-IJCNLP. find
  18. K. Ganchev, J. Graça, J. Gillenwater, and B. Taskar. 2010. Posterior regularization for structured latent variable models. Journal of Machine Learning Research. find
  19. D. Gildea. 2001. Corpus variation and parser performance. In Proc of EMNLP. find our card
  20. K. Hall, R. McDonald, J. Katz-Brown, and M. Ringgaard. 2011. Training dependency parsers by jointly optimizing multiple objectives. In Proc. of EMNLP. find
  21. R. Hwa, P. Resnik, A. Weinberg, C. Cabezas, and O. Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel texts. Natural Language Engineering, 11(03):311–325. find
  22. D. Klein and C. D. Manning. 2004. Corpus-based induction of syntactic structure: models of dependency and constituency. In Proc. of ACL. find
  23. P. Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT Summit. find
  24. M. P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn treebank. Computational Linguistics, 19. find our card
  25. D. McClosky, E. Charniak, and M. Johnson. 2006. Reranking and self-training for parser adaptation. In Proc. of ACL. find our card
  26. R. McDonald, K. Crammer, and F. Pereira. 2005. Online large-margin training of dependency parsers. In Proc. of ACL. find our card
  27. T. Naseem, H. Chen, R. Barzilay, and M. Johnson. 2010. Using universal linguistic knowledge to guide grammar induction. In Proc. of EMNLP. find
  28. J. Nivre and J. Nilsson. 2005. Pseudo-projective dependency parsing. In Proc. of ACL. find
  29. J. Nivre, J. Hall, and J. Nilsson. 2006. Maltparser: A data-driven parser-generator for dependency parsing. In Proc. of LREC. find our card
  30. J. Nivre, J. Hall, S. Kübler, R. McDonald, J. Nilsson, S. Riedel, and D. Yuret. 2007. find
  31. The CoNLL 2007 shared task on dependency parsing. In Proc. of EMNLP-CoNLL. find
  32. J. Nivre. 2008. Algorithms for deterministic incremental dependency parsing. Computational Linguistics, 34(4):513–553. find our card
  33. S. Petrov, L. Barrett, R. Thibaux, and D. Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proc. of ACL. find our card
  34. S. Petrov, P. Chang, M. Ringgaard, and H. Alshawi. 2010. Uptraining for accurate deterministic question parsing. In EMNLP ’10. find
  35. S. Petrov, D. Das, and R. McDonald. 2011. A universal part-of-speech tagset. In ArXiv:1104.2086. find
  36. Y. Seginer. 2007. Fast unsupervised incremental parsing. In Proc. of ACL. find
  37. L. Shen and A.K. Joshi. 2008. Ltag dependency parsing with bidirectional incremental construction. In Proc. of EMNLP. find
  38. N.A. Smith and J. Eisner. 2005. Contrastive estimation: Training log-linear models on unlabeled data. In Proc. of ACL. find
  39. D.A. Smith and J. Eisner. 2009. Parser adaptation and projection with quasi-synchronous grammar features. In Proc. of EMNLP. find
  40. D.A. Smith and N.A. Smith. 2004. Bilingual parsing with factored estimation: Using english to parse korean. In Proc. of EMNLP. find
  41. B. Snyder, T. Naseem, J. Eisenstein, and R. Barzilay. 2009. Adding more languages improves unsupervised multilingual part-of-speech tagging: A Bayesian non-parametric approach. In Proc. of NAACL. find
  42. A. Søgaard. 2011. Data point selection for cross-language adaptation of dependency parsers. In Proc. ACL. find
  43. V.I. Spitkovsky, H. Alshawi, and D. Jurafsky. 2010. From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing. In Proc. of NAACL-HLT. find
  44. S. Vogel, H. Ney, and C. Tillmann. 1996. HMM-based word alignment in statistical translation. In Proc. of COLING. find
  45. W. Wang and M. P. Harper. 2004. A statistical constraint dependency grammar (CDG) parser. In Proc. of the Workshop on Incremental Parsing: Bringing Engineering and Cognition Together. find
  46. D. Zeman and P. Resnik. 2008. Cross-language parser adaptation between related languages. In NLP for Less Privileged Languages. find
  47. Y. Zhang and S. Clark. 2008. A Tale of Two Parsers: Investigating and Combining Graph-based and Transition-based Dependency Parsing. In Proc. of EMNLP. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mcdonald-2011-multi-source-transfer-delexicalized-dependency/