Elamur Research⌕
☰
home / papers / 2013 / Target Language Adaptation of Discriminative Transfer Parsers

Target Language Adaptation of Discriminative Transfer Parsers

Paper · 2013 · Oscar Täckström, Ryan McDonald and Joakim Nivre

Summary

The paper addresses dependency parsing for languages that have no treebank of their own. It does this by transferring parsers trained on other languages. Earlier state-of-the-art work by Naseem et al. (2012) shared parameters selectively across source languages based on typology. That work relied on a weak generative model whose supervised accuracy stays well below discriminative parsers.

The authors bring selective sharing into a delexicalized, arc-factored discriminative graph-based parser. They split its features into a language-generic part and parts tied to WALS word-order traits and language family. They then adapt this transfer parser to the target language using unlabeled target text. Their learning objective accepts ambiguous labelings, meaning a set of allowed heads per word rather than a single tree. This supports two variants:

  • Self-training: arcs filtered by the base parser's marginals.
  • Ensemble-training: those arcs plus the predictions of another transfer parser.

Both variants also relexicalize the model. Evaluated on 16 CoNLL treebanks, the final model beats Naseem et al. on 15 of them. The work shows that rich discriminative models and cheap unlabeled data can push cross-lingual transfer further.

Key points

  • Setup: 16 target languages, following the data and setup of Naseem et al. (2012).

    • The remaining treebanks serve as source training data.
    • Coarse "universal" POS tags use the Naseem et al. (2010) mapping.
    • Stripped CoNLL training sets act as unlabeled target data.

  • Typological features: WALS features 81A, 85A, 86A and 87A drive selective sharing. Features 88A and 89A were dropped because they barely vary across the studied languages.

  • Gap motivating the work (supervised, averaged over 16 languages):

    • Naseem et al.'s generative model: 67.1%.
    • Lexicalized arc-factored discriminative parser: 84.1%.
    • Delexicalized discriminative parser: 78.9%.

  • Average transfer accuracy with unlabeled target data:

    ModelAvg. accuracy
    NBG+EM60.4
    Family (base parser)62.0
    Viterbi self-training62.4
    AAST64.0
    AAET65.4

    The Family model alone beats Naseem et al. on 12 of 16 languages. AAET cuts errors by 13% relative to NBG+EM.

  • Ambiguity level and headroom:

    • With the threshold σ = 0.95 (and λ = 1, since tuning was impossible), dependents keep about 1.4–3.2 candidate heads on average.
    • The oracle accuracy of AAST's ambiguous labelings is 75.7%.

Interesting ideas

  • I1. Typology-conditioned feature templates. Order-sensitive features can be conjoined with a language's WALS values, while order-free attachment features stay shared. This lets a single model pool many source languages without pretending the target is a blend of them. The approach could suit transfer to Ukrainian from a mixed set of Slavic and non-Slavic sources.
  • I2. Ambiguous labelings instead of single 1-best trees. The base parser's high-marginal arcs are kept as a set of acceptable heads, and the model is trained to put mass on any of them. Unlike plain Viterbi self-training, this gave consistent gains rather than occasional drops. It is a general recipe for noisy pseudo-labels in any structured task.
  • I3. Ensembles as label sets. Predictions from a different parser can simply be added to the allowed-arc set. This merges heterogeneous systems without needing their probabilities, which is useful when only another system's outputs are available.
  • I4. Relexicalize during adaptation. Training the self-trained model with lexical features on target text is where the gain comes from. Delexicalized self-training moved the average only from 62.0 to 62.1, so unlabeled data pays off mainly when it unlocks new feature types.
  • I5. Report the supervised ceiling and the oracle of pseudo-labels. Comparing each transfer model with its supervised upper bound, and measuring how often the ambiguous set contains the gold arc, shows how much room is left. This diagnostic is cheap and portable to other low-resource evaluations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓Uppsala UniversityUppsala · Sweden56 works · 13,046 citations🤝Swedish Institute of Computer ScienceKista · Sweden4 works · 587 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Oscar Täckström, Ryan McDonald and Joakim Nivre. 2013. Target Language Adaptation of Discriminative Transfer Parsers. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{tackstrom2013target,
  title = {{Target Language Adaptation of Discriminative Transfer Parsers}},
  author = {Oscar Täckström, Ryan McDonald and Joakim Nivre},
  booktitle = {Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2013},
  url = {https://aclanthology.org/N13-1126/}
}

References

35 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Taylor Berg-Kirkpatrick and Dan Klein. 2010. Phylogenetic grammar induction. In Proceedings of ACL. find
  2. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of ACL.
  3. Shay B. Cohen and Noah A. Smith. 2009. Shared logistic normal distributions for soft parameter tying in unsupervised grammar induction. In Proceedings of NAACL.
  4. Shay B. Cohen, Dipanjan Das, and Noah A. Smith. 2011. Unsupervised structure prediction with non-parallel multilingual guidance. In Proceedings of EMNLP. find
  5. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In Proceedings of ACL-HLT.
  6. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of ACL.
  7. Mark Dredze, Partha Pratim Talukdar, and Koby Crammer. 2009. Sequence learning from data with multiple labels. In Proceedings of the ECML/PKDD Workshop on Learning from Multi-Label Data. find
  8. Matthew S. Dryer and Martin Haspelmath, editors. 2011. The World Atlas of Language Structures Online. Munich: Max Planck Digital Library. http://wals.info/. ↗
  9. Greg Durrett, Adam Pauls, and Dan Klein. 2012. Syntactic transfer using a bilingual lexicon. In Proceedings of EMNLP-CoNLL. find
  10. Jason Eisner. 1996. Three new probabilistic models for dependency parsing: an exploration. In Proceedings of COLING.
  11. Jenny Rose Finkel and Christopher D. Manning. 2009. Hierarchical Bayesian domain adaptation. In Proceedings of HLT-NAACL. find
  12. Kuzman Ganchev, Jennifer Gillenwater, and Ben Taskar. 2009. Dependency grammar induction via bitext projection constraints. In Proceedings of ACL-IJCNLP.
  13. Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel texts. Natural Language Engineering, 11(03):311–325. find
  14. Rong Jin and Zoubin Ghahramani. 2002. Learning with multiple labels. In Proceedings of NIPS. find
  15. Dan Klein and Chris D. Manning. 2004. Corpus-based induction of syntactic structure: models of dependency and constituency. In Proceedings of ACL. find
  16. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of COLING.
  17. Dong C. Liu and Jorge Nocedal. 1989. On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45. find
  18. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Reranking and self-training for parser adaptation. In Proceedings of ACL.
  19. Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Online large-margin training of dependency parsers. In Proceedings of ACL.
  20. Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proceedings of EMNLP.
  21. Tahira Naseem, Harr Chen, Regina Barzilay, and Mark Johnson. 2010. Using universal linguistic knowledge to guide grammar induction. In Proceedings of EMNLP.
  22. Tahira Naseem, Regina Barzilay, and Amir Globerson. 2012. Selective sharing for multilingual dependency parsing. In Proceedings of ACL.
  23. Joakim Nivre. 2008. Algorithms for deterministic incremental dependency parsing. Computational Linguistics, 34(4):513–553.
  24. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proceedings of LREC.
  25. Stefan Riezler, Tracy H. King, Ronald M. Kaplan, Richard Crouch, John T. Maxwell, III, and Mark Johnson. 2002. Parsing the Wall Street Journal using a lexical-functional grammar and discriminative estimation techniques. In Proceedings of ACL.
  26. Kenji Sagae and Alon Lavie. 2006. Parser combination by reparsing. In Proceedings of NAACL.
  27. Kenji Sagae and Jun’ichi Tsujii. 2007. Dependency parsing and domain adaptation with LR models and parser ensembles. In Proceedings of EMNLP-CoNLL.
  28. Benjamin Snyder, Tahira Naseem, Jacob Eisenstein, and Regina Barzilay. 2009. Adding more languages improves unsupervised multilingual part-of-speech tagging: A Bayesian non-parametric approach. In Proceedings of NAACL. find
  29. Anders Søgaard and Julie Wulff. 2012. An empirical study of non-lexical extensions to delexicalized transfer. In Proceedings of COLING. find
  30. Anders Søgaard. 2011. Data point selection for cross-language adaptation of dependency parsers. In Proceedings of ACL.
  31. Kathrin Spreyer and Jonas Kuhn. 2009. Data-driven dependency parsing of new languages using incomplete and noisy training data. In Proceedings of CONLL. find
  32. Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proceedings of NAACL-HLT.
  33. Oscar Täckström. 2012. Nudging the envelope of direct transfer methods for multilingual named entity recognition. In Proceedings of the NAACL-HLT 2012 Workshop on Inducing Linguistic Structure (WILS 2012). find
  34. David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of HLT.
  35. Daniel Zeman and Philip Resnik. 2008. Cross-language parser adaptation between related languages. In IJC-NLP Workshop: NLP for Less Privileged Languages.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tackstrom-2013-target-language-adaptation-discriminative-transfer/