📄 Data point selection for cross-language adaptation of dependency parsers

Summary

This short ACL 2011 paper asks how to obtain a dependency parser for a language that has no treebank of its own. The resources it relies on are labeled treebanks in other, possibly distant languages and unannotated target text. Søgaard extends the delexicalized transfer approach of Zeman and Resnik (2008). Source treebanks are stripped of words, and their POS tags are mapped to a shared tagset. A trigram language model is then trained on the target language's POS sequences. Each source sentence is scored by per-word perplexity under that model, and only the most target-like sentences are used to train the parser. The method is framed as a crude form of instance weighting under covariate shift, with binary weights. Experiments use the Arabic, Bulgarian, Danish and Portuguese treebanks from CoNLL 2006, a much less related set of languages than earlier work used. Dropping the least similar 10% of source sentences beats the unfiltered delexicalized baseline, sometimes by a wide margin. It also outperforms recent unsupervised parsers and roughly matches projection methods that need parallel text.

Key points

• Setup:
  • Four CoNLL 2006 treebanks with standard splits.
  • Tags unified via the Interset mappings used by Zeman and Resnik.
  • For each target language, the other three treebanks serve as the training material.
  • The parser is MSTParser (second-order, non-projective, MIRA) with default settings.
• Language model: an unsmoothed trigram POS model built with SRILM. The author notes that modified Kneser-Ney smoothing does not apply to POS sequences.
• Full-length accuracy (baseline → 90% most similar):
  • Bulgarian: 44.5 → 70.2
  • Portuguese: 37.1 → 75.1
  • Arabic: 45.5 → 48.4
  • Danish: 51.7 → 51.9
• Effect of the selection ratio:
  • Choosing the ratio with 100 held-out sentences adds a little (e.g. Arabic 49.2, Danish 52.8).
  • Even 10% of the Bulgarian source data gives 66.9, well above the baseline.
• Single-source exploration: using one source language can match or slightly beat the three-source setup. The best pairs were:
  • Portuguese from Bulgarian (76.9)
  • Bulgarian from Portuguese (70.3)
  • Danish from Arabic (56.5)
  • Arabic from Bulgarian (50.2)
• Weak Danish gains: the author attributes these partly to the Danish treebank's unusual annotation, such as its DP-analysis of noun phrases.

Interesting ideas

• I1. Perplexity as a similarity filter: a language model trained on unlabeled target data can rank labeled source examples by how target-like they are. This filtering idea could carry over to domain adaptation in tagging, NER or MT, where target labels are missing.
• I2. Binary instance weights: the importance weight P_t(x)/P_s(x) is reduced to keep-or-drop. This turns a theoretically motivated reweighting into a parameter-free data filter that wraps any learner without changing its internals.
• I3. A sensible default ratio: in this paper, discarding just the worst 10% gave most of the gain, and the exact amount kept mattered less. This makes a fixed default viable when no target dev data exists.
• I4. Abstracting away the lexicon: delexicalization plus a universal POS tagset lets structurally distant languages share training data. The pairwise experiments suggest checking non-obvious source languages rather than assuming that genealogical closeness is best.
• I5. Error analysis by dependency length and root attachment: breaking errors down this way reveals where an adaptation method actually helps. For Portuguese, the largest improvements came at the root and on long dependencies.