📄 Frustratingly Easy Domain Adaptation

Summary

The paper tackles fully supervised domain adaptation, where a large labeled corpus exists for a source domain (e.g., newswire) but only a small labeled set is available for the target domain. Rather than designing a new learner, the author turns adaptation into ordinary supervised learning by expanding the feature space. Each original feature is copied into three variants: shared, source-only and target-only. Source examples fill the shared and source slots, target examples fill the shared and target slots, and the unused slots stay zero. Any standard classifier, such as maxent, SVMs or the perceptron, can then decide for each feature whether its behavior carries over between domains. The transformation fits into a tiny Perl preprocessing script and generalizes to K domains with K+1 copies. It also has a kernel interpretation in which same-domain points count as twice as similar. The method matches or beats strong baselines and the Prior model while being much simpler than earlier EM-based adaptation. Its value is practical: it gives a strong adaptation baseline that anyone can apply almost immediately.

Key points

• The complete transformation is given in the appendix as a Perl script of 10 lines and 194 characters.
• The method is compared against several simple baselines and the Prior model of Chelba and Acero (2004):
  • SrcOnly, TgtOnly and All;
  • Weighted (source examples down-weighted, with the weight chosen by cross-validation);
  • Pred (the source model's prediction used as an extra feature);
  • LinInt (interpolation of the source and target models).
• The earlier three-component EM model of Daumé III and Marcu (2006) performed well but was hard to implement and roughly 10–15 times slower than training Prior.
• In kernel form, the expanded kernel equals 2K(x, x′) for same-domain pairs and K(x, x′) across domains. All experiments, however, use a linear kernel.
• The learned weights on ACE-NER were inspected with Hinton diagrams across a general domain and the domains bn, bc, nw, wl, un and cts.
  • The capitalization feature /Aa+/ is ignored in bn, which has no case information.
  • The same feature is negative in usenet because emails and URLs match the pattern.
• On the CoNLL section of the treebank, a small but real improvement is reported, from 5.35 to 5.11.

Interesting ideas

• I1. Domain-tagged feature duplication as pure preprocessing. The learner stays untouched, so the trick can be dropped into any existing linear pipeline. One example is a Ukrainian tagger trained mostly on news but meant for forums or legal text.
• I2. Shared plus per-domain copies for many domains. Using K+1 copies gives a cheap way to train one model over several corpora at once (e.g., different text genres or trade-document sources). Each domain can then deviate from the shared behavior only where its data demands it.
• I3. Reading the per-domain weights as a diagnostic. Comparing the shared and domain-specific weights of a feature shows where domains truly differ, such as capitalization or name lists. This is a useful sanity check before deciding whether domains should be merged or kept separate.
• I4. Joint rather than sequential optimization. The author relates the method to the Prior regularizer, which pulls source and target weights together. He argues the advantage comes from learning both weight sets in a single training run instead of chaining two models, which also avoids per-task hyperparameter tuning.
• I5. A tunable domain-similarity factor. The kernel view suggests replacing the fixed factor of 2 with a hyperparameter tuned on development data. This is a natural knob to try whenever domains are known to be closer or further apart.