📄 Instance Weighting for Domain Adaptation in NLP

Summary

Jiang and Zhai address domain adaptation: a model trained on plentiful labeled data from a source domain must work in a related target domain that has few or no labels. They split the problem in two by factoring the joint distribution. In labeling adaptation, the conditional p(y|x) itself differs between domains. In instance adaptation, p(y|x) stays roughly the same but the input distribution p(x) shifts. Each case calls for different remedies. The authors then build one weighted log-likelihood objective that combines three heuristics:

• discarding source examples whose labels likely mislead in the target domain;
• up-weighting the small labeled target set;
• adding unlabeled target examples with predicted labels.

They test it on POS tagging (Oncology target), NE type classification (CTS and WL targets) and personalized spam filtering (users u00–u02). Methods that model the domain gap beat plain supervised and semi-supervised baselines. Exploiting more target-domain data helps much more than pruning source data. The work matters because it offers a principled, model-agnostic way to transfer discriminative NLP models across domains.

Key points

• The theory separates two adaptation needs: a shift in p(y|x) (labeling adaptation) and a shift in p(x) (instance adaptation).
• For labeling shift, the paper lists three remedies: changing the feature representation, putting a prior on model parameters, and pruning misleading instances. Each needs target labels or prior knowledge about the target domain.
• For input shift, the ideal reweighting factor is the density ratio p_t(x)/p_s(x), an idea known in statistics (Shimodaira, 2000).
• Adding labeled target data with larger weights helps substantially. For spam user u02, accuracy climbs from 0.7644 with source only to 0.9633 when target instances get 10× weight.
• Without any target labels, balanced bootstrapping edges out standard bootstrapping in every setting:
  • POS Oncology: 0.8750 vs 0.8728 (supervised baseline 0.8630).
  • NE type WL: 0.7523 vs 0.7498.
  • Spam u00: 0.8816 vs 0.8720.

Interesting ideas

• I1. Diagnose the kind of shift first. Asking whether labels or inputs changed tells you which fix to try: new features or priors for the former, reweighting or unlabeled data for the latter. This works as a checklist for any transfer project, for example adapting Ukrainian NLP models from news to social media.
• I2. Integrate all heuristics in a single weighted objective. Pruning, up-weighting and self-training become per-instance weights in one loss function. Strategies can then be mixed and tuned without changing the learner.
• I3. Balanced bootstrapping. Weight pseudo-labeled target examples so their combined influence matches that of the whole source set. This keeps a large source corpus from drowning out the domain you actually care about, and it is cheap to add to any self-training loop.
• I4. Be cautious about stacking fixes. Pruning source data on top of up-weighted target labels often hurt. The authors suggest the small labeled target sample is biased, so over-emphasizing it backfires. This is a useful warning when tuning on tiny in-domain sets.
• I5. Use feature priors to curb domain-specific cues. Penalizing large weights on features like capitalization stops the model from depending on signals that disappear in the target domain, such as lowercase ASR transcripts.