Elamur Research⌕
☰
home / papers / 2006 / Domain Adaptation with Structural Correspondence Learning

Domain Adaptation with Structural Correspondence Learning

Summary

The paper addresses domain adaptation for discriminative NLP models: a model trained on a well-annotated source domain often performs poorly on a target domain that has a different vocabulary and little or no labeled data. The authors propose structural correspondence learning (SCL). SCL uses unlabeled text from both domains to learn a shared low-dimensional feature representation. They first choose pivot features, which are frequent in both domains and behave alike in both. They then train linear classifiers that predict each pivot from the other features. An SVD of the stacked classifier weights gives a projection that links domain-specific features related to the same pivots. The projected features are added to the original ones when training a source-domain classifier. In the experiments, a part-of-speech tagger trained on Wall Street Journal text is adapted to MEDLINE abstracts. With no labeled target data, SCL beats both a supervised tagger and a semi-supervised ASO baseline. It also combines with target labeled data through the classifier combination of Florian et al. (2004), and the better tags raise parsing accuracy on biomedical text. The method matters because it needs no labeled data in the new domain.

Key points

  • The setting is adapting a PoS tagger from WSJ (financial news) to MEDLINE (biomedical abstracts). Test sets come from PennBioIE annotations of MEDLINE.
  • Pivot features are left, right and middle tokens for words that are frequent in both domains, giving many thousands of pivots.
  • Pivot predictors are trained with the modified Huber loss. Features that trivially reveal the pivot are excluded from its predictor.
  • Implementation choices:
    • The projection dimension is h = 25; values from 20 to 100 changed little.
    • Only positive weights enter the SVD.
    • Block SVDs are computed separately for each of the 58 Ratnaparkhi (1996) feature types, giving 1450 projection features.
  • SCL beats supervised and ASO taggers without target labels. Top-k ASO, a co-training-like variant, failed to beat the supervised baseline. The SCL tagger also improves downstream parsing on MEDLINE.

Interesting ideas

  • I1. Pivots as a bridge between vocabularies: Context features that are frequent in both domains act as anchors. Unseen target words, such as biomedical nouns, can then be aligned with familiar source words that share those contexts. This could help adapt Ukrainian taggers from news to legal, medical or trade texts.
  • I2. Self-supervised auxiliary tasks from unlabeled text: Predicting whether a pivot is present needs no annotation. Masking the pivot's own feature forces the predictor to learn real correlations instead of trivial ones. The same trick suits any task where cheap proxy targets can be built from raw data.
  • I3. Augment, don't replace: The projected features are appended to the original ones. The representation can therefore be added to an existing classifier, or combined with other adaptation methods, without discarding what already works.
  • I4. Rescaling learned features: A single scaling factor, tuned on source held-out data, gives the dense projected features enough weight under regularization. The authors found this essential to reproduce the ASO results. It is a useful check whenever dense learned features are mixed with sparse binary ones.
  • I5. Extrinsic evaluation through a pipeline: The authors measure gains in tagging and also in downstream parsing. This shows whether an improvement in an upstream component actually carries through to the end task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Part-of-speech tagging

Cited by

265 papers of our library cite it (of the 140884 whose bibliographies we read); 20 of them have a page here, 245 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{blitzer2006domain,
  title = {{Domain Adaptation with Structural Correspondence Learning}},
  author = {Blitzer, John and McDonald, Ryan and Pereira, Fernando},
  booktitle = {Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing},
  year = {2006},
  doi = {10.3115/1610075.1610094},
  url = {https://doi.org/10.3115/1610075.1610094}
}

References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Ando and T. Zhang. 2005a. A framework for learning predictive structures from multiple tasks and unlabeled data. JMLR, 6:1817–1853. find
  2. R. Ando and T. Zhang. 2005b. A high-performance semi-supervised learning method for text chunking. In ACL. find
  3. R. Ando. 2004. Exploiting unannotated corpora for tagging and chunking. In ACL. Short paper. find
  4. D. Blei, A. Ng, and M. Jordan. 2003. Latent dirichlet allocation. JMLR, 3:993–1022. find our card
  5. A. Blum and T. Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Workshop on Computational Learning Theory. find our card
  6. P. Brown, V. Della Pietra, P. deSouza, J. Lai, and R. Mercer. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467–479. find
  7. C. Chelba and A. Acero. 2004. Adaptation of maximum entropy capitalizer: Little data can help a lot. In EMNLP. find
  8. K. Crammer, Dekel O, J. Keshet, S. Shalev-Shwartz, and Y. Singer. 2006. Online passive-aggressive algorithms. JMLR, 7:551–585. find
  9. H. Daum´e III and D. Marcu. 2006. Domain adaptation for statistical classifiers. JAIR. find
  10. R. Florian, H. Hassan, A.Ittycheriah, H. Jing, N. Kambhatla, X. Luo, N. Nicolov, and S. Roukos. 2004. A statistical model for multilingual entity detection and tracking. In of HLT-NAACL. find
  11. L. Gillick and S. Cox. 1989. Some statistical issues in the comparison of speech recognition algorithms. In ICASSP. find
  12. R. Kuhn, P. Nguyen, J.C. Junqua, L. Goldwasser, N. Niedzielski, S. Fincke, K. Field, and M. Con-tolini. 1998. Eigenvoices for speaker adaptation. In ICSLP. find
  13. M. Lease and E. Charniak. 2005. Parsing biomedical literature. In IJCNLP. find
  14. M. Marcus, B. Santorini, and M. Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  15. R. McDonald, K. Crammer, and F. Pereira. 2005a. Flexible text segmentation with structured multilabel classification. In HLT-EMNLP. find
  16. R. McDonald, K. Crammer, and F. Pereira. 2005b. Online large-margin training of dependency parsers. In ACL. find our card
  17. S. Miller, J. Guinness, and A. Zamanian. 2004. Name tagging with word clusters and discriminative training. In HLT-NAACL. find our card
  18. F. Och. 2003. Minimum error rate training in statistical machine translation. In Proc. of ACL. find
  19. PennBioIE. 2005. Mining The Bibliome Project. http://bioie.ldc.upenn.edu/. ↗
  20. F. Pereira, N. Tishby, and L. Lee. 1993. Distributional clustering of english words. In ACL. find our card
  21. A. Ratnaparkhi. 1996. A maximum entropy model for part-of-speech tagging. In EMNLP. find our card
  22. B. Roark and M. Bacchiani. 2003. Supervised and unsupervised PCFG adaptation to novel domains. In HLT-NAACL. find
  23. B. Roark, M. Saraclar, M. Collins, and M. Johnson. 2004. Discriminative language modeling with conditional random fields and the perceptron algorithm. In ACL. find
  24. F. Sha and F. Pereira. 2003. Shallow parsing with conditional random fields. In HLT-NAACL. find
  25. K. Toutanova, D. Klein, C. D. Manning, and Y. Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In NAACL. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/blitzer-2006-domain-adaptation-structural-correspondence-learning/