Elamur Research⌕
☰
home / papers / 2010 / Discriminative Instance Weighting for Domain Adaptation in Statistical Machine Translation

Discriminative Instance Weighting for Domain Adaptation in Statistical Machine Translation

Summary

The paper tackles domain adaptation for phrase-based SMT. A large out-of-domain parallel corpus (OUT) is used to help a smaller in-domain one (IN), which is hard when the two differ a lot. The authors do not decide at the level of whole sentences or corpora. Instead, they assign a weight to every OUT phrase pair. The weight is a logistic function of features that estimate how similar the pair is to IN and how much it looks like general language, including the output of an SVM. The weighted counts are MAP-smoothed and then mixed linearly with the IN phrase table. All mixing, smoothing and feature weights are fitted together by maximizing likelihood on a small IN dev set with L-BFGS. The method was tested in two settings:

  • Europarl helping the EMEA medical domain (English↔French).
  • Non-news NIST09 data helping Chinese→English news.

In both, it beat a wide set of baselines. This shows that fine-grained, cheaply trained weighting can recover the useful part of distant background data.

Key points

  • Two experimental settings: EMEA/EP for fren and enfr, and NIST Chinese→English evaluated on NIST06 and NIST08. Results are reported in BLEU.
  • Best instance-weighting model:
    • beats the same mixture model without instance weights by 0.6–1.8 BLEU;
    • is over 2 BLEU above the best non-adapted baseline.
  • Baselines include:
    • IN+OUT concatenation;
    • log-linear combination of separate models tuned with MERT;
    • linear and MAP (Dirichlet-prior) mixtures;
    • sentence selection by IN-LM perplexity.
  • Log-linear combination trails linear/MAP mixtures by a wide margin on EMEA, yet performs comparably on NIST, whose domains are wider and more similar to each other. Linear and MAP mixtures show no large systematic gap.
  • Keeping original OUT frequencies and smoothing the weighted counts with a uniform prior (non-zero γ) clearly outperforms the unsmoothed and "flattened" variants.

Interesting ideas

  • I1. Separate "general" from "similar". The paper does not only look for OUT data resembling the target domain. It also scores whether an example is general language, which matters when domains are far apart. This is a useful lens for any transfer setup with distant source data.
  • I2. Use the overlap as positive labels. An SVM is trained with phrase pairs found in both IN and OUT as positive examples. This yields a "generality" signal without manual annotation and could be reused to label transferable items in other cross-domain tasks.
  • I3. Weight at the finest natural unit. Weighting phrase pairs rather than sentences lets general fragments inside domain-specific sentences still contribute. Similar unit-level weighting could suit n-gram LMs or lexicons.
  • I4. Tune mixtures by dev-set likelihood instead of MERT/BLEU. Fitting linear mixture weights to a joint phrase-pair distribution taken from the dev set is cheap and stable. It only needs evaluation over that small support, so it runs much faster than metric-driven tuning while still optimizing many parameters at once.
  • I5. Perplexity-ranked selection as a simple baseline. Ranking OUT sentence pairs by IN-LM perplexity of the target side, with the cutoff tuned on dev BLEU, is an easy and well-motivated data-selection baseline for any corpus-filtering experiment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

102 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 99 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{foster2010discriminative,
  title = {{Discriminative Instance Weighting for Domain Adaptation in Statistical Machine Translation}},
  author = {Foster, George and Goutte, Cyril and Kuhn, Roland},
  booktitle = {Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://aclanthology.org/D10-1044/}
}

References

29 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. ACL. 2007. Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL), Prague, Czech Republic, June. find
  2. Michel Bacchiani, Brian Roark, and Murat Saraclar. 2004. Language model adaptation with MAP estimation and the perceptron algorithm. In NAACL04 (NAA, 2004). find
  3. Nicola Bertoldi and Marcello Federico. 2009. Domain adaptation for statistical machine translation with monolingual resources. In WMT09 (WMT, 2009). find our card
  4. Jorge Civera and Alfons Juan. 2007. Domain adaptation in Statistical Machine Translation with mixture modelling. In WMT07 (WMT, 2007). find
  5. Hal Daumé III and Daniel Marcu. 2006. Domain Adaptation for Statistical Classifiers. Journal of Artificial Intelligence Research, 26:101–126. find
  6. Hal Daumé III. 2007. Frustratingly Easy Domain Adaptation. In ACL-07 (ACL, 2007). find our card
  7. Andrew Finch and Eiichiro Sumita. 2008. Dynamic model interpolation for statistical machine translation. In Proceedings of the ACL Workshop on Statistical Machine Translation, Columbus, June. WMT. find
  8. Jenny Rose Finkel and Christopher D. Manning. 2009. Hierarchical Bayesian domain adaptation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Boulder, June. NAACL. find
  9. George Foster and Roland Kuhn. 2007. Mixture-model adaptation for SMT. In WMT07 (WMT, 2007). find our card
  10. George Foster and Roland Kuhn. 2009. Stabilizing minimum error rate training. In WMT09 (WMT, 2009). find
  11. Almut Silja Hildebrand, Matthias Eck, Stephan Vogel, and Alex Waibel. 2005. Adaptation of the translation model for statistical machine translation based on information retrieval. In Proceedings of the 10th EAMT Conference, Budapest, May. find our card
  12. Jing Jiang and ChengXiang Zhai. 2007. Instance Weighting for Domain Adaptation in NLP. In ACL-07 (ACL, 2007). find our card
  13. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 1995, pages 181–184, Detroit, Michigan. IEEE. find
  14. Philipp Koehn and Josh Schroeder. 2007. Experiments in domain adaptation for statistical machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 224–227, Prague, Czech Republic, June. Association for Computational Linguistics. find our card
  15. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 127–133, Edmonton, May. NAACL. find
  16. D. C. Liu and J. Nocedal. 1989. On the limited memory method for large scale optimization. Mathematical Programming B, 45(3):503–528. find
  17. Yajuan Lü, Jin Huang, and Qun Liu. 2007. Improving Statistical Machine Translation Performance by Training Data Selection and Optimization. In Proceedings of the 2007 Conference on Empirical Methods in Natural Language Processing (EMNLP), Prague, Czech Republic. find
  18. Spyros Matsoukas, Antti-Veikko I. Rosti, and Bing Zhang. 2009. Discriminative corpus weight estimation for machine translation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), Singapore. find
  19. NAACL. 2004. Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Boston, May. find
  20. Franz Josef Och. 2003. Minimum error rate training for statistical machine translation. In Proceedings of the 41th Annual Meeting of the Association for Computational Linguistics (ACL), Sapporo, July. ACL. find
  21. Holger Schwenk and Jean Senellart. 2009. Translation model adaptation for an arabic/french news translation system by lightly-supervised training. In Proceedings of MT Summit XII, Ottawa, Canada, September. International Association for Machine Translation. find
  22. Yik-Cheung Tam, Ian Lane, and Tanja Schultz. 2007. Bilingual-LSA Based LM Adaptation for Spoken Language Translation. In ACL-07 (ACL, 2007). find
  23. Jorg Tiedemann. 2009. News from opus - a collection of multilingual parallel corpora with tools and interfaces. In N. Nicolov, K. Bontcheva, G. Angelova, and R. Mitkov, editors, Recent Advances in Natural Language Processing, volume V, pages 237–248. John Benjamins, Amsterdam/Philadelphia. find
  24. Nicola Ueffing, Gholamreza Haffari, and Anoop Sarkar. 2007. Transductive learning for statistical machine translation. In ACL-07 (ACL, 2007). find
  25. WMT. 2007. Proceedings of the ACL Workshop on Statistical Machine Translation, Prague, June. find
  26. WMT. 2009. Proceedings of the 4th Workshop on Statistical Machine Translation, Athens, March. find
  27. Hua Wu, Haifeng Wang, and Zhanyi Liu. 2005. Alignment model adaptation for domain-specific word alignment. In Proceedings of the 43th Annual Meeting of the Association for Computational Linguistics (ACL), Ann Arbor, Michigan, July. ACL. find
  28. Jia Xu, Yonggang Deng, Yuqing Gao, and Hermann Ney. 2007. Domain dependent statistical machine translation. In MT Summit XI, Copenhagen, September. find
  29. Bing Zhao, Matthias Eck, and Stephan Vogel. 2004. Language model adaptation for statistical machine translation with structured query models. In Proceedings of the International Conference on Computational Linguistics (COLING) 2004, Geneva, August. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/foster-2010-discriminative-instance-weighting-domain-adaptation/