Elamur Research⌕
☰
home / papers / 2017 / Instance Weighting for Neural Machine Translation Domain Adaptation

Instance Weighting for Neural Machine Translation Domain Adaptation

Paper · 2017 · Rui Wang, Masao Utiyama, Lemao Liu et al.

Summary

The paper asks how to make neural machine translation benefit from a large general-domain corpus when the target domain (TED talks) has little parallel data. Instance weighting was a standard adaptation tool for phrase-based SMT, but because NMT models are non-linear, the technique cannot simply be carried over. The authors instead build the weights into the NMT training objective, using two variants:

  • Sentence weighting scores each out-of-domain pair by bilingual cross-entropy difference.
  • Domain weighting raises the importance of in-domain data. Its practical form, batch weighting, fixes the share of in-domain sentences in every mini-batch.

They also add a dynamic schedule that gradually increases this share. Experiments in Nematus adapt WMT data to IWSLT English-German and English-French. Simply mixing in out-of-domain data actually hurts NMT. The proposed methods gain up to 2.7–6.7 BLEU over in-domain-only training and beat the strongest baselines by 1.6–3.6 BLEU. These baselines include ensembling, oversampling, Kobus et al.'s domain control and Axelrod et al.'s data selection. The work shows that a cheap change to batch composition can outperform more elaborate adaptation schemes.

Key points

  • Data:
    • In-domain sets are IWSLT 2015 EN-DE (207.1K sentences) and IWSLT 2014 EN-FR (178.1K).
    • Out-of-domain sets are WMT 2014 EN-DE (4.5M) and WMT 2015 EN-FR (17.8M).
    • Development and test sets are TED tst/dev sets.
  • Sentence weights:
    • The score is the sum of out-minus-in language-model cross-entropy on the source and target sides.
    • Scores are min-max normalised to [0, 1].
    • In-domain sentences keep weight 1.
  • Batch weighting:
    • Each mini-batch of 80 sentences normally holds only about 4 in-domain sentences (a natural ratio of roughly 1:20).
    • This is changed to, for example, 40 in-domain and 40 out-of-domain sentences.
    • The mixing is done with two separately shuffled data iterators.
  • Dynamic tuning:
    • Training starts with 0% in-domain data in each batch.
    • The in-domain share grows by 10% whenever development cost has not improved over ten checks, with a check every 1K batches.
    • This beats choosing the best fixed ratio by 0.6–2.4 BLEU.
    • The best fixed ratio was about 50% and the dynamic peak about 60% on EN-DE dev.
  • Results and setup:
    • Best scores were 28.58 BLEU on EN-DE tst2013 and 41.90 on EN-FR tst2011, both with batch weighting plus dynamic tuning.
    • Training used Nematus defaults for 500K batches with ADADELTA, taking 7–10 days on one P100 GPU.

Interesting ideas

  • I1. Control batch composition instead of reweighting the loss. Fixing the in-domain share of every mini-batch is trivial to code with two data iterators, yet it was the strongest method here. It could help any task where a small target set is overwhelmed by a large auxiliary corpus, such as low-resource Ukrainian NLP mixed with Russian or English data.
  • I2. Curriculum driven by dev-set stagnation. Raising the in-domain share whenever development loss plateaus replaces training many fixed-ratio systems with a single run. The same rule can schedule data mixtures in other multi-source training setups.
  • I3. Fine-tuning as a special case. The authors frame standard further training as the extreme schedule that jumps from 0% to 100% in-domain data. Seeing it this way suggests smoother intermediate mixtures that keep some general data in every batch, which may reduce overfitting to the small domain. The two approaches can also be stacked.
  • I4. Distinguish oversampling from per-batch balancing. Duplicating in-domain data to a 1:1 corpus ratio is not the same as forcing 1:1 inside each batch, since the two only coincide when a batch covers the whole corpus. The paper's results show the per-batch version works better, which makes it worth testing in imbalanced-data experiments generally.
  • I5. Reuse cross-entropy difference scores as soft weights. Moore-Lewis/Axelrod-style scores can be normalised and used as continuous loss weights instead of a hard selection cutoff. This keeps all the data while emphasising pseudo-in-domain sentences, and it applies to corpus building for any domain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China76 works · 27,918 citations🏢TencentShenzhen · China66 works · 17,945 citations🔬National Institute of Information and Communications TechnologyTokyo · Japan13 works · 1,970 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen and Eiichiro Sumita. 2017. Instance Weighting for Neural Machine Translation Domain Adaptation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1155

@inproceedings{wang2017instance,
  title = {{Instance Weighting for Neural Machine Translation Domain Adaptation}},
  author = {Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen and Eiichiro Sumita},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1155},
  url = {https://doi.org/10.18653/v1/d17-1155}
}

References

32 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355–362, Edinburgh, Scotland, U.K.
  2. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2015. find
  3. The IWSLT 2015 evaluation campaign. In Proceedings of the International Workshop on Spoken Language Translation, pages 2–14, Da Nang, Vietnam. *Table 4: Further training (Luong and Manning, 2015) is the baseline for significance test.* Table 4 shows that batch weighting worked synergistically with L… find
  4. Boxing Chen, Roland Kuhn, George Foster, Colin Cherry, and Fei Huang. 2016. Bilingual methods for adaptive training data selection for machine translation. In The Twelfth Conference of The Association for Machine Translation in the Americas, pages 93–106, Austin, Texas. find
  5. Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017. An empirical comparison of simple domain adaptation methods for neural machine translation. arXiv preprint arXiv:1701.03214. — [arXiv:1701.03214](https://arxiv.org/abs/1701.03214) find
  6. Kevin Duh, Graham Neubig, Katsuhito Sudoh, and Hajime Tsukada. 2013. Adaptation data selection using neural language models: Experiments in machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 678–683, Sofia,… find
  7. Nadir Durrani, Hassan Sajjad, Shafiq Joty, and Ahmed Abdelali. 2016. A deep fusion model for domain adaptation in phrase-based MT. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3177–3187, Osaka, Japan. find
  8. Nadir Durrani, Hassan Sajjad, Shafiq Joty, Ahmed Abdelali, and Stephan Vogel. 2015. Using joint models for domain adaptation in statistical machine translation. In Proceedings of MT Summit XV, pages 117–130, Miami, FL, USA. find
  9. George Foster, Cyril Goutte, and Roland Kuhn. 2010. Discriminative instance weighting for domain adaptation in statistical machine translation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 451–459, Cambridge, MA.
  10. Cuong Hoang and Khalil Sima’an. 2014a. Latent domain phrase-based models for adaptation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 566–576, Doha, Qatar. find
  11. Cuong Hoang and Khalil Sima’an. 2014b. Latent domain translation models in mix-of-domains haystack. In Proceedings of the 25th International Conference on Computational Linguistics: Technical Papers, pages 1928–1939, Dublin, Ireland. find
  12. Kenji Imamura and Eiichiro Sumita. 2016. Multi-domain adaptation for statistical machine translation based on feature augmentation. In Proceedings of the 12th Conference of the Association for Machine Translation in the Americas, Austin, Texas, USA. find
  13. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language P…
  14. Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in NLP. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 264–271, Prague, Czech Republic.
  15. Catherine Kobus, Josep Crego, and Jean Senellart. 2016. Domain control for neural machine translation. arXiv preprint arXiv:1612.06140. — [arXiv:1612.06140](https://arxiv.org/abs/1612.06140) find
  16. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395, Barcelona, Spain.
  17. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc… find
  18. Minh-Thang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spoken language domains. In Proceedings of the International Workshop on Spoken Language Translation, pages 76–79, Da Nang, Vietnam. find
  19. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal.
  20. Spyros Matsoukas, Antti-Veikko I. Rosti, and Bing Zhang. 2009. Discriminative corpus weight estimation for machine translation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 708–717, Singapore.
  21. Robert C Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 220–224, Uppsala, Sweden.
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania.
  23. Kevin L. Priddy and Paul E. Keller. 2005. Artificial Neural Networks: An Introduction (SPIE Tutorial Texts in Optical Engineering, Vol. TT68). SPIE-International Society for Optical Engineering. find
  24. Anthony Rousseau, Fethi Bougares, Paul Deléglise, Holger Schwenk, and Yannick Estève. 2011. Liums systems for the iwslt 2011 speech translation tasks. In International Workshop on Spoken Language Translation, San Francisco, USA. find
  25. Rico Sennrich. 2012. Perplexity minimization for translation model domain adaptation in statistical machine translation. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 539–549, Avignon, France.
  26. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine translation. In Proceedings of the Software Demonstrations of…
  27. Rico Sennrich, Holger Schwenk, and Walid Aransa. 2013. A multi-domain translation model framework for statistical machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 832–840, Sofia, Bulgaria. find
  28. Kashif Shah, Loı̈c Barrault, and Holger Schwenk. 2010. Translation model adaptation by resampling. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 392–399. find
  29. Rui Wang, Andrew Finch, Masao Utiyama, and Eiichro Sumita. 2017. Sentence embedding for neural machine translation domain adaptation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada. find
  30. Rui Wang, Hai Zhao, Bao-Liang Lu, Masao Utiyama, and Eiichiro Sumita. 2016. Connecting phrase based statistical machine translation adaptation. In Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers, pages 3135–3145, Osaka, Japan. find
  31. Matthew D Zeiler. 2012. ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701.
  32. Xinpeng Zhou, Hailong Cao, and Tiejun Zhao. 2015. Domain adaptation for SMT using sentence weight. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data, pages 153–163, Guangzhou, China. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2017-instance-weighting-neural-machine-translation/