Elamur Research⌕
☰
home / papers / 2017 / Dynamic Data Selection for Neural Machine Translation

Dynamic Data Selection for Neural Machine Translation

Paper · 2017 · Marlies van der Wees, Arianna Bisazza and Christof Monz

Summary

This paper asks whether the domain-oriented data selection that works well for phrase-based MT (PBMT) also helps neural MT (NMT). The authors rank a large German→English bitext by the bilingual cross-entropy difference method of Axelrod et al. (2011). They then train PBMT and NMT systems on the same top-ranked subsets and evaluate on four test sets: EMEA, Movies, TED and WMT. Static selection turns out much weaker and less reliable for NMT, likely because NMT suffers on small data and has no separate large target-side LM to compensate. To address this, they propose dynamic data selection, which changes the training subset from epoch to epoch without touching the NMT architecture. Two variants are tested: relevance-weighted sampling and "gradual fine-tuning", which repeatedly shrinks the data to its most relevant part. Gradual fine-tuning works best. It gains up to +2.6 BLEU over static selection and up to +3.1 BLEU over a full-bitext baseline. It also comes close to in-domain fine-tuning in about 20% of the training time, without needing parallel in-domain data.

Key points

  • Sentence pairs are scored by CED, a summed source- and target-side cross-entropy difference between in-domain and general-domain LMs. The LM vocabulary is restricted to words seen at least twice in the in-domain corpus.
  • The same subsets are used for both paradigms. The PBMT system is Moses-like, tuned per test set with PRO, with an interpolated Kneser-Ney LM on 480M tokens.
  • The authors also try replacing n-gram LMs with four-layer LSTM LMs (1,024 embedding/hidden size, trained for 30 epochs) for ranking.
  • Gradual fine-tuning follows the schedule n(i) = α·|G|·β^⌊(i−1)/η⌋. For example, it starts from the full bitext and keeps the top 60% every two epochs over 16 epochs, costing roughly 30% of full training.
  • Ablation on learning-rate decay:
    • Decaying the rate after a fixed number of updates instead of per epoch costs 1.1–2.0 BLEU.
    • Keeping the rate fixed leaves BLEU roughly unchanged or slightly better.
  • Code is released at github.com/marliesvanderwees/dds-nmt.

Interesting ideas

  • I1. Shrink the data gradually instead of cutting it once. Begin with all data and progressively narrow toward the most relevant slice. This keeps early vocabulary coverage while still specializing the model. It could suit any low-resource or domain-shifted training, such as Ukrainian-domain NLP models.
  • I2. Sample by relevance score per epoch. Turn ranking scores into normalized sampling weights and draw a fresh subset each epoch. Low-ranked examples still appear occasionally, and the sampling plugs into any epoch-based trainer.
  • I3. Treat the ranking criterion as pluggable. The epoch-wise schedule is separate from how data are ranked. One could rank by redundancy, coverage or diversity instead of domain similarity, which makes the method reusable for data-curation pipelines beyond MT.
  • I4. Compare paradigms on identical subsets. Training PBMT and NMT on exactly the same selections, with random selections run three times as controls, isolates the effect of selection itself. This is a clean evaluation design for any data-filtering study.
  • I5. Check for hidden optimizer effects. Shrinking epochs implicitly speeds up learning-rate decay, and the authors ablate this explicitly. It is a useful reminder that a data schedule may change optimization dynamics as well as data content.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands89 works · 81,337 citations🎓Leiden UniversityLeiden · Netherlands16 works · 936 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Marlies van der Wees, Arianna Bisazza and Christof Monz. 2017. Dynamic Data Selection for Neural Machine Translation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1147

@inproceedings{wees2017dynamic,
  title = {{Dynamic Data Selection for Neural Machine Translation}},
  author = {Marlies {van der Wees}, Arianna Bisazza and Christof Monz},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1147},
  url = {https://doi.org/10.18653/v1/d17-1147}
}

References

42 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355–362.
  2. Amittai Axelrod, Philip Resnik, Xiaodong He, and Mari Ostendorf. 2015. Data selection with fewer words. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 58–65. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  4. Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus phrase-based machine translation quality: a case study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 257–267.
  5. Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. 2016. Findings of the 2016 conference on machine translation (WMT16). In Proceedings of the first conference on machine trans…
  6. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. Wit 3 : Web inventory of transcribed and translated talks. In Proceedings of the 16 th Conference of the European Association for Machine Translation (EAMT), pages 261–268. find
  7. Boxing Chen and Fei Huang. 2016. Semi-supervised convolutional networks for translation adaptation with tiny amount of in-domain data. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), pages 314–323. find
  8. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langua… find
  9. Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017. An empirical comparison of simple domain adaptation methods for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. find
  10. Josep Crego, Jungi Kim, Guillaume Klein, Anabel Rebollo, Kathy Yang, Jean Senellart, Egor Akhanov, Patrice Brunelle, Aurelien Coquard, Yongchao Deng, et al. 2016. SYSTRAN’s pure neural machine translation systems. arXiv preprint arXiv:1610.05540. — [arXiv:1610.05540](https://arxiv.org/abs/1610.05540) find
  11. Kevin Duh, Graham Neubig, Katsuhito Sudoh, and Hajime Tsukada. 2013. Adaptation data selection using neural language models: Experiments in machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 678–683. find
  12. Matthias Eck, Stephan Vogel, and Alex Waibel. 2005. Low cost portability for statistical machine translation based on n-gram frequency and TF-IDF. In Proceedings of the 2005 International Workshop on Spoken Language Translation, pages 61–67. find
  13. Sauleh Eetemadi, William Lewis, Kristina Toutanova, and Hayder Radha. 2015. Survey of data-selection methods in statistical machine translation. Machine Translation, 29(3-4):189–223. find
  14. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.
  15. M. Amin Farajian, Marco Turchi, Matteo Negri, Nicola Bertoldi, and Marcello Federico. 2017. Neural vs. phrase-based machine translation in a multi-domain scenario. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 280–284. find
  16. Markus Freitag and Yaser Al-Onaizan. 2016. Fast domain adaptation for neural machine translation. arXiv preprint arXiv:1612.06897. find
  17. Jianfeng Gao, Joshua Goodman, Mingjing Li, and Kai-Fu Lee. 2002. Toward a unified approach to statistical language modeling for chinese. ACM Transactions on Asian Language Information Processing (TALIP), 1(1):3–33. find
  18. Guillem Gascó, Martha-Alicia Rocha, Germán Sanchis-Trilles, Jesús Andrés-Ferrer, and Francisco Casacuberta. 2012. Does more data always yield better translations? In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 152–161. find
  19. Mark Hopkins and Jonathan May. 2011. Tuning as ranking. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1352–1362.
  20. Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. 2016. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099. find
  21. Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327.
  22. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc… find
  23. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. arXiv preprint arXiv:1706.03872. find
  24. William D. Lewis and Sauleh Eetemadi. 2013. Dramatically reducing training data size through vocabulary saturation. In Proceedings of the 8th Workshop on Statistical Machine Translation, pages 281–291. find
  25. Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016).
  26. Minh-Thang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spoken language domains. In Proceedings of the 12th International Workshop on Spoken Language Translation, pages 76–79. find
  27. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421.
  28. Minh-Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the rare word problem in neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lang…
  29. Shachar Mirkin and Laurent Besacier. 2014. Data selection for compact adapted SMT models. In Proceedings of the 11th Conference of the Association for Machine Translation in the Americas, pages 301–314. find
  30. Robert C. Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Papers, pages 220–224.
  31. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318.
  32. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 86–96. find
  33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725.
  34. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  35. Jörg Tiedemann. 2009. News from OPUS-a collection of multilingual parallel corpora with tools and interfaces. In Recent advances in natural language processing, volume 5, pages 237–248. find
  36. Antonio Toral and Vı́ctor M. Sánchez-Cartagena. 2017. A multifaceted evaluation of neural versus phrase-based machine translation for 9 language directions. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1… find
  37. Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2016. Measuring the effect of conversational aspects on machine translation quality. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics, pages 2571– 2581. find
  38. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  39. Keiji Yasuda, Ruiqiang Zhang, Hirofumi Yamamoto, and Eiichiro Sumit. 2008. Method of selecting training data to build a compact and efficient translation model. In International Joint Conference on Natural Language Processing (IJCNLP), pages 655–660. find
  40. Dakun Zhang, Jungi Kim, Joseph Crego, and Jean Senellart. 2016. Boosting neural machine translation. arXiv preprint arXiv:1612.06138. find
  41. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. arXiv preprint arXiv:1604.02201. find
  42. Sebastian Ruder and Barbara Plank. 2017. Learning to select data for transfer learning with bayesian optimization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wees-2017-dynamic-data-selection-neural-machine/