Elamur Research⌕
☰
home / papers / 2017 / Copied Monolingual Data Improves Low-Resource Neural Machine Translation

Copied Monolingual Data Improves Low-Resource Neural Machine Translation

Summary

Neural MT needs large amounts of parallel text, and most language pairs have very little. Target-language monolingual text is plentiful, but back-translation, the standard way to use it, depends on a reverse-direction system that is weak when bitext is scarce. Currey, Miceli Barone and Heafield propose a simple alternative. They turn target-side monolingual sentences into pseudo-bitext by using each sentence as its own source. These copied pairs are then shuffled into the real parallel data, optionally alongside back-translated data. The model architecture, the training procedure and the data tags stay unchanged. A shared BPE vocabulary lets one encoder read both languages.

Experiments with Nematus attentional GRU models covered Turkish, Romanian and German paired with English, using WMT data. The copied data raised BLEU by up to 1.2 points over a back-translation baseline for the two low-resource pairs. It left high-resource EN↔DE essentially unchanged. The gains mostly came from named entities and words that should pass through untranslated, which makes the method complementary to back-translation. It is a cheap, architecture-free boost for under-resourced translation.

Key points

  • Data sizes:
    • EN↔TR: about 207k parallel and 415k monolingual sentences.
    • EN↔RO: 608,320 of each.
    • EN↔DE: about 5.85M parallel and 10M monolingual.
    • Monolingual text came from News Crawl 2015/2016. Parallel data came from WMT17 (TR, DE) and WMT16 (RO).
  • Notable BLEU gains:
    • EN→TR: 12.8→14.0 (newstest2016) and 14.2→15.2 (2017).
    • RO→EN: 34.5→35.7.
    • EN→RO: 23.8→24.5.
    • EN↔DE: changes were not significant.
  • Model setup: Nematus with a bidirectional GRU encoder, attention, 1024-dimensional hidden layers, 512-dimensional embeddings, Adam, and a shared BPE model with 90,000 merges.
  • Amount of monolingual data: For EN→TR, monolingual-to-parallel ratios of 1:1, 2:1 and 3:1 were tested, with no oversampling of the bitext. The copied system stayed about 1.0 BLEU ahead of the baseline at every ratio. Adding monolingual data gave 0.2–0.6 BLEU for both systems.
  • Fluency check: KenLM 5-gram models were used to measure perplexity, testing whether the gains reflect more than better fluency.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

88 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 82 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-4715

@inproceedings{currey2017copied,
  title = {{Copied Monolingual Data Improves Low-Resource Neural Machine Translation}},
  author = {Currey, Anna and Barone, Antonio Valerio Miceli and Heafield, Kenneth},
  booktitle = {Proceedings of the Second Conference on Machine Translation},
  year = {2017},
  doi = {10.18653/v1/w17-4715},
  url = {https://doi.org/10.18653/v1/w17-4715}
}

References

13 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations. find our card
  2. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turc… find our card
  3. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700–1709. Association for Computational Linguistics. find
  4. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations. find
  5. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In 4th International Conference on Learning Representations. find our card
  6. Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation. In Proceedings of the 54th Annual Meeting of the ACL, pages 1965–1974. Association for Computational Linguistics. find
  7. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the ACL, pages 311– 318. Association for Computational Linguistics. find our card
  8. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmu… find our card
  9. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation, pages 371–376. Association for Computational Linguistics. find our card
  10. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Improving neural machine translation models with monolingual data. In Proceedings of NAACL-HLT, pages 86–96. Association for Computational Linguistics. find
  11. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016c. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the ACL, pages 1715–1725. Association for Computational Linguistics. find
  12. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27. find
  13. Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1535–1545. Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/currey-2017-copied-monolingual-data-improves-low/