Elamur Research
☰
home / papers / 2017 / Data Augmentation for Low-Resource Neural Machine Translation

Data Augmentation for Low-Resource Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-2090.pdf

Summary

Neural machine translation needs large parallel corpora. When data are scarce, words that appear only a few times end up with poorly learned representations. Borrowing the idea of data augmentation from computer vision, the authors introduce translation data augmentation (TDA), which builds extra training pairs by inserting rare words into existing sentences. Forward and backward LSTM language models, trained on large monolingual data, identify positions where a frequent English word can be swapped for a rare one. The aligned German word is then replaced with a translation chosen using fast-align lexical probabilities and a German LM. The new sentences must read fluently but need not keep the original meaning. In a simulated low-resource setting (10% of WMT15 English–German), TDA outperforms both the baseline and back-translation (Sennrich et al., 2016a) in both directions on newstest 2014–2016. The model also produces many more correct rare words. The study shows that cheap, targeted synthetic data can substantially improve translation when parallel text is limited.

Key points

  • Setup: The low-resource baseline uses 371K sentence pairs, compared with 3.9M in the full data. The NMT model is a 4-layer attention encoder-decoder (Luong et al., 2015) with a 30K vocabulary and BPE with 30K merges.
  • Language models: Two-layer LSTM LMs were trained on 3.5B English and 0.9B German monolingual tokens. Rare words are those with fewer than R=100 occurrences, with K=1000 candidate substitutions and at most N=500 augmentations per word.
  • Results: The best variant (TDA r≥1, about 6M pairs) gains up to +2.9 BLEU over the baseline (En-De newstest2016: 13.9 vs 11.0). It gains up to +3.2 BLEU over back-translation. Simple oversampling to the same data size gives smaller gains.
  • Analysis: TDA output contains about three times as many reference rare words as baseline output. Source-side attention to rare words rises by 8.8% on average.
  • Noise: Some augmented sentences are nonsensical or ungrammatical, but this hurts NMT only slightly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cited by

168 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 158 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-2090

@inproceedings{fadaee2017data,
  title = {{Data Augmentation for Low-Resource Neural Machine Translation}},
  author = {Fadaee, Marzieh and Bisazza, Arianna and Monz, Christof},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-2090},
  url = {https://doi.org/10.18653/v1/p17-2090}
}

References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, Curran Associates, Inc., pages 1097–1105. htt… find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations (ICLR). find our card
  3. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi,… find our card
  4. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 1412– 1421. http… find
  5. Yuval Marton, Chris Callison-Burch, and Philip Resnik. 2009. Improved statistical machine translation using monolingually-derived paraphrases. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Singapore, pages 381–39… find
  6. Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Return of the devil in the details: Delving deep into convolutional nets. In Proceedings of the British Machine Vision Conference. BMVA Press. https://doi.org/http://dx.doi.org/10.5244/C.28.6. — [doi:10.5244/c.28.6](https://doi.org/10.5244/c.28.6) find
  7. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pages… find our card
  8. Kyunghyun Cho, B van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST-8), 2014. find our card
  9. Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015. A neural network model for low-resource universal dependency parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 339–348. http… find
  10. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany,… find
  11. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Lin… find
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8):1735–1780. find
  13. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages… find
  14. Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning. PMLR, Lille, France, volume 37 of Proceedings of Machine Learning… find
  15. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, Curran Associates, Inc., pages 3104– 3112. http://pape… find
  16. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 1568–1575. https:/… find