Elamur Research⌕
☰
home / papers / 2018 / SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation

SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation

Summary

Data augmentation for neural machine translation (NMT) is difficult because small edits to a sentence can change its meaning or make it inconsistent with its translation. The authors treat the choice of an augmentation policy as an optimization problem. The objective rewards two things: augmented pairs that stay close to observed training pairs (smoothness) and a high-entropy augmentation distribution (diversity). The resulting maximum-entropy problem has a closed-form solution. Word dropout and RAML both turn out to be special cases of it. The framework also motivates a new method, SwitchOut. SwitchOut independently swaps random tokens in both the source and the target sentence for words drawn uniformly from the matching vocabulary. The number of swaps is sampled from a temperature-controlled distribution tied to negative Hamming distance. On WMT 15 English-German, IWSLT 2016 German-English and IWSLT 2015 English-Vietnamese, using a Transformer, SwitchOut adds roughly 0.5 BLEU. It matches or beats word dropout and RAML, and it combines well with RAML and back-translation. The method needs only a few lines of code, which makes it a cheap and widely applicable regularizer for NMT.

Key points

  • Generic solution: the optimal augmentation distribution is proportional to exp(similarity / τ).
    • RAML corresponds to target-only reward-based similarity.
    • Word dropout corresponds to a particular similarity function.
    • SwitchOut factorizes similarity into separate source and target terms, with temperatures τ_x and τ_y.
  • Main results (Transformer, median BLEU):
    • WMT 15 en-de: 21.73 → 22.78 with SwitchOut, and 23.13 with RAML+SwitchOut.
    • IWSLT 16 de-en: up to 30.98 with RAML+SwitchOut.
    • IWSLT 15 en-vi: up to 29.09 with RAML+SwitchOut.
    • Word dropout (p = 0.1) hurt en-de, scoring 20.63.
  • Evaluation protocol: runs with SwitchOut were repeated 9 times and runs without it 4 times. Paired bootstrap tests showed significant gains over the best baseline in 4 of 6 settings.
  • Back-translation on en-de:
    • Back-translation alone gave only 21.82.
    • Back-translation + RAML + SwitchOut reached 23.76, so the two approaches are complementary.
  • Where it helps:
    • Gains increase for test sentences that are farther, by WER, from their nearest training neighbor.
    • The best dev scores always used a non-zero source-side temperature τ_x.
    • TensorFlow and PyTorch code is provided in the appendix.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

93 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 86 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1100

@inproceedings{wang2018switchout,
  title = {{SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation}},
  author = {Wang, Xinyi and Pham, Hieu and Dai, Zihang and Neubig, Graham},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1100},
  url = {https://doi.org/10.18653/v1/d18-1100}
}

References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, and more authors. 2016. Deep speech 2: End-to-end speech recognition in english and mandarin. In ICML. find our card
  2. Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. 2016. Reward augmented maximum likelihood for neural structured prediction. In NIPS. find
  3. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In ICLR. find
  4. Alberto Poncelas, Dimitar Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018. Investigating backtranslation in neural machine translation. Arxiv, 1804.06189. find our card
  5. Adam L Berger, Vincent J Della Pietra, and Stephen A Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational linguistics, 22(1):39–71. find our card
  6. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In ICLR. find our card
  7. Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. 2009. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542–542. find
  8. Jonathan Clark, Chris Dyer, Alon Lavie, and Noah Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In ACL. find
  9. Terrance DeVries and Graham W. Taylor. 2017. Improved regularization of convolutional neural networks with cutout. Arxiv, 1708.04552. find
  10. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In ACL. find our card
  11. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In NIPS. find our card
  12. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In ICLR. find our card
  13. Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2016. Densely connected convolutional networks. In CVPR. find
  14. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  15. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In NIPS. find
  16. Minh-Thang Luong and Christopher D. Manning. 2015. Stanford neural machine translation systems for spoken language domain. In IWLST. find
  17. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP. find
  18. Xuezhe Ma, Pengcheng Yin, Jingzhou Liu, Graham Neubig, and Eduard Hovy. 2017. Softmax q-distribution estimation for structured prediction: A theoretical interpretation for raml. Arxiv, 1705.07136. find
  19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for wmt 16. In WMT. find our card
  20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Improving neural machine translation models with monolingual data. In ACL. find
  21. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2014. Going deeper with convolutions. In CVPR. find
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. find
  23. Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. In BMVC. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2018-switchout-efficient-data-augmentation-algorithm/