Elamur Research⌕
☰
home / papers / 2019 / Soft Contextual Data Augmentation for Neural Machine Translation

Soft Contextual Data Augmentation for Neural Machine Translation

Paper · 2019 · Fei Gao, Jinhua Zhu, Lijun Wu et al.

Summary

Data augmentation is routine in computer vision but uncommon in neural machine translation. Simple text perturbations such as swapping, dropping or blanking words often distort meaning. Contextual replacement with language-model samples keeps meaning better, but it covers only a small part of the possible variations unless it is repeated many times. Gao et al. propose an alternative. Instead of substituting a discrete token, a randomly selected word, chosen with probability γ, gets a "soft" representation. This is the probability distribution over the vocabulary that a pretrained Transformer language model predicts from the preceding words. The word's input embedding then becomes the weighted average of all vocabulary embeddings under that distribution. Separate source and target language models are trained only on the bilingual training corpus and kept frozen. They feed both the encoder and decoder of a Transformer NMT system, and no extra monolingual data is needed. On three IWSLT2014 tasks and WMT2014 En→De, the method outperformed every baseline augmentation scheme. This suggests that blending many plausible, context-aware substitutes is a cheap and robust way to regularize NMT.

Key points

  • Benchmarks: IWSLT2014 De→En (160k pairs), Es→En (181k) and He→En (151k), each with a joint 10K BPE vocabulary, plus WMT2014 En→De (4.5M pairs, 40k BPE). Quality was measured with case-sensitive BLEU.
  • BLEU gains over the Transformer baseline:
    • De→En: 34.79 → 35.78
    • Es→En: 41.58 → 42.61
    • He→En: 33.64 → 34.91
    • En→De: 28.40 → 29.70 (+1.3)
  • Baselines beaten: Swap, Dropout, Blank, Smooth and LM sample (Kobayashi, 2018). The soft method was the best on every task.
  • Replacement probability: On De→En the best γ was 0.15. The method kept its gains across a wide range of γ values, whereas competing methods could fall below the baseline when γ > 0.15.
  • Code: Available at github.com/teslacool/SCA. The authors plan to extend the method to monolingual data and to other tasks such as summarization.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Microsoft Research AsiaBeijing · China54 works · 59,921 citations🎓University of Science and Technology of ChinaHefei · China26 works · 111,012 citations🏛️Chinese Academy of SciencesBeijing · China68 works · 22,753 citations🎓Sun Yat-sen UniversityGuangzhou · China23 works · 8,980 citations🔬Institute of Computing TechnologyBeijing · China18 works · 3,078 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou and Tie-Yan Liu. 2019. Soft Contextual Data Augmentation for Neural Machine Translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1555

@inproceedings{gao2019soft,
  title = {{Soft Contextual Data Augmentation for Neural Machine Translation}},
  author = {Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou and Tie-Yan Liu},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1555},
  url = {https://doi.org/10.18653/v1/p19-1555}
}

References

21 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041.
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  3. Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1965–1974.
  4. Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. 2018. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501. — [arXiv:1805.09501](https://arxiv.org/abs/1805.09501) find
  5. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. arXiv preprint arXiv:1705.00440.
  6. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. In Proc. of ICML.
  7. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535.
  8. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pages 820–828. find
  9. Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative back-translation for neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 18–24.
  10. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural L…
  11. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  12. Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. arXiv preprint arXiv:1805.06201.
  13. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105. find
  14. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043.
  15. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. find
  16. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
  17. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
  19. Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. Switchout: an efficient data augmentation algorithm for neural machine translation. arXiv preprint arXiv:1808.07512.
  20. Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2018. Conditional bert contextual augmentation. arXiv preprint arXiv:1812.06705. — [arXiv:1812.06705](https://arxiv.org/abs/1812.06705) find
  21. Ziang Xie, Sida I Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng. 2017. Data noising as smoothing in neural network language models. arXiv preprint arXiv:1703.02573. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gao-2019-soft-contextual-data-augmentation-neural/