Elamur Research⌕
☰
home / papers / 2017 / A Convolutional Encoder Model for Neural Machine Translation

A Convolutional Encoder Model for Neural Machine Translation

Summary

Standard neural machine translation systems encode the source sentence with bi-directional LSTMs. Because these process words one after another, they are hard to parallelize. Gehring et al. (Facebook AI Research) replace the recurrent encoder with stacked convolutional layers, so every source position can be computed at once. The decoder remains an LSTM with dot-product attention.

The encoder has several key ingredients:

  • position embeddings added to the word embeddings;
  • residual connections, which make deep stacks trainable;
  • two separate convolutional networks: CNN-a produces the representations used to score attention, and CNN-c produces the vectors that are averaged into the decoder's conditional input.

The authors tune the architecture on IWSLT'14 German-English. They then evaluate on WMT'16 English-Romanian, WMT'15 English-German and WMT'14 English-French. The convolutional encoder:

  • comes close to the best published English-Romanian result;
  • beats several recent English-German systems;
  • nearly reaches a much deeper LSTM system on English-French.

It also decodes more than twice as fast on a CPU as a strong BiLSTM baseline, with equal or better BLEU. This shows that recurrence is not required on the source side, and it opens the way to faster, more parallel translation models.

Key points

  • Data: IWSLT'14 De-En (167K sentence pairs after removing sentences over 175 words), WMT'16 En-Ro (2.8M pairs including back-translated data, word-based rather than BPE, tested on newstest2016), and WMT'15 En-De (3.9M pairs).
  • Separate networks matter: a single shared network for attention and conditional input reached only 22.9 BLEU on IWSLT'14, versus 28.5 BLEU with separate single-layer CNN-a and CNN-c.
  • Depth: without residual connections, deeper encoders lose accuracy. A good rule of thumb is to give CNN-a two to three times as many layers as CNN-c.
  • Speed on WMT'15 En-De (one CPU core): Deep Conv. 8/4 translates 231.1 words/s at 23.7 BLEU, compared with 109.9 words/s at 23.6 BLEU for a 2-layer BiLSTM. Deep Conv. 15/5 reaches 24.0 BLEU.
  • Code: the implementation was released in the fairseq repository (facebookresearch/fairseq).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

82 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 69 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1012 · arXiv 1611.02344

@inproceedings{gehring2017convolutional,
  title = {{A Convolutional Encoder Model for Neural Machine Translation}},
  author = {Gehring, Jonas and Auli, Michael and Grangier, David and Dauphin, Yann},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1012},
  eprint = {1611.02344},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p17-1012}
}

References

37 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In Proc. of EMNLP. find our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR. find our card
  3. Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. 2016. Neural Machine Translation in Linear Time. arXiv . find
  4. James Bradbury and Richard Socher. 2016. MetaMind Neural Machine Translation System for WMT 2016. In Proc. of WMT. find
  5. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. Proc. of ICLR . find
  6. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. 2014. Report on the 11th IWSLT evaluation campaign. In Proc. of IWSLT. find our card
  7. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proc.… find
  8. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the Properties of Neural Machine Translation: Encoder-decoder Approaches. In Proc. of SSST. find our card
  9. Andrew Lamb and Michael Xie. 2016. Convolutional Encoders for Neural Machine Translation. https://cs224d.stanford.edu/ reports/LambAndrew.pdf. Accessed: 2010-10-31. ↗
  10. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014b. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proc. of EMNLP. find
  11. Gurvan L’Hostis, David Grangier, and Michael Auli. 2016. Vocabulary Selection Strategies for Neural Machine Translation. arXiv preprint arXiv:1610.00072 . — [arXiv:1610.00072](https://arxiv.org/abs/1610.00072) find
  12. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A Character-level Decoder without Explicit Segmentation for Neural Machine Translation. arXiv preprint arXiv:1603.06147 . find our card
  13. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015a. Effective approaches to attention-based neural machine translation. In Proc. of EMNLP. find
  14. Ronan Collobert, Koray Kavukcuoglu, and Clement Farabet. 2011a. Torch7: A Matlab-like Environment for Machine Learning. In BigLearn, NIPS Workshop. http://torch.ch. ↗
  15. Minh-Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the Rare Word Problem in Neural Machine Translation. In Proc. of ACL. find our card
  16. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011b. Natural Language Processing (almost) from scratch. JMLR 12(Aug):2493–2537. find our card
  17. Fandong Meng, Zhengdong Lu, Mingxuan Wang, Hang Li, Wenbin Jiang, and Qun Liu. 2015. Encoding Source Language with Convolutional Neural Network for Machine Translation. In Proc. of ACL. find
  18. Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A Simple, Fast, and Effective Reparameterization of IBM Model 2. Proc. of ACL. find
  19. Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016. Vocabulary Manipulation for Neural Machine Translation. arXiv preprint arXiv:1605.03209 . find
  20. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. In Proc. of CVPR. find
  21. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the Difficulty of Training Recurrent Neural Networks. ICML (3) 28:1310–1318. find
  22. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735– 1780. find
  23. Ngoc-Quan Pham, Germn Kruszewski, and Gemma Boleda. 2016. Convolutional Neural Network Language Models. In Proc. of EMNLP. find
  24. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2014. On Using Very Large Target Vocabulary for Neural Machine Translation. arXiv preprint arXiv:1412.2007v2 . find our card
  25. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level Training with Recurrent Neural Networks. In Proc. of ICLR. find our card
  26. Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. Montreal Neural Machine Translation systems for WMT15. In Proc. of WMT. pages 134–140. find
  27. Holger Schwenk. 2014. http://www-lium. univ-lemans.fr/˜schwenk/cslm_joint_ paper/. Accessed: 2016-10-15. find
  28. Marcin Junczys-Dowmunt, Tomasz Dwojak, and Hieu Hoang. 2016. Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions. arXiv preprint arXiv:1610.01108 . find our card
  29. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for wmt 16. find our card
  30. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural Machine Translation of Rare Words with Subword Units. In Proc. of ACL. find
  31. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent Neural Networks from overfitting. JMLR 15:1929–1958. find
  32. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, and Arthur Szlam. 2015. End-to-end Memory Networks. In Proc. of NIPS. pages 2440–2448. find
  33. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In Proc. of NIPS. pages 3104–3112. find
  34. Zhaopeng Tu, Baotian Hu, Zhengdong Lu, and Hang Li. 2015. Context-dependent Translation selection using Convolutional Neural Network. In Proc. of ACL-IJCNLP. find
  35. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint arXiv:1609.08144 . find
  36. Zichao Yang, Zhiting Hu, Yuntian Deng, Chris Dyer, and Alex Smola. 2016. Neural Machine Translation with Recurrent Attention Modeling. arXiv preprint arXiv:1607.05108 . find
  37. Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. 2016. Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation. arXiv preprint arXiv:1606.04199 . find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-encoder-model-neural-machine/