Elamur Research
☰
home / papers / 2017 / Convolutional Sequence to Sequence Learning

Convolutional Sequence to Sequence Learning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1705.03122

Summary

Before this paper, neural sequence-to-sequence models for tasks such as machine translation and summarization were mostly recurrent encoder–decoders (LSTM or GRU with attention). Recurrent models process tokens one step at a time, which limits how well they can use GPUs. Researchers at Facebook AI Research propose ConvS2S, an encoder–decoder built only from stacked 1-D convolutions. Each block applies a convolution, gated linear units (GLU) and a residual connection. Learned absolute position embeddings tell the model about word order. Every decoder layer has its own dot-product attention over the encoder output ("multi-step attention"), and source embeddings are added to the values it attends to. Without recurrence, all positions are computed in parallel during training, and stacking layers widens the context each layer sees. The model was tested on WMT'14 English-German, WMT'14 English-French and WMT'16 English-Romanian, and on Gigaword and DUC-2004 summarization. It beats the deep LSTM system of Wu et al. (2016) and translates about ten times faster on both GPU and CPU. This showed that convolutional models are a practical, efficient alternative to RNNs for NMT.

Key points

  • Translation results: sets a new state of the art on WMT'16 English-Romanian (+1.9 BLEU over the previous best). It beats Wu et al. (2016) by 0.5 BLEU on WMT'14 English-German and by 1.6 BLEU on WMT'14 English-French (compared with the likelihood-trained version of that system).
  • Speed: translates about an order of magnitude faster than Wu et al. on GPU and CPU. Adding attention to 4 more decoder layers slows the model by only about 4%, because attention is computed for all target words in one batch.
  • Ablations: removing both source and target position embeddings costs only about 0.5 BLEU. Encoder accuracy improves with depth up to about 9 layers, while 2 decoder layers already perform well. Narrow kernels (width 3) with more layers beat wider kernels.
  • Summarization: a small model (6 encoder and 6 decoder layers, hidden size 256, trained overnight on one GPU) beats the likelihood-trained RNN of Shen et al. (2016) on Gigaword and DUC-2004. It comes close to the best systems, which use task-specific training or architecture.
  • Resource: code and trained models are released in fairseq (github.com/facebookresearch/fairseq).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

612 papers of our library cite it (of the 140673 whose bibliographies we read); 37 of them have a page here, 575 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1705.03122 · arXiv 1705.03122

@misc{gehring2017convolutional,
  title = {{Convolutional Sequence to Sequence Learning}},
  author = {Gehring, Jonas and Auli, Michael and Grangier, David and Yarats, Denis and Dauphin, Yann N.},
  year = {2017},
  doi = {10.48550/arxiv.1705.03122},
  eprint = {1705.03122},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1705.03122}
}

References

49 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. find our card
  2. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. find our card
  3. Bojar, Ondej, Chatterjee, Rajen, Federmann, Christian, Graham, Yvette, Haddow, Barry, Huck, Matthias, Jimeno-Yepes, Antonio, Koehn, Philipp, Logacheva, Varvara, Monz, Christof, Negri, Matteo, Névéol, Aurélie, Neves, Mariana L., Popel, Martin, Post, Matt, Rubino, Raphaël, Scarton, Carolina, Speci… find our card
  4. Bradbury, James, Merity, Stephen, Xiong, Caiming, and Socher, Richard. Quasi-Recurrent Neural Networks. arXiv preprint arXiv:1611.01576, 2016. — [arXiv:1611.01576](https://arxiv.org/abs/1611.01576) find
  5. Cho, Kyunghyun, Van Merriënboer, Bart, Gulcehre, Caglar, Bahdanau, Dzmitry, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proc. of EMNLP, 2014. find
  6. Chorowski, Jan K, Bahdanau, Dzmitry, Serdyuk, Dmitriy, Cho, Kyunghyun, and Bengio, Yoshua. Attention-based models for speech recognition. In Advances in Neural Information Processing Systems, pp. 577–585, 2015. find our card
  7. Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Clement. Torch7: A Matlab-like Environment for Machine Learning. In BigLearn, NIPS Workshop, 2011. URL http://torch.ch. ↗
  8. Dauphin, Yann N., Fan, Angela, Auli, Michael, and Grangier, David. Language modeling with gated linear units. arXiv preprint arXiv:1612.08083, 2016. find our card
  9. Dyer, Chris, Chahuneau, Victor, and Smith, Noah A. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proc. of ACL, 2013. find
  10. Elman, Jeffrey L. Finding Structure in Time. Cognitive Science, 14:179–211, 1990. find
  11. Gehring, Jonas, Auli, Michael, Grangier, David, and Dauphin, Yann N. A Convolutional Encoder Model for Neural Machine Translation. arXiv preprint arXiv:1611.02344, 2016. find
  12. Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. The handbook of brain theory and neural networks, 2010. find
  13. Graff, David, Kong, Junbo, Chen, Ke, and Maeda, Kazuaki. English gigaword. Linguistic Data Consortium, Philadelphia, 2003. find
  14. Ha, David, Dai, Andrew, and Le, Quoc V. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016. — [arXiv:1609.09106](https://arxiv.org/abs/1609.09106) find
  15. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep Residual Learning for Image Recognition. In Proc. of CVPR, 2015a. find
  16. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034, 2015b. find
  17. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  18. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of The 32nd International Conference on Machine Learning, pp. 448–456, 2015. find
  19. Jean, Sébastien, Firat, Orhan, Cho, Kyunghyun, Memisevic, Roland, and Bengio, Yoshua. Montreal Neural Machine Translation systems for WMT15. In Proc. of WMT, pp. 134–140, 2015. find
  20. Kalchbrenner, Nal, Espeholt, Lasse, Simonyan, Karen, van den Oord, Aaron, Graves, Alex, and Kavukcuoglu, Koray. Neural Machine Translation in Linear Time. arXiv, 2016. find
  21. LeCun, Yann and Bengio, Yoshua. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995. find
  22. L’Hostis, Gurvan, Grangier, David, and Auli, Michael. Vocabulary Selection Strategies for Neural Machine Translation. arXiv preprint arXiv:1610.00072, 2016. — [arXiv:1610.00072](https://arxiv.org/abs/1610.00072) find
  23. Lin, Chin-Yew. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pp. 74–81, 2004. find
  24. Luong, Minh-Thang, Pham, Hieu, and Manning, Christopher D. Effective approaches to attention-based neural machine translation. In Proc. of EMNLP, 2015. find
  25. Meng, Fandong, Lu, Zhengdong, Wang, Mingxuan, Li, Hang, Jiang, Wenbin, and Liu, Qun. Encoding Source Language with Convolutional Neural Network for Machine Translation. In Proc. of ACL, 2015. find
  26. Mi, Haitao, Wang, Zhiguo, and Ittycheriah, Abe. Vocabulary Manipulation for Neural Machine Translation. In Proc. of ACL, 2016. find
  27. Miller, Alexander H., Fisch, Adam, Dodge, Jesse, Karimi, Amir-Hossein, Bordes, Antoine, and Weston, Jason. Key-value memory networks for directly reading documents. In Proc. of EMNLP, 2016. find
  28. Nallapati, Ramesh, Zhou, Bowen, Gulcehre, Caglar, Xiang, Bing, et al. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proc. of EMNLP, 2016. find
  29. Oord, Aaron van den, Kalchbrenner, Nal, and Kavukcuoglu, Koray. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759, 2016a. — [arXiv:1601.06759](https://arxiv.org/abs/1601.06759) find
  30. Oord, Aaron van den, Kalchbrenner, Nal, Vinyals, Oriol, Espeholt, Lasse, Graves, Alex, and Kavukcuoglu, Koray. Conditional image generation with pixelcnn decoders. arXiv preprint arXiv:1606.05328, 2016b. — [arXiv:1606.05328](https://arxiv.org/abs/1606.05328) find
  31. Over, Paul, Dang, Hoa, and Harman, Donna. Duc in context. Information Processing & Management, 43(6): 1506–1520, 2007. find
  32. Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua. On the difficulty of training recurrent neural networks. In Proceedings of The 30th International Conference on Machine Learning, pp. 1310–1318, 2013. find
  33. Rush, Alexander M, Chopra, Sumit, and Weston, Jason. A neural attention model for abstractive sentence summarization. In Proc. of EMNLP, 2015. find
  34. Salimans, Tim and Kingma, Diederik P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07868, 2016. — [arXiv:1602.07868](https://arxiv.org/abs/1602.07868) find
  35. Schuster, Mike and Nakajima, Kaisuke. Japanese and korean voice search. In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on, pp. 5149–5152. find
  36. Sennrich, Rico, Haddow, Barry, and Birch, Alexandra. Neural Machine Translation of Rare Words with Subword Units. In Proc. of ACL, 2016a. find
  37. Sennrich, Rico, Haddow, Barry, and Birch, Alexandra. Edinburgh Neural Machine Translation Systems for WMT 16. In Proc. of WMT, 2016b. find our card
  38. Shazeer, Noam, Mirhoseini, Azalia, Maziarz, Krzysztof, Davis, Andy, Le, Quoc, Hinton, Geoffrey, and Dean, Jeff. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ArXiv e-prints, January 2016. find
  39. Shen, Shiqi, Zhao, Yu, Liu, Zhiyuan, Sun, Maosong, et al. Neural headline generation with sentence-wise optimization. arXiv preprint arXiv:1604.01904, 2016. — [arXiv:1604.01904](https://arxiv.org/abs/1604.01904) find
  40. Srivastava, Nitish, Hinton, Geoffrey E., Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: a simple way to prevent Neural Networks from overfitting. JMLR, 15:1929–1958, 2014. find
  41. Sukhbaatar, Sainbayar, Weston, Jason, Fergus, Rob, and Szlam, Arthur. End-to-end Memory Networks. In Proc. of NIPS, pp. 2440–2448, 2015. find
  42. Sutskever, Ilya, Martens, James, Dahl, George E., and Hinton, Geoffrey E. On the importance of initialization and momentum in deep learning. In ICML, 2013. find
  43. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. Sequence to Sequence Learning with Neural Networks. In Proc. of NIPS, pp. 3104–3112, 2014. find
  44. Suzuki, Jun and Nagata, Masaaki. Cutting-off redundant repeating generations for neural abstractive summarization. arXiv preprint arXiv:1701.00138, 2017. find
  45. Waibel, Alex, Hanazawa, Toshiyuki, Hinton, Geoffrey, Shikano, Kiyohiro, and Lang, Kevin J. Phoneme Recognition using Time-delay Neural Networks. IEEE transactions on acoustics, speech, and signal processing, 37 (3):328–339, 1989. find
  46. Wu, Yonghui, Schuster, Mike, Chen, Zhifeng, Le, Quoc V, Norouzi, Mohammad, Macherey, Wolfgang, Krikun, Maxim, Cao, Yuan, Gao, Qin, Macherey, Klaus, et al. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint arXiv:1609.08144, 2016. find
  47. Yang, Zichao, Hu, Zhiting, Deng, Yuntian, Dyer, Chris, and Smola, Alex. Neural Machine Translation with Recurrent Attention Modeling. arXiv preprint arXiv:1607.05108, 2016. find
  48. Zhou, Jie, Cao, Ying, Wang, Xuguang, Li, Peng, and Xu, Wei. Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation. arXiv preprint arXiv:1606.04199, 2016. find
  49. A. Weight Initialization We derive a weight initialization scheme tailored to the GLU activation function similar to Glorot & Bengio (2010); He et al. (2015b) by focusing on the variance of activations within the network for both forward and backward passes. We also detail how we modify the weight i… find