Elamur Research⌕
☰
home / papers / 2014 / Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Paper · 2014 · Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre et al.

Summary

The authors propose the RNN Encoder–Decoder neural network to improve phrase-based statistical machine translation (phrase-based SMT). The model consists of two recurrent networks. The first, the encoder, compresses an input sequence of arbitrary length into a fixed-size vector. The second, the decoder, unfolds this vector into the target sequence. Both components are trained jointly to maximize the conditional probability of the translation. Together with the architecture, a new hidden unit is proposed: a simplified analogue of the LSTM with two gates, reset and update. The model was trained on pairs of English and French phrases from a phrase table. The pair frequencies were deliberately ignored so that the network would learn linguistic regularities rather than simply reproduce corpus statistics. The model's scores were added as an additional feature to the log-linear SMT model, and this improved BLEU. A qualitative analysis showed that the model prefers more literal and shorter translations. Moreover, it builds a vector space in which phrases cluster both semantically and syntactically. This paper was the first to introduce the architecture later named GRU, and the work influenced the subsequent development of seq2seq-type neural machine translation.

Key points

  • Task and data: English-to-French translation. The model does not replace the system; instead, it scores each phrase pair in the phrase table as an additional component of an existing SMT system.
  • New hidden unit: it has only two gates, compared with four gating components and a memory cell in the LSTM. The reset gate allows irrelevant history to be discarded, while the update gate determines how much of the previous state to retain. Without gates, using ordinary tanh units, the authors were unable to obtain meaningful results.
  • Result: the RNN Encoder–Decoder scores improve BLEU. Their contribution barely overlaps with that of the neural language model (CSLM), so combining both approaches yields an even larger gain.
  • Generation: for each source phrase the model generated 50 candidates, of which the authors show the top five. These are grammatically correct phrases that only partially coincide with the phrase table. The authors therefore believe that in the future the model could replace the table entirely or in part.
  • Representations: the phrase vector c has 1000 dimensions. A visualization using Barnes-Hut-SNE (5,000 points) reveals distinct clusters of phrases about durations of time and about countries and regions, as well as groups of syntactically similar phrases.

The omitted middle part of the text contained the specific BLEU values, so they are not included in the summary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Le Mans UniversitéLe Mans · France15 works · 31,097 citations🎓Polytechnique MontréalMontreal · Canada9 works · 27,467 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 1,087 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Natural language processing · 131 works

Cite

Cite as Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/d14-1179 · arXiv 1406.1078

@inproceedings{cho2014learning,
  title = {{Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation}},
  author = {Kyunghyun Cho, Bart {van Merriënboer}, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk and Yoshua Bengio},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1179},
  eprint = {1406.1078},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.3115/v1/d14-1179}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Auli, Michel Galley, Chris Quirk, and Geoffrey Zweig. 2013. Joint language and translation modeling with recurrent neural networks. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1044–1054. find
  2. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 355–362.
  3. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. 2012. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop. find
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn. Res., 3:1137–1155, March.
  5. Y. Bengio, N. Boulanger-Lewandowski, and R. Pascanu. 2013. Advances in optimizing recurrent networks. In Proceedings of the 38th International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2013), May. find
  6. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June. Oral Presentation. find
  7. Sarath Chandar, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas Raykar, and Amrita Saha. 2014. An autoencoder approach to learning bilingual word representations. arXiv:1402.1454 [cs.CL], February. ↗
  8. George E. Dahl, Dong Yu, Li Deng, and Alex Acero. 2012. Context-dependent pre-trained deep neural networks for large vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):33–42. find
  9. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, , and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proceedings of the ACL 2014 Conference, ACL ’14, pages 1370–1380.
  10. Jianfeng Gao, Xiaodong He, Wen tau Yih, and Li Deng. 2013. Learning semantic representations for the phrase translation model. Technical report, Microsoft Research. find
  11. X. Glorot, A. Bordes, and Y. Bengio. 2011. Deep sparse rectifier neural networks. In AISTATS’2011. find
  12. Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. 2013. Maxout networks. In ICML’2013. find
  13. Alex Graves. 2012. Supervised Sequence Labelling with Recurrent Neural Networks. Studies in Computational Intelligence. Springer. find
  14. S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. ↗
  15. Nal Kalchbrenner and Phil Blunsom. 2013. Two recurrent continuous translation models. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1700–1709.
  16. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03, pages 48–54. find
  17. P. Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Machine Translation Summit X, pages 79–86, Phuket, Thailand. find
  18. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS’2012).
  19. Daniel Marcu and William Wong. 2002. A phrase-based, joint probability model for statistical machine translation. In Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10, EMNLP ’02, pages 133–139.
  20. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111–3119.
  21. Robert C. Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Papers, ACLShort ’10, pages 220–224, Stroudsburg, PA, USA.
  22. R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio. 2014. How to construct deep recurrent neural networks. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014), April. find
  23. Andrew M. Saxe, James L. McClelland, and Surya Ganguli. 2014. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014), April. find
  24. Holger Schwenk, Marta R. Costa-Jussà, and José A. R. Fonollosa. 2006. Continuous space language models for the iwslt 2006 task. In IWSLT, pages 166–173. find
  25. Holger Schwenk. 2007. Continuous space language models. Comput. Speech Lang., 21(3):492–518, July. find
  26. Holger Schwenk. 2012. Continuous space translation models for phrase-based statistical machine translation. In Martin Kay and Christian Boitet, editors, Proceedings of the 24th International Conference on Computational Linguistics (COLIN), pages 1071–1080. find
  27. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems 24. find
  28. Le Hai Son, Alexandre Allauzen, and François Yvon. 2012. Continuous space translation models with neural networks. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT ’12, pages 39–48, Stroudsburg, PA, USA. find
  29. Laurens van der Maaten. 2013. Barnes-hut-sne. In Proceedings of the First International Conference on Learning Representations (ICLR 2013), May. find
  30. Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang. 2013. Decoding with large-scale neural language models improves translation. Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1387–1392.
  31. Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. Technical report, arXiv 1212.5701. ↗
  32. Will Y. Zou, Richard Socher, Daniel M. Cer, and Christopher D. Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1393–1398.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cho-2014-learning-phrase-representations-rnn-encoder/