Elamur Research
☰
home / papers / 2014 / Neural Machine Translation by Jointly Learning to Align and Translate

Neural Machine Translation by Jointly Learning to Align and Translate

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1409.0473

Summary

The authors work on neural machine translation. In the encoder–decoder models of the time, the entire source sentence was compressed into a single fixed-length vector. As a result, quality dropped quickly as sentences grew longer. The authors propose the RNNsearch architecture. In it, a bidirectional RNN encodes each source word as a separate annotation that takes into account both the left and the right context. When generating each word of the translation, the decoder itself chooses which parts of the input sentence to attend to. The weights are computed by a small feedforward alignment network, which is trained jointly with the whole model. This is how the attention mechanism in its modern sense came about. The model was evaluated on English-to-French translation using the ACL WMT '14 parallel corpora. RNNsearch clearly outperforms the baseline RNN Encoder–Decoder, and the gap is especially large on long sentences. The quality of a single model approaches that of the phrase-based statistical translation system Moses. The attention weight matrices show word alignments that look linguistically plausible. This work introduced the attention mechanisms from which the Transformer later grew.

Key points

  • Models: RNNenc (the baseline encoder–decoder) and RNNsearch (with attention). Each was trained on sentences of up to 30 and up to 50 words (RNNsearch-30/-50, RNNenc-30/-50). The comparison system is the phrase-based Moses.
  • Data: the WMT '14 parallel corpora for the English→French pair. The vocabulary was limited to roughly 30,000 most frequent words in each language.
  • BLEU on the test set (from the omitted part of the excerpt, taken from the known version of the paper): RNNsearch-50 — 26.75 on all sentences and 34.16 on sentences without unknown words. RNNenc-50 scores 17.82 and 26.71, respectively. For Moses on sentences without unknown words — 35.63, and for the longer-trained RNNsearch-50* — 36.15.
  • Hyperparameters: hidden layer 1000, embeddings 620, maxout layer 500, 1000 hidden units in the alignment model. Optimization: SGD with Adadelta, minibatches of 80 sentences, gradients clipped at norm 1.
  • On sentences of 30 words or more, RNNsearch-50 translates coherently and accurately, while RNNenc-50 loses or distorts the meaning. For the authors, soft alignment as a differentiable alternative to hard latent alignment made it possible to train the system as a single whole.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics: Machine translation

Cited by

4775 papers of our library cite it (of the 140673 whose bibliographies we read); 254 of them have a page here, the first 100 below, 4521 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1409.0473 · arXiv 1409.0473

@misc{bahdanau2014neural,
  title = {{Neural Machine Translation by Jointly Learning to Align and Translate}},
  author = {Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua},
  year = {2014},
  doi = {10.48550/arxiv.1409.0473},
  eprint = {1409.0473},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1409.0473}
}

References

30 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Axelrod, A., He, X., and Gao, J. (2011). Domain adaptation via pseudo in-domain data selection. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 355–362. Association for Computational Linguistics. find our card
  2. Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I. J., Bergeron, A., Bouchard, N., and Bengio, Y. (2012). Theano: new features and speed improvements. find
  3. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop. find
  4. Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. find
  5. Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. (2003). A neural probabilistic language model. J. Mach. Learn. Res., 3, 1137–1155. find
  6. Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y. (2010). Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy). Oral Presentation. find
  7. Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. (2013). Audio chord recognition with recurrent neural networks. In ISMIR. find
  8. Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., and Bengio, Y. (2014a). Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014). to appear. find
  9. Cho, K., van Merriënboer, B., Bahdanau, D., and Bengio, Y. (2014b). On the properties of neural machine translation: Encoder–Decoder approaches. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. to appear. find our card
  10. Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., and Makhoul, J. (2014). Fast and robust neural network joint models for statistical machine translation. In Association for Computational Linguistics. find
  11. Forcada, M. L. and Ñeco, R. P. (1997). Recursive hetero-associative memories for translation. In J. Mira, R. Moreno-Dı́az, and J. Cabestany, editors, Biological and Artificial Computation: From Neuroscience to Technology, volume 1240 of Lecture Notes in Computer Science, pages 453–462. Springer Ber… find
  12. Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013). Maxout networks. In Proceedings of The 30th International Conference on Machine Learning, pages 1319– 1327. find
  13. Graves, A. (2012). Sequence transduction with recurrent neural networks. In Proceedings of the 29th International Conference on Machine Learning (ICML 2012). find our card
  14. Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv:1308.0850 [cs.NE]. find
  15. Graves, A., Jaitly, N., and Mohamed, A.-R. (2013). Hybrid speech recognition with deep bidirectional LSTM. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, pages 273–278. find
  16. Hermann, K. and Blunsom, P. (2014). Multilingual distributed representations without word alignment. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014). find
  17. Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut für Informatik, Lehrstuhl Prof. Brauer, Technische Universität München. find
  18. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. find
  19. Kalchbrenner, N. and Blunsom, P. (2013). Recurrent continuous translation models. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1700–1709. Association for Computational Linguistics. find
  20. Koehn, P. (2010). Statistical Machine Translation. Cambridge University Press, New York, NY, USA. find
  21. Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03, pages 48–54, Stroudsburg, PA, USA. Association f… find
  22. Pascanu, R., Mikolov, T., and Bengio, Y. (2013a). On the difficulty of training recurrent neural networks. In ICML’2013. find
  23. Pascanu, R., Mikolov, T., and Bengio, Y. (2013b). On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning (ICML 2013). find
  24. Pascanu, R., Gulcehre, C., Cho, K., and Bengio, Y. (2014). How to construct deep recurrent neural networks. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014). find
  25. Pouget-Abadie, J., Bahdanau, D., van Merriënboer, B., Cho, K., and Bengio, Y. (2014). Overcoming the curse of sentence length for neural machine translation using automatic segmentation. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. to appear. find
  26. Schuster, M. and Paliwal, K. K. (1997). Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45(11), 2673–2681. find
  27. Schwenk, H. (2012). Continuous space translation models for phrase-based statistical machine translation. In M. Kay and C. Boitet, editors, Proceedings of the 24th International Conference on Computational Linguistics (COLIN), pages 1071–1080. Indian Institute of Technology Bombay. find
  28. Schwenk, H., Dchelotte, D., and Gauvain, J.-L. (2006). Continuous space language models for statistical machine translation. In Proceedings of the COLING/ACL on Main conference poster sessions, pages 723–730. Association for Computational Linguistics. find
  29. Sutskever, I., Vinyals, O., and Le, Q. (2014). Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS 2014). find
  30. ADADELTA: An adaptive learning rate method. arXiv:1212.5701 find our card