Elamur Research⌕
☰
home / papers / 2015 / Effective Approaches to Attention-based Neural Machine Translation

Effective Approaches to Attention-based Neural Machine Translation

Summary

The paper addresses neural machine translation (NMT) with an attention mechanism, which lets the decoder attend to the relevant parts of the source sentence when generating each word. The authors, from Stanford, proposed two simple variants of attention. Global attention weighs all encoder hidden states at every step. Local attention first predicts an alignment position in the source sentence and looks only at a window around it, so it is cheaper to compute while remaining differentiable. Both variants are built on a multilayer LSTM model. The authors also compared several scoring functions for the match between states. The models were evaluated on WMT tasks for English and German in both directions. Compared with models lacking attention but already trained with dropout, local attention improved results by as much as 5.0 BLEU. An ensemble of models set a new state of the art for English→German translation on WMT'14 and WMT'15. The work is significant because it simplified the attention architecture previously proposed by Bahdanau et al. and studied it systematically. Its terms global/local attention and dot/general scoring became standard in subsequent work.

Key points

  • The ensemble reached 25.9 BLEU on WMT'15 English→German. This is 1.0 BLEU higher than the previous best system, which combined NMT with an n-gram reranker.
  • The scoring functions dot, general and concat are examined, along with an early location-based variant. According to the authors, the alternatives outperform concat, which was used by Bahdanau et al.
  • Local attention blends ideas of soft and hard attention from Xu et al. (2015): it is cheaper than global attention and, unlike hard attention, is trained with standard backpropagation.
  • In alignment quality (AER), the models matched the Berkeley aligner, with the local variants achieving a lower AER than the global one.
  • Attentional models translate proper names and long sentences better. The code and models are published at nlp.stanford.edu/projects/nmt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1166 · arXiv 1508.04025

@inproceedings{luong2015effective,
  title = {{Effective Approaches to Attention-based Neural Machine Translation}},
  author = {Luong, Thang and Pham, Hieu and Manning, Christopher D.},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1166},
  eprint = {1508.04025},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d15-1166}
}

References

16 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  2. Christian Buck, Kenneth Heafield, and Bas van Ooyen. 2014. N-gram counts and language models from the common crawl. In LREC. find our card
  3. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP. find
  4. Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. End-to-end continuous speech recognition using attention-based recurrent NN: first results. CoRR, abs/1412.1602. — [arXiv:1412.1602](https://arxiv.org/abs/1412.1602) find
  5. Alexander Fraser and Daniel Marcu. 2007. Measuring word alignment quality for statistical machine translation. Computational Linguistics, 33(3):293–303. find
  6. Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. 2015. DRAW: A recurrent neural network for image generation. In ICML. find
  7. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In ACL. find our card
  8. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP. find our card
  9. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In NAACL. find
  10. Percy Liang, Ben Taskar, and Dan Klein. 2006. Alignment by agreement. In NAACL. find our card
  11. Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. 2015. Addressing the rare word problem in neural machine translation. In ACL. find our card
  12. Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. 2014. Recurrent models of visual attention. In NIPS. find
  13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL. find our card
  14. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS. find
  15. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In ICML. find our card
  16. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2015. Recurrent neural network regularization. In ICLR. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/