Elamur Research
☰
home / papers / 2014 / On the Properties of Neural Machine Translation: Encoder-Decoder Approaches

On the Properties of Neural Machine Translation: Encoder-Decoder Approaches

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1409.1259

Summary

Encoder–decoder neural machine translation was new in 2014, and little was known about when it works and when it fails. This paper looks at that question with two English-to-French models. Both use a gated RNN decoder but have different encoders. One is the RNN Encoder–Decoder (RNNenc). The other is a new gated recursive convolutional network (grConv), which builds a sentence vector by applying a shared, gated binary convolution repeatedly. The models were trained on a 348M-word parallel corpus selected from Europarl, news commentary, UN and crawled data. Training used only sentences of up to 30 words and the 30,000 most frequent words in each language. BLEU was measured on the news-test2012–2014 sets and compared with a Moses SMT system. Both models translate short sentences without rare words reasonably well. Quality drops sharply as sentences get longer and as more unknown words appear. The grConv also learns a tree-like sentence structure without any syntactic supervision. These results helped set later research directions: larger vocabularies, handling long sentences, and stronger decoders.

Key points

  • Two encoders were compared: RNNenc (1000 hidden units, tanh) and grConv (2000 hidden units, rectifier). Both use 620-dimensional word embeddings and AdaDelta training.
  • Each model trained for about 110 hours, which was 846,322 updates for RNNenc and 296,144 for grConv. Neither reached convergence, and grConv scored lower on BLEU.
  • Words outside the 30,000-word vocabulary are replaced by an [UNK] token. Translation quality falls as sentence length and the number of unknown words grow.
  • The neural models needed only about 500MB of memory, while typical phrase-based SMT systems need tens of gigabytes.
  • grConv's gating chooses at each node between merging the two child nodes and passing up the left or right child unchanged. On the sentence "Obama is the President…", it produced phrase groupings that look like a sensible parse.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

634 papers of our library cite it (of the 140673 whose bibliographies we read); 38 of them have a page here, 596 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1409.1259 · arXiv 1409.1259

@misc{cho2014properties,
  title = {{On the Properties of Neural Machine Translation: Encoder-Decoder Approaches}},
  author = {Cho, Kyunghyun and Merrienboer, Bart van and Bahdanau, Dzmitry and Bengio, Yoshua},
  year = {2014},
  doi = {10.48550/arxiv.1409.1259},
  eprint = {1409.1259},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1409.1259}
}

References

12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 355–362. Association for Computational Linguistics. find our card
  2. Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. 2013. Audio chord recognition with recurrent neural networks. In ISMIR. find
  3. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), October.… find
  4. Alex Graves. 2012. Sequence transduction with recurrent neural networks. In Proceedings of the 29th International Conference on Machine Learning (ICML 2012). find our card
  5. A. Graves. 2013. Generating sequences with recurrent neural networks. arXiv:1308.0850 [cs.NE], August. find
  6. S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. find
  7. Nal Kalchbrenner and Phil Blunsom. 2013. Two recurrent continuous translation models. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1700–1709. Association for Computational Linguistics. find
  8. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03, pages 48–54, Stroudsburg, PA, USA. As… find
  9. Chang Liu, Daniel Dahlmeier, and Hwee Tou Ng. 2011. Better evaluation metrics lead to better machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 375–384. Association for Computational Linguistics. find
  10. Xingyi Song, Trevor Cohn, and Lucia Specia. 2013. BLEU deconstructed: Designing a better MT evaluation metric. In Proceedings of the 14th International Conference on Intelligent Text Processing and Computational Linguistics (CICLING), March. find
  11. Ilya Sutskever, Oriol Vinyals, and Quoc Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS 2014), December. find
  12. Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. Technical report, arXiv 1212.5701. find our card