Elamur Research
☰
home / papers / 2018 / Tensor2Tensor for Neural Machine Translation

Tensor2Tensor for Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-1819.pdf

Summary

This paper presents Tensor2Tensor (T2T), an open-source TensorFlow library from Google. It aims to make deep learning research, especially neural machine translation, faster to run and easier to reproduce. The authors first trace NMT from recurrent sequence-to-sequence models with attention, through convolutional systems such as ByteNet and the Extended Neural GPU, to the self-attention-based Transformer. T2T contains the reference implementation of the Transformer. Each training run is broken into interchangeable parts: dataset definitions, device setup, named hyperparameter sets, models, and TensorFlow's training machinery. As a result, researchers can apply one model to many tasks, or many models to one task, while changing only the part they care about. The same code runs on CPU, GPU or TPU, either locally or in the cloud. Several building blocks first developed in T2T later moved into TensorFlow or supported follow-up work. The paper reports no new benchmark scores. Its value is practical: shared infrastructure that speeds up Transformer-style research and keeps results reproducible as the code changes.

Key points

  • A training run in T2T is defined by five components: Problem (dataset), device configuration, hyperparameters, Model, and Estimator/Experiment.
  • T2T supports CPU, GPU and TPU, single or multi-device, with synchronous or asynchronous data-parallel training. It started with NMT and later added image, video and audio tasks.
  • Research components first built or tested in T2T include the Image Transformer, tf.contrib.layers.rev_block (reversible layers), the memory-saving Adafactor optimizer, and tf.contrib.data.bucket_by_sequence_length.
  • Reproducibility relies on three mechanisms: versioned named hyperparameter sets, regular end-to-end regression tests on key model-problem pairs, and random seeds fixed in Python, numpy and TensorFlow.
  • The code is developed openly at github.com/tensorflow/tensor2tensor, with contributors from inside and outside Google.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

160 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 148 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1803.07416

@inproceedings{vaswani2018tensor2tensor,
  title = {{Tensor2Tensor for Neural Machine Translation}},
  author = {Vaswani, Ashish and Bengio, Samy and Brevdo, Eugene and Chollet, Francois and Gomez, Aidan and Gouws, Stephan and Jones, Llion and Kaiser, Łukasz and Kalchbrenner, Nal and Parmar, Niki and Sepassi, Ryan and Shazeer, Noam and Uszkoreit, Jakob},
  booktitle = {Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)},
  year = {2018},
  eprint = {1803.07416},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.07416}
}

References

14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D. G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., and Zheng, X. (2016). Tensorflow: A system for large-… find
  2. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. find our card
  3. Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, abs/1406.1078. find
  4. Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. N. (2017). Convolutional sequence to sequence learning. CoRR, abs/1705.03122. find our card
  5. Gomez, A. N., Ren, M., Urtasun, R., and Grosse, R. B. (2017). The reversible residual network: Backpropagation without storing activations. CoRR, abs/1707.04585. — [arXiv:1707.04585](https://arxiv.org/abs/1707.04585) find
  6. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8):1735– 1780. find
  7. Kaiser, Ł. and Bengio, S. (2016). Can active memory replace attention? In Advances in Neural Information Processing Systems, pages 3781–3789. find
  8. Kaiser, L., Gomez, A. N., and Chollet, F. (2017). Depthwise separable convolutions for neural machine translation. CoRR, abs/1706.03059. — [arXiv:1706.03059](https://arxiv.org/abs/1706.03059) find
  9. Kalchbrenner, N. and Blunsom, P. (2013). Recurrent continuous translation models. In Proceedings EMNLP 2013, pages 1700–1709. find
  10. Kalchbrenner, N., Espeholt, L., Simonyan, K., van den Oord, A., Graves, A., and Kavukcuoglu, K. (2016). Neural machine translation in linear time. CoRR, abs/1610.10099. — [arXiv:1610.10099](https://arxiv.org/abs/1610.10099) find
  11. Meng, F., Lu, Z., Wang, M., Li, H., Jiang, W., and Liu, Q. (2015). Encoding source language with convolutional neural network for machine translation. In ACL, pages 20–30. find
  12. Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., and Ku, A. (2018). Image Transformer. ArXiv e-prints. find
  13. Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu,… find our card
  14. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. CoRR. find