Elamur Research
☰
home / papers / 2018 / Scaling Neural Machine Translation

Scaling Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W18-6301.pdf

Summary

Training large Transformer translation models could take several days even on a server with 8 GPUs. This paper from Facebook AI Research shows how to cut that time sharply without losing translation quality. Using the "big" Transformer in fairseq, the authors combine three techniques. First, they compute in FP16 with dynamic loss scaling. Second, they use very large batches, from 25k up to about 400k target tokens. They build these by accumulating gradients over several passes before each update and pair them with a doubled learning rate. Third, they run synchronous data-parallel training on up to 16 DGX-1 nodes (128 V100 GPUs). They also analyse stragglers, which arise because GPUs take uneven time to process their sub-batches. On one machine, training becomes almost 5x faster. The model reaches the accuracy of Vaswani et al. (2017) on WMT En–De in under 5 hours, or roughly half an hour on 128 GPUs. Fully converged models set new state-of-the-art BLEU on En–De and En–Fr. The faster pipeline also makes it practical to add filtered Paracrawl data. Shorter training means quicker experiments and makes far larger corpora usable.

Key points

  • On 8 V100 GPUs (WMT En–De, target: validation NLL 2.11), the training time drops in stages:

    • 1,429 min for the FP32 reimplementation;
    • 495 min with FP16;
    • 311 min with gradient accumulation (cumul=16, ~402k-token batches) and 2x learning rate.

    On 16 nodes it falls to 37 min, or 32 min when gradient communication overlaps with the backward pass, a 44.7x speedup.

  • On newstest14, En–De reaches 29.3 BLEU after 85 minutes on 128 GPUs, versus 29.2 for Shaw et al. (2018). WMT'14 En–Fr reaches 43.2 BLEU in 8.5 hours, versus 41.5.

  • Paracrawl En–De was filtered in two steps:

    • rules on length ratio, sentence length and source/target copies reduced it from 4.6B to 700M pairs;
    • a model trained on clean data then scored the remaining pairs, and the 140M best-scoring pairs were kept.

    Sampling WMT and Paracrawl 1:1 raised En–De to 29.8 BLEU. For En–Fr (129M filtered pairs, 3:1 ratio), the extra data hurt: 42.1 vs 43.2 BLEU.

  • Sub-batch processing times vary from 0.049 to 0.228 s (mean 0.11 s). Sizing sub-batches by estimated processing time raised throughput from 143k to 150k tokens/s. However, gradient accumulation helped more.

  • Scaling efficiency is limited: 16 nodes give only about 10x the speed of one node. The code is released in fairseq (github.com/pytorch/fairseq).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

269 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 249 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6301 · arXiv 1806.00187

@inproceedings{ott2018scaling,
  title = {{Scaling Neural Machine Translation}},
  author = {Ott, Myle and Edunov, Sergey and Grangier, David and Auli, Michael},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Research Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6301},
  eprint = {1806.00187},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w18-6301}
}

References

39 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Karim Ahmed, Nitish Shirish Keskar, and Richard Socher. 2017. Weighted transformer network for machine translation. arxiv, 1711.02132. find
  2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. find our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR. find our card
  4. Jianmin Chen, Rajat Monga, Samy Bengio, and Rafal Józefowicz. 2016. Revisiting distributed synchronous sgd. Arxiv, 1604.00981. find
  5. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in neural machine translation. arxiv, 1804.09849. find our card
  6. Adam Coates, Brody Huval, Tao Wang, David J. Wu, Bryan Catanzaro, and Andrew Y. Ng. 2013. Deep learning with cots hpc systems. In Proc. of ICML. find
  7. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2015. Training deep neural networks with low precision multiplications. \#\# Conclusions Jeff Dean. 2017. Machine learning for systems and systems for machine learning. In Proc. of NIPS Workshop on ML Systems. find
  8. We explored how to train state-of-the-art NMT models on large scale parallel hardware. We in-Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. In Proc. of ICLR Workshop. find our card
  9. Jeffrey Dean, Gregory S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc’Aurelio Ranzato, Andrew W. Senior, Paul A. Tucker, Ke Yang, and Andrew Y. Ng. 2012. Large scale distributed deep networks. In Proc. of NIPS. find
  10. Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv, 1804.08771. find our card
  11. Ondrej Dusek and Filip Jurcı́cek. 2016. Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings. In Proc. of ACL. find
  12. Raul Puri, Robert Kirby, Nikolai Yakovenko, and Bryan Catanzaro. 2018. Large scale language modeling: Converging on 40gb of text in four hours. arXiv preprint arXiv:1808.01371. find
  13. Sergey Edunov, Myle Ott, and Sam Gross. 2017. Fairseq. https://github.com/pytorch/ fairseq. ↗
  14. Jerry Quinn and Miguel Ballesteros. 2018. Pieces of eight: 8-bit neural machine translation. In Proc. of NAACL. find
  15. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. In Proc. of ICML. find our card
  16. Christopher De Sa, Megan Leszczynski, Jian Zhang, Alana Marzoev, Christopher R. Aberger, Kunle Olukotun, and Christopher Ré. 2018. High-accuracy low-precision training. Arxiv, 1803.03383. find
  17. Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. In Proc. of CVPR. find
  18. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proc. of ACL. find
  19. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. In Proc. of CVPR. find
  20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. of ACL. find
  21. Elad Hoffer, Itay Hubara, and Daniel Soudry. 2017. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In Proc. of NIPS, pages 1729–1739. find
  22. Iulian Serban, Alessandro Sordoni, Ryan Joseph Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proc. of AAAI. find our card
  23. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Proc. of ICLR. find
  24. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proc. of NAACL. find our card
  25. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In ACL… find
  26. Patrice Y. Simard and Hans Peter Graf. 1993. Backpropagation without multiplication. In Proc. of NIPS. find
  27. Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. 2018. Don’t decay the learning rate, increase the batch size. In Proc. of ICLR. find
  28. Jaroslaw Kutylowski. 2018. Deepl press information. https://www.deepl.com/press.html. ↗
  29. Alessandro Sordoni, Michel Galley2, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proc. of ACL. find our card
  30. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed Precision Training. In Proc. of ICLR. find
  31. Myle Ott, Michael Auli, David Grangier, and MarcAu-relio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning (ICML). find our card
  32. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  33. ParaCrawl. 2018. ParaCrawl. http: //paracrawl.eu/download.html. find
  34. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2015. Rethinking the Inception Architecture for Computer Vision. arXiv preprint arXiv:1512.00567. find
  35. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In Proc. of ICLR. find our card
  36. Rashish Tandon, Qi Lei, Alexandros G. Dimakis, and Nikos Karampatziakis. 2017. Gradient Coding: Avoiding Stragglers in Distributed Learning. In Proc. of ICML. find
  37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Proc. of NIPS. find
  38. Hainan Xu and Philipp Koehn. 2017. Zipporah: a fast and scalable data cleaning system for noisy web-crawled parallel corpora. In Proc. of EMNLP. find
  39. Min Ye and Emmanuel Abbe. 2018. Communication-Computation Efficient Gradient Coding. In Proc. of ICML. find