Elamur Research⌕
☰
home / papers / 2018 / Training Tips for the Transformer Model

Training Tips for the Transformer Model

Summary

The paper offers a practical guide to training the Transformer NMT model in the Tensor2Tensor (T2T) toolkit, version 1.2.9. Hyper-parameter advice in deep learning has mostly come from theory or from image recognition, so the authors run a systematic set of English-to-Czech experiments instead. They train on about 58M sentence pairs, mostly CzEng 1.7. Their experiments examine how results depend on the learning rate and its number of warmup steps, the batch size, the length cap on training sentences, model size, GPU count and checkpoint averaging, and each experiment ends with concrete recommendations. Because NMT training rarely converges cleanly, they never rely on a single stopping point. Instead they plot full BLEU learning curves against wall-clock time on fixed hardware. Their best setup is the BIG model on 8 GPUs for 8 days with checkpoint averaging. It beats the WMT17 English-to-Czech winner on several automatic metrics, without back-translation, reranking or ensembling. The work matters because it turns roughly four years of GPU time into reusable advice for researchers with limited hardware and data.

Key points

  • Setup: Transformer in T2T v1.2.9 on English→Czech, using CzEng 1.7, europarl-v7, news-commentary-v11 and commoncrawl (about 58M sentences, 640M English words). Evaluation uses case-insensitive sacréBLEU on WMT13, and chrF showed no substantial differences.
  • Metric warning: T2T's default "approx-bleu" is computed on internal subwords and comes out about 1.2–1.8× the real BLEU. The authors therefore released t2t-bleu, t2t-translate-all and t2t-avg-all to evaluate and average every checkpoint.
  • Batch size and GPUs: Larger batches gave both faster training and better final quality. Two GPUs were more than three times faster than one. Given at least a day and an 11GB GPU, the BIG configuration is always preferable.
  • Checkpoint averaging: Averaging checkpoints saved hourly worked slightly better than the original recipe of 20 checkpoints at 10-minute intervals. It usually gains about 0.2 BLEU, sometimes over 1 BLEU, and averaging 8 checkpoints takes about 5 minutes.
  • WMT17 comparison: The best model scored 23.8 BLEU, against 22.8 for uedin-nmt. It also led on TER, CharacTER and BEER.

Interesting ideas

  • I1. Full learning curves instead of a single score: The authors plot BLEU against wall-clock time on identical hardware, rather than reporting one number at an arbitrary stopping point. This lets readers judge whether a setup might still overtake another, and it suits any model that never visibly converges.
  • I2. "Time Till Score" metric: This is the time after which quality never drops below a given threshold. It is a simple, conservative way to compare training efficiency across configurations whose step costs differ.
  • I3. Treat throughput and convergence speed as separate quantities: The paper distinguishes steps per hour, subwords per hour and BLEU gain per hour. This helps explain why a configuration that is faster per step can still be worse in practice.
  • I4. Forking a run for cheap pseudo-ensembles: Resumed training starts from a random data position. Restarting one run into several copies therefore yields semi-independent models that can be averaged like checkpoints, giving a small extra gain at little cost.
  • I5. Schedule-aware resumption for domain adaptation: One can switch to small in-domain data for the last few epochs. When doing so, adjust the learning-rate schedule, or fake the stored global step, so the learning rate is not already too small to adapt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Computing & GPUs

Topics: Machine translation

Cited by

103 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 94 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.2478/pralin-2018-0002 · arXiv 1804.00247

@misc{popel2018training,
  title = {{Training Tips for the Transformer Model}},
  author = {Popel, Martin and Bojar, Ondvrej},
  year = {2018},
  doi = {10.2478/pralin-2018-0002},
  eprint = {1804.00247},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.2478/pralin-2018-0002}
}

References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of ICLR, 2015. find our card
  2. Bojar, Ondřej, Zdeněk Žabokrtský, Ondřej Dušek, Petra Galuščáková, Martin Majliš, David Mareček, Jiří Maršík, Michal Novák, Martin Popel, and Aleš Tamchyna. The Joy of Parallelism with CzEng 1.0. In Proceedings of the Eighth International Language Resources and Evaluation Conference (LREC’12), pages… find
  3. Bojar, Ondřej, Ondřej Dušek, Tom Kocmi, Jindřich Libovický, Michal Novák, Martin Popel, Roman Sudarikov, and Dušan Variš. CzEng 1.6: Enlarged Czech-English Parallel Corpus with Processing Tools Dockered. In Sojka, Petr, Aleš Horák, Ivan Kopeček, and Karel Pala, editors, Text, Speech, and Dialogue: 1… find
  4. Masaryk University, Springer International Publishing, 2016. ISBN 978-3-319-45509-9. find
  5. Bojar, Ondřej, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. Findings of the 2017 Conference on Machine Translation (WMT17). In Proceedings o… find our card
  6. Bojar, Ondřej, Yvette Graham, and Amir Kamran. Results of the WMT17 Metrics Shared Task. In Proceedings of the Second Conference on Machine Translation, Copenhagen, Denmark, September 2017b. ACL. find our card
  7. Bottou, Léon. Stochastic Gradient Descent Tricks, pages 421–436. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642-35289-8. doi: 10.1007/978-3-642-35289-8_25. URL https://doi.org/10.1007/978-3-642-35289-8_25 . — [doi:10.1007/978-3-642-35289-8_25](https://doi.org/10.1007/978-3-642-35289-8_25) ↗
  8. Bottou, L., F. E. Curtis, and J. Nocedal. Optimization Methods for Large-Scale Machine Learning. ArXiv e-prints, June 2016. URL https://arxiv.org/abs/1606.04838 . — [arXiv:1606.04838](https://arxiv.org/abs/1606.04838) ↗
  9. Cettolo, Mauro, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. Overview of the IWSLT 2017 Evaluation Campaign. In Proceedings of the 14th International Workshop on Spoken Language Translation (IWSLT), pages 2–14, Tokyo,… find our card
  10. Goyal, Priya, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. CoRR, 2017. URL http://arxiv.org/abs/1706.02677 . ↗
  11. Hoffer, Elad, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In Guyon, I., U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing… find
  12. Ioffe, Sergey and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. CoRR, abs/1502.03167, 2015. URL http://arxiv.org/ abs/1502.03167 . ↗
  13. Jastrzebski, Stanislaw, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos J. Storkey. Three Factors Influencing Minima in SGD. CoRR, abs/1711.04623, 2017. URL http://arxiv.org/abs/1711.04623 . — [arXiv:1711.04623](https://arxiv.org/abs/1711.04623) ↗
  14. Keskar, Nitish Shirish, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. In Proceedings of ICLR, 2017. URL http://arxiv.org/abs/1609.04836 . ↗
  15. Krizhevsky, Alex. One weird trick for parallelizing convolutional neural networks. CoRR, abs/1404.5997, 2014. URL http://arxiv.org/abs/1404.5997 . ↗
  16. Lee, Jason, Kyunghyun Cho, and Thomas Hofmann. Fully Character-Level Neural Machine Translation without Explicit Segmentation. CoRR, 2016. URL http://arxiv.org/abs/1610. 03017 . ↗ our card
  17. Lei Ba, J., J. R. Kiros, and G. E. Hinton. Layer Normalization. ArXiv e-prints, July 2016. find our card
  18. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of ACL 2002, pages 311–318, Philadelphia, Pennsylvania, 2002. find our card
  19. Popović, Maja. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal, September 2015. ACL. URL http://aclweb.org/anthology/W15-3049 . ↗ our card
  20. Sennrich, Rico, Barry Haddow, and Alexandra Birch. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of ACL 2016, pages 1715–1725, Berlin, Germany, August 2016. ACL. URL http://www.aclweb.org/anthology/P16-1162 . ↗
  21. Shazeer, N. and M. Stern. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. ArXiv e-prints, Apr. 2018. URL https://arxiv.org/abs/1804.04235 . ↗
  22. Smith, Samuel L. and Quoc V. Le. A Bayesian Perspective on Generalization and Stochastic Gradient Descent. In Proceedings of Second workshop on Bayesian Deep Learning (NIPS 2017), Long Beach, CA, USA, 2017. URL http://arxiv.org/abs/1710.06451 . — [arXiv:1710.06451](https://arxiv.org/abs/1710.06451) ↗
  23. Smith, Samuel L., Pieter-Jan Kindermans, and Quoc V. Le. Don’t Decay the Learning Rate, Increase the Batch Size. CoRR, 2017. URL http://arxiv.org/abs/1711.00489 . ↗
  24. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Guyon, I., U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing… find
  25. Wu, Yonghui, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian,… find
  26. You, Yang, Igor Gitman, and Boris Ginsburg. Scaling SGD Batch Size to 32K for ImageNet Training. CoRR, abs/1708.03888, 2017. URL http://arxiv.org/abs/1708.03888 . — [arXiv:1708.03888](https://arxiv.org/abs/1708.03888) ↗
  27. Address for correspondence: Martin Popel popel@ufal.mff.cuni.cz Institute of Formal and Applied Linguistics Faculty of Mathematics and Physics, Charles University Malostranské náměstí 25, 118 00 Praha 1 Czech Republic find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/popel-2018-training-tips-transformer-model/