Elamur Research
☰
home / papers / 2018 / The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation

The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1008.pdf

Summary

New neural MT architectures such as ConvS2S and the Transformer usually come bundled with their own modeling and training techniques. This makes it hard to tell whether their gains come from the architecture or from those extra techniques. The Google AI authors separate the two. They implement all models in one framework, use identical preprocessing, and evaluate on the WMT'14 English→French and English→German benchmarks.

Ablation studies show that techniques such as multi-head attention, layer normalization, label smoothing and synchronous replica training carry over across architectures. Adding them to a GNMT-style recurrent model produces RNMT+, which outperforms the RNMT, ConvS2S and Transformer baselines.

The authors then compare the strengths of each model family. A Transformer encoder extracts features better, while the stateful RNMT+ decoder generates text better. Hybrid encoders that combine both components, either stacked (cascaded) or in parallel (multi-column), improve further on RNMT+.

The work argues for controlled, fair architecture comparisons. It also points to reusable techniques and hybrid designs for future NMT architecture search.

Key points

  • Data: WMT'14 En→Fr (36.3M sentence pairs) and En→De (4.5M sentence pairs).

    • Wordpiece segmentation with a shared 32K vocabulary per language pair.
    • Test set: newstest 2014; validation set: newstest 2012 + 2013.
    • Scores are reported as mean BLEU (with standard deviation) over a window of 21 dev-set evaluations after convergence.

  • RNMT+ architecture: 6 bidirectional LSTM encoder layers and 8 unidirectional LSTM decoder layers. The first decoder layer computes the context vector using multi-head additive attention.

  • BLEU (En→Fr / En→De):

    ModelEn→FrEn→De
    Transformer Big40.7327.94
    RNMT+41.0028.49
    Cascaded hybrid41.6728.62
    Multi-column hybrid41.6628.84

  • Encoder-decoder hybrids: A Transformer encoder paired with an RNMT+ decoder works best. The authors take this as support for the value of a stateful decoder.

  • Best cascaded encoder: Transformer layers are fine-tuned on top of a frozen, pre-trained RNMT+ encoder.

  • Best multi-column encoder: The outputs of the separate encoder columns are simply concatenated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

134 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 125 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1008 · arXiv 1804.09849

@inproceedings{chen2018best,
  title = {{The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation}},
  author = {Chen, Mia Xu and Firat, Orhan and Bapna, Ankur and Johnson, Melvin and Macherey, Wolfgang and Foster, George and Jones, Llion and Schuster, Mike and Shazeer, Noam and Parmar, Niki and Vaswani, Ashish and Uszkoreit, Jakob and Kaiser, Lukasz and Chen, Zhifeng and Wu, Yonghui and Hughes, Macduff},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1008},
  eprint = {1804.09849},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p18-1008}
}

References

38 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, MarcAurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng. 2012. Large scale distributed deep networks. In NIPS. find
  2. Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR abs/1607.06450. http://arxiv.org/abs/1607.06450. ↗ our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations. http://arxiv.org/abs/1409.0473. ↗ our card
  4. Michael Denkowski and Graham Neubig. 2017. Stronger baselines for trustable results in neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation. Association for Computational Linguistics, Vancouver, pages 18–27. http://www.aclweb.org/anthology/W17-3203. ↗
  5. Y. Bengio, P. Simard, and P. Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. Trans. Neur. Netw. 5(2):157–166. find
  6. Jacob Devlin. 2017. Sharp models on dull hardware: Fast and accurate neural machine translation decoding on the cpu. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 2820–2825. http://aclweb.org/anthology/D17-… ↗
  7. Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc Le. 2017. Massive exploration of neural machine translation architectures. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, pages 1442–1451.… find
  8. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science 14(2):179 – 211. find
  9. Hugh Chen, Scott Lundberg, and Su-In Lee. 2017. Checkpoint ensembles: Ensemble methods from a single training process. CoRR abs/1710.03282. http://arxiv.org/abs/1710.03282. — [arXiv:1710.03282](https://arxiv.org/abs/1710.03282) ↗
  10. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. CoRR abs/1705.03122. http://arxiv.org/abs/1705.03122. ↗ our card
  11. Jianmin Chen, Rajat Monga, Samy Bengio, and Rafal Józefowicz. 2016. Revisiting distributed synchronous SGD. CoRR abs/1604.00981. http://arxiv.org/abs/1604.00981. — [arXiv:1604.00981](https://arxiv.org/abs/1604.00981) ↗
  12. Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 2000. Learning to forget: Continual prediction with LSTM. Neural computation 12(10):2451–2471. find
  13. Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aäron van den… find
  14. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks 18(5):602 – 610. find
  15. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR abs/1412.6980. http://arxiv.org/abs/1412.6980. ↗
  16. Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. 2015. Learning to transduce with unbounded memory. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2. MIT Press, Cambridge, MA, USA, NIPS’15, pages 1828–1836. http://dl… find
  17. Yann LeCun and Yoshua Bengio. 1998. The handbook of brain theory and neural networks. MIT Press, Cambridge, MA, USA, chapter Convolutional Networks for Images, Speech, and Time Series, pages 255–258. http://dl.acm.org/citation.cfm?id=303568.303704. ↗
  18. Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In EMNLP. find our card
  19. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. CoRR abs/1512.03385. http://arxiv.org/abs/1512.03385. ↗
  20. Razvan Pascanu, Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. 2013. How to construct deep recurrent neural networks. CoRR abs/1312.6026. http://arxiv.org/abs/1312.6026. ↗
  21. Sepp Hochreiter. 1991. Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Universität München 91:1. find
  22. Tim Salimans and Diederik P. Kingma. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. CoRR abs/1602.07868. http://arxiv.org/abs/1602.07868. — [arXiv:1602.07868](https://arxiv.org/abs/1602.07868) ↗
  23. Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jrgen Schmidhuber. 2001. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. find
  24. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780. find
  25. M. Schuster and K.K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing 45(11):2673–2681. find
  26. Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer feedforward networks are universal approximators. Neural Networks 2(5):359 – 366. find
  27. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. 2012 IEEE International Conference on Acoustics, Speech and Signal Processing . find
  28. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language P… find our card
  29. H.T. Siegelmann and E.D. Sontag. 1995. On the computational power of neural nets. Journal of Computer and System Sciences 50(1):132 – 150. find
  30. Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, and et al. 2017. In-datacenter performance analysis of a tensor processing unit. CoRR abs/1704.04760. http://arxiv.org/abs/1704.04760. ↗
  31. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. pages 3104–3112. find
  32. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. CoRR abs/1505.00387. find our card
  33. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2015. Rethinking the inception architecture for computer vision. CoRR abs/1512.00567. http://arxiv.org/abs/1512.00567. ↗
  34. Marcin Junczys-Dowmunt, Tomasz Dwojak, and Rico Sennrich. 2016. The amu-uedin submission to the wmt16 news translation task: Attention-based nmt models as feature functions in phrase-based smt. arXiv preprint arXiv:1605.04809 . find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR abs/1706.03762. http://arxiv.org/abs/1706.03762. ↗
  36. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Conference on Empirical Methods in Natural Language Processing. find
  37. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  38. Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. 2016. Deep recurrent models with fast-forward connections for neural machine translation. CoRR abs/1606.04199. http://arxiv.org/abs/1606.04199. ↗