Non-Autoregressive Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1711.02281

Summary

Standard neural machine translation systems produce target words one at a time, each conditioned on earlier outputs. This makes decoding sequential and slow, even for models like the Transformer that train in parallel. The authors propose the Non-Autoregressive Transformer (NAT), which generates all target tokens simultaneously. Simply removing the sequential dependency fails because of the "multimodality problem": independent per-position predictions can mix pieces of incompatible valid translations. To address this, NAT adds a fertility predictor to the encoder. Fertilities are counts of how many target words align to each source word, derived from IBM Model 2-style alignment. They act as a latent sentence plan, and source inputs are copied into the decoder according to them. Training combines knowledge distillation from an autoregressive Transformer teacher, supervised fertility prediction and policy-gradient fine-tuning. At inference, noisy parallel decoding samples several fertility sequences, decodes them concurrently, and lets the teacher pick the best candidate. Experiments show roughly tenfold lower latency at a modest BLEU cost, an early demonstration that parallel decoding can be competitive.

Key points

Experiments cover English–German from IWSLT 2016 plus a pair of WMT translation tasks. The best reported result is 29.8 BLEU on WMT 2016 English–Romanian, close to the state of the art at the time.
The quality gap relative to the autoregressive Transformer teacher is as small as 2.0 BLEU. Latency is about one-tenth that of an equally sized autoregressive model.
Each of the three training components (distillation, fertility latent variable, policy-gradient fine-tuning) contributes a cumulative improvement.
The decoder replaces causal masking with self-attention that only blocks each position from attending to itself, and adds a positional attention module to strengthen ordering signals.
On the IWSLT development set, NAT latency stays nearly constant across typical sentence lengths, even with noisy parallel decoding of 10 samples. With 100 samples the GPU saturates and latency grows linearly again.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

311 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 300 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1711.02281 · arXiv 1711.02281

@misc{gu2017non,
  title = {{Non-Autoregressive Neural Machine Translation}},
  author = {Gu, Jiatao and Bradbury, James and Xiong, Caiming and Li, Victor O.K. and Socher, Richard},
  year = {2017},
  doi = {10.48550/arxiv.1711.02281},
  eprint = {1711.02281},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1711.02281}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. find our card
Peter Brown, Vincent della Pietra, Stephen della Pietra, and Robert Mercer. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311, 1993. find
Kyunghyun Cho. Noisy parallel approximate decoding for conditional recurrent language model. arXiv preprint arXiv:1605.03835, 2016. — [arXiv:1605.03835](https://arxiv.org/abs/1605.03835) find
Chris Dyer, Victor Chahuneau, and Noah Smith. A simple, fast, and effective reparameterization of IBM Model 2. In NAACL, 2013. find
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122, 2017. find our card
Łukasz Kaiser, Aidan Gomez, and François Chollet. Depthwise separable convolutions for neural machine translation. arXiv preprint arXiv:1706.03059, 2017. — [arXiv:1706.03059](https://arxiv.org/abs/1706.03059) find
Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukçuoǧlu. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099, 2016. — [arXiv:1610.10099](https://arxiv.org/abs/1610.10099) find
Yoon Kim and Alexander Rush. Sequence-level knowledge distillation. In EMNLP, 2016. find our card
Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. arXiv preprint arXiv:1706.03872, 2017. find
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. In EMNLP, 2015. find
Randi Martin, Jason Crowther, Meredith Knight, Franklin Tamborello, and Chin-Lung Yang. Planning in sentence production: Evidence for the phrase as a default planning scope. Cognition, 116 (2):177–192, 2010. find
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In ACL, pp. 311–318, 2002. find our card
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. find
Ilya Sutskever, Oriol Vinyals, and Quôc Lê. Sequence to sequence learning with neural networks. In NIPS, 2014. find
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017. find
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229–256, 1992. find
Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, Ł. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corra… find