Elamur Research
☰
home / papers / 2018 / Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement

Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1149.pdf

Summary

Autoregressive sequence models such as the Transformer generate output one token at a time. Decoding is therefore slow and hard to parallelize, and beam search adds further cost. Lee, Mansimov and Cho propose a deterministic non-autoregressive model that predicts the whole target sequence at once and then improves it over several passes. They frame the model in two ways: as a latent variable model whose intermediate variables are draft outputs, and as a conditional denoising autoencoder. Training combines both views by randomly switching between lower-bound terms and reconstruction from corrupted references. It also uses knowledge distillation from an autoregressive teacher and a separate target-length predictor. At inference time, a single shared refinement network runs repeatedly until the output stops changing, so the number of steps adapts to each input. On machine translation, the model decodes about 2–3× faster than autoregressive beam search and retains 90–95% of its quality. On MS COCO captioning, it keeps about 85% of the quality. This shows that parallel decoding can offer a practical speed–quality trade-off for any sequence generation task.

Key points

  • Evaluated on IWSLT'16 En↔De, WMT'16 En↔Ro and WMT'14 En↔De for translation, and on MS COCO for image captioning. The baseline is a Transformer.
  • Translation decoding is roughly 2–3× faster than beam search on both GPU and CPU while preserving 90–95% of translation quality. Captioning is about 3× faster on GPU and 5× faster on CPU, with roughly 85% of caption quality.
  • The training objective mixes the latent-variable loss and the denoising loss per refinement step using a Bernoulli variable with probability p_DAE. The corruption process, adapted from Hill et al. (2016), repeats tokens, substitutes random tokens or swaps adjacent tokens.
  • Refinement stops once the Jaccard distance between consecutive outputs falls below a threshold. Qualitatively, each pass adds missing words and removes spurious ones.
  • The code, preprocessed data and trained models are released at github.com/nyu-dl/dl4mt-nonauto.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

183 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 179 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1149 · arXiv 1802.06901

@inproceedings{lee2018deterministic,
  title = {{Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement}},
  author = {Lee, Jason and Mansimov, Elman and Cho, Kyunghyun},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1149},
  eprint = {1802.06901},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d18-1149}
}

References

38 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Guillaume Alain and Yoshua Bengio. 2014. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1). find
  2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041. find our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  4. Florian Bordes, Sina Honari, and Pascal Vincent. 2017. Learning to generate samples from noise through infusion training. In ICLR. find
  5. Chung-Cheng Chiu, Tara N Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron J Weiss, Kanishka Rao, Katya Gonina, et al. 2017. State-of-the-art speech recognition with sequence-to-sequence models. arXiv preprint arXiv:1712.01769. — [arXiv:1712.01769](https://arxiv.org/abs/1712.01769) find
  6. Kyunghyun Cho. 2016. Noisy parallel approximate decoding for conditional recurrent language model. arXiv preprint arXiv:1605.03835. — [arXiv:1605.03835](https://arxiv.org/abs/1605.03835) find
  7. Kyunghyun Cho, Aaron Courville, and Yoshua Bengio. 2015. Describing multimedia content using attention-based encoder-decoder networks. IEEE Transactions on Multimedia, 17(11). find
  8. Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-based models for speech recognition. In NIPS. find our card
  9. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR. find
  10. Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In ACL. find
  11. David Grangier and Michael Auli. 2017. Quickedit: Editing text & translations via simple delete actions. arXiv preprint arXiv:1711.04805. — [arXiv:1711.04805](https://arxiv.org/abs/1711.04805) find
  12. Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2017. Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02281. find our card
  13. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. find
  14. Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In NAACL. find our card
  15. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. find our card
  16. Cong Duy Vu Hoang, Gholamreza Haffari, and Trevor Cohn. 2017. Decoding as continuous optimization in neural machine translation. arXiv preprint arXiv:1701.02854. — [arXiv:1701.02854](https://arxiv.org/abs/1701.02854) find
  17. Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. 2017. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Archite… find
  18. Łukasz Kaiser and Samy Bengio. 2016. Can active memory replace attention? In NIPS. find
  19. Łukasz Kaiser and Ilya Sutskever. 2015. Neural GPUs learn algorithms. arXiv preprint arXiv:1511.08228. find
  20. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP. find
  21. Andrej Karpathy and Fei-Fei Li. 2015. Deep visual-semantic alignments for generating image descriptions. In CVPR. find our card
  22. Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947. find our card
  23. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  24. Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improved variational inference with inverse autoregressive flow. In NIPS. find
  25. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In ACL. find
  26. Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043. find our card
  27. T.Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. 2014. Microsoft COCO: Common objects in context. In ECCV. find our card
  28. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association. find
  29. Roman Novak, Michael Auli, and David Grangier. 2016. Iterative refinement for machine translation. arXiv preprint arXiv:1610.06602. — [arXiv:1610.06602](https://arxiv.org/abs/1610.06602) find
  30. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499. find our card
  31. Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C Cobo, Florian Stimberg, et al. 2017. Parallel wavenet: Fast high-fidelity speech synthesis. arXiv preprint arXiv:1711.10433. — [arXiv:1711.10433](https://arxiv.org/abs/1711.10433) find
  32. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL. find our card
  33. Holger Schwenk. 2012. Continuous space translation models for phrase-based statistical machine translation. Proceedings of COLING 2012: Posters. find
  34. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL. find
  35. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. arXiv preprint arXiv:1505.00387. find our card
  36. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS. find
  37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. find
  38. Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017. Deliberation networks: Sequence generation beyond one-pass decoding. In NIPS. find