Elamur Research⌕
☰
home / papers / 2019 / FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow

FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow

Paper · 2019 · Xuezhe Ma, Chunting Zhou, Xian Li et al.

Summary

Autoregressive seq2seq models produce output one token after another, which limits parallel decoding on GPUs. Non-autoregressive models predict all positions at once, but they usually lose accuracy because they struggle to capture how output words depend on each other. FlowSeq addresses this with a sequence of continuous latent vectors, one per target position. The prior over these vectors, conditioned on the source, is modeled with a generative flow adapted from Glow. The model is trained variationally with a diagonal-Gaussian posterior network, and the decoder then predicts every target token independently, without causal masking. The authors evaluated it on the WMT2014, WMT2016 and IWSLT-2014 translation benchmarks. FlowSeq performs on par with leading non-autoregressive systems and stays only slightly behind the contemporaneous CMLM. Its decoding time stays nearly flat as sentences get longer, while a Transformer's grows. The work matters because it is, by the authors' account, the first non-autoregressive seq2seq model built on generative flows. It also shows that an expressive latent prior can recover much of the quality lost by parallel generation.

Key points

  • Each step of the prior flow stacks three invertible components: actnorm, an invertible multi-head linear layer, and a coupling layer. These steps are arranged in a Glow-style multi-scale architecture.
  • Two training techniques are used. Token-level dropout in the posterior stops latents from simply copying their own target token. Zero-initializing the final layers that produce the posterior mean and log-variance stabilizes very deep flows.
  • With a batch size of 128 on a single NVIDIA TITAN X (WMT14 EN-DE test set), FlowSeq decoding sped up by 594% for the base model and 403% for the large model. Transformer with beam search gained less from batching.
  • NPD and IWD rescoring used reduced-temperature sampling, with temperatures from 0.1 to 1.0. The number of samples per length mattered most, and temperatures of 0.3–0.5 worked best.
  • A diversity study on a WMT14 EN-DE subset with ten references used pairwise-BLEU and leave-one-out BLEU. Raising the sampling temperature made outputs more diverse with almost no loss in quality, though FlowSeq still trails autoregressive generation. Code is published at github.com/XuezheMax/flowseq.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🏢MetaMenlo Park · United States158 works · 211,459 citations🏢Meta (United Kingdom)London · United Kingdom39 works · 11,541 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig and Eduard Hovy. 2019. FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1437

@inproceedings{ma2019flowseq,
  title = {{FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow}},
  author = {Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig and Eduard Hovy},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1437},
  url = {https://doi.org/10.18653/v1/d19-1437}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR). our card
  2. Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2015. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349. our card
  3. Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. 2015. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519. — [arXiv:1509.00519](https://arxiv.org/abs/1509.00519) find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. our card
  5. Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. 2016. Density estimation using real nvp. arXiv preprint arXiv:1605.08803. — [arXiv:1605.08803](https://arxiv.org/abs/1605.08803) find
  6. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Constant-time machine translation with conditional masked language models. arXiv preprint arXiv:1904.09324. — [arXiv:1904.09324](https://arxiv.org/abs/1904.09324) find
  7. Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In International Conference on Learning Representations (ICLR). our card
  8. Jiatao Gu, Qi Liu, and Kyunghyun Cho. 2019. Insertion-based decoding with automatically inferred generation order. arXiv preprint arXiv:1902.01370. find
  9. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448–456. find
  10. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  11. Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improving variational inference with inverse autoregressive flow. The 29th Conference on Neural Information Processing Systems. find
  12. Durk P Kingma and Prafulla Dhariwal. 2018. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pages 10215–10224. find
  13. Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1173– 1182. our card
  14. Jindřich Libovickỳ and Jindřich Helcl. 2018. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3016– 3021. our card
  15. Xuezhe Ma and Eduard Hovy. 2019. Macow: Masked convolutional generative flow. arXiv preprint arXiv:1902.04208. — [arXiv:1902.04208](https://arxiv.org/abs/1902.04208) find
  16. Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stack-pointer networks for dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1403–1414. find
  17. Xuezhe Ma, Chunting Zhou, and Eduard Hovy. 2019. Mae: Mutual posterior-divergence regularization for variational autoencoders. In Proceedings of the 7th International Conference on Learning Representations (ICLR-2019), New Orleans, Louisiana, USA. find
  18. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional LSTM. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 51–61, Berlin, Germany. Association for Computational Linguistics. our card
  19. Myle Ott, Michael Auli, David Grangier, et al. 2018. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning, pages 3953–3962. our card
  20. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. our card
  21. George Papamakarios, Theo Pavlakou, and Iain Murray. 2017. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pages 2338–2347. find
  22. Ryan Prenger, Rafael Valle, and Bryan Catanzaro. 2019. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617–3621. IEEE. find
  23. Danilo Jimenez Rezende and Shakir Mohamed. 2015. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volume 37, pages 1530–1538. JMLR. org. find
  24. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389. find
  25. Tianxiao Shen, Myle Ott, Michael Auli, et al. 2019. Mixture models for diverse machine translation: Tricks of the trade. In International Conference on Machine Learning, pages 5719–5728. find
  26. Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. 2019. Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior. arXiv preprint arXiv:1908.07181. — [arXiv:1908.07181](https://arxiv.org/abs/1908.07181) find
  27. Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. Insertion transformer: Flexible sequence generation via insertion operations. arXiv preprint arXiv:1902.03249. find
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. our card
  29. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164. our card
  30. Martin J Wainwright, Michael I Jordan, et al. 2008. Graphical models, exponential families, and variational inference. Foundations and Trends R in Machine Learning, 1(1–2):1–305. find
  31. Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019. Non-autoregressive machine translation with auxiliary regularization. arXiv preprint arXiv:1902.10245. — [arXiv:1902.10245](https://arxiv.org/abs/1902.10245) find
  32. Zachary Ziegler and Alexander Rush. 2019. Latent normalizing flows for discrete sequences. In International Conference on Machine Learning, pages 7673–7682. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2019-flowseq-non-autoregressive-conditional-sequence/