Elamur Research
☰
home / papers / 2016 / Generating Sentences from a Continuous Space

Generating Sentences from a Continuous Space

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/K16-1002.pdf

Summary

Conventional RNN language models build a sentence token by token and never form one explicit vector for the whole sentence. As a result, global properties such as topic, style or broad syntactic shape stay hidden. Bowman et al. adapt the variational autoencoder to text to address this. A single-layer LSTM encoder maps a sentence to a Gaussian latent code, and an LSTM decoder rebuilds the sentence from that code. Plain training tends to collapse: the posterior matches the prior, the KL term drops to zero, and the model acts like an ordinary RNNLM. The authors propose training techniques to prevent this, including word dropout in the decoder. They evaluate the model on Penn Treebank language modeling, on imputing missing words in a larger fiction corpus (scored with an adversarial classifier), and on TREC question classification. Perplexity is roughly the same as a standard RNNLM, so the language modeling result is negative. However, the latent variable helps with word imputation. Samples drawn from the prior and decoded deterministically are varied and grammatical. Paths between two sentences in latent space pass through plausible intermediate sentences. This showed that continuous sentence spaces can support controllable generation.

Key points

  • Architecture: LSTM encoder and decoder with a diagonal-Gaussian latent code, trained with the reparameterization trick. Variants with feedforward layers between the encoder, latent code and decoder only trained when 4-layer highway networks were used.
  • Alternatives tried: More elaborate posteriors, DRAW-style multistep sampling and normalizing flows, gave no clear improvement over the plain VAE.
  • Penn Treebank hyperparameters: These were found by Bayesian tuning over 200 runs. The standard VAE used a 13-dimensional latent code and a word-dropout keep rate of 0.62.
  • TREC Question Classification accuracy:
    • VAE codes: 87.0
    • Sequence autoencoder: 84.2
    • CBOW: 87.3
    • Skip-thought: 91.4
    • Combine-ST: 92.2
    • VAE + combine-ST: 92.0
  • Comparison with a plain autoencoder: Interpolating between sentence codes of a plain sequence autoencoder produces ungrammatical intermediate sentences. VAE paths stay grammatical and locally consistent in topic and vocabulary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

390 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 374 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k16-1002 · arXiv 1511.06349

@inproceedings{bowman2016generating,
  title = {{Generating Sentences from a Continuous Space}},
  author = {Bowman, Samuel R. and Vilnis, Luke and Vinyals, Oriol and Dai, Andrew and Jozefowicz, Rafal and Bengio, Samy},
  booktitle = {Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning},
  year = {2016},
  doi = {10.18653/v1/k16-1002},
  eprint = {1511.06349},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/k16-1002}
}

References

36 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proc. ACL. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR. find our card
  3. Michael Kearns, Yishay Mansour, and Andrew Y Ng. 1998. An information-theoretic analysis of hard and soft assignment methods for clustering. In Learning in graphical models, Springer, pages 495–520. find
  4. Justin Bayer and Christian Osendorfer. 2015. Learning stochastic recurrent networks. arXiv preprint arXiv:1411.7610. — [arXiv:1411.7610](https://arxiv.org/abs/1411.7610) find
  5. Julian Besag. 1986. On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society Series B (Methodological) pages 48–259. find
  6. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proc. EMNLP . find our card
  7. Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. 2015. A recurrent latent variable model for sequential data. In Proc. NIPS . find
  8. Diederik P. Kingma and Max Welling. 2015. Autoencoding variational bayes. In Proc. ICLR. find
  9. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Proc. NIPS . find
  10. Andrew M. Dai and Quoc V. Le. 2015. Semi-supervised sequence learning. In Proc. NIPS . find
  11. Emily Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. 2015. Deep generative image models using a laplacian pyramid of adversarial networks. In Proc. NIPS . find
  12. Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In Proc. ICML. find our card
  13. Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proc. COLING. find our card
  14. Quoc V. Le and Tomáš Mikolov. 2014. Distributed representations of sentences and documents. In Proc. ICML. find our card
  15. Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent convolutional networks for visual recognition and description. In Proc. CVPR. find our card
  16. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015a. A hierarchical neural autoencoder for paragraphs and documents. In Proc. ACL. find
  17. Xin Li and Dan Roth. 2002. Learning question classifiers. In Proc. COLING. find
  18. Otto Fabius and Joost R. van Amersfoort. 2014. Variational recurrent auto-encoders. arXiv preprint arXiv:1412.6581. — [arXiv:1412.6581](https://arxiv.org/abs/1412.6581) find
  19. Yujia Li, Kevin Swersky, and Richard Zemel. 2015b. Generative moment matching networks. In Proc. ICML. find
  20. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proc. NIPS . find
  21. Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. 2015. Deep captioning with multimodal recurrent neural networks (m-RNN). In Proc. ICLR. find
  22. Karol Gregor, Ivo Danihelka, Alex Graves, and Daan Wierstra. 2015. DRAW: A recurrent neural network for image generation. In Proc. ICML. find
  23. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational linguistics 19(2):313–330. find our card
  24. Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In Proc. ICML. find
  25. Tomáš Mikolov, Stefan Kombrink, Lukáš Burget, Jan Honza Černockỳ, and Sanjeev Khudanpur. 2011. Extensions of recurrent neural network language model. In Proc. ICASSP . find
  26. Tapani Raiko, Mathias Berglund, Guillaume Alain, and Laurent Dinh. 2015. Techniques for learning binary stochastic feedforward neural networks. In Proc. ICLR. find
  27. Danilo J. Rezende and Shakir Mohamed. 2015. Variational inference with normalizing flows. In Proc. ICML. find
  28. Danilo J. Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proc. ICML. find our card
  29. Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian optimization of machine learning algorithms. In Proc. NIPS . find
  30. Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D. Manning, and Andrew Y. Ng. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Proc. NIPS . find
  31. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. JMLR 15(1):1929–1958. find
  32. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In Proc. NIPS . find
  33. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proc. NIPS . find
  34. Lucas Theis and Matthias Bethge. 2015. Generative image modeling using spatial LSTMs. In Proc. NIPS . find
  35. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proc. CVPR. find our card
  36. Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015. Self-adaptive hierarchical sentence model. In Proc. IJCAI . find