Elamur Research
☰
home / papers / 2017 / Toward Controlled Generation of Text

Toward Controlled Generation of Text

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1703.00955

Summary

Neural text generators based on VAEs or GANs usually produce sentences whose properties are random and cannot be steered. Two obstacles cause this: discrete tokens block gradients from sentence-level critics, and latent codes tend to be entangled. Hu et al. split the latent space into an unstructured vector z and a structured code c. Each element of c is assigned one attribute, such as sentiment or tense. A VAE decoder is trained alongside per-attribute classifiers that judge whole generated sentences. Gradients reach the decoder through an annealed softmax relaxation of word choices. The encoder is reused to check that z survives changes to c, which limits leakage between attributes. Training follows an extended wake-sleep scheme in which synthetic sentences become extra labeled data for both generator and classifiers. Each classifier can be trained on its own dataset, so attributes never annotated together can still be combined. On short movie-review sentences, the model controls sentiment and tense using little supervision. This made it an early, influential step toward controllable text generation and style transfer.

Key points

  • Architecture: a VAE is extended with holistic attribute discriminators. An independency constraint (Eq. 7) uses the encoder to keep unspecified features fixed when an attribute code is changed. Removing this constraint visibly degrades disentanglement in the paper's examples.
  • Training: a continuous softmax approximation, annealed toward discrete outputs, makes end-to-end learning possible. An extended sleep phase updates both the generator and the discriminators using generated samples.
  • Data and supervision: no corpus labels sentiment and tense together, so the authors use separate datasets. SST-full serves for sentiment, and tense is learned from word-level labels only. Roughly hundreds of annotated sentences suffice.
  • Evaluation: pretrained classifiers act as judges. The model beats earlier generative models on attribute accuracy, and its generated samples also improve classifiers as augmentation data. The authors report that output is limited to short sentences and show some failure cases.
  • Publication and code: the paper appeared at ICML 2017 (PMLR 70). An adapted implementation is available in the Texar toolkit under examples/text_style_transfer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

394 papers of our library cite it (of the 140673 whose bibliographies we read); 27 of them have a page here, 367 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1703.00955 · arXiv 1703.00955

@misc{hu2017toward,
  title = {{Toward Controlled Generation of Text}},
  author = {Hu, Zhiting and Yang, Zichao and Liang, Xiaodan and Salakhutdinov, Ruslan and Xing, Eric P.},
  year = {2017},
  doi = {10.48550/arxiv.1703.00955},
  eprint = {1703.00955},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1703.00955}
}

References

34 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. find our card
  2. Bowman, Samuel R, Vilnis, Luke, Vinyals, Oriol, Dai, Andrew M, Jozefowicz, Rafal, and Bengio, Samy. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015. find our card
  3. Chen, Xi, Duan, Yan, Houthooft, Rein, Schulman, John, Sutskever, Ilya, and Abbeel, Pieter. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2172–2180, 2016. find
  4. Diao, Qiming, Qiu, Minghui, Wu, Chao-Yuan, Smola, Alexander J, Jiang, Jing, and Wang, Chong. Jointly modeling aspects, ratings and sentiments for movie recommendation (JMARS). In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 193–202. ACM, 201… find
  5. Dosovitskiy, Alexey and Brox, Thomas. Generating images with perceptual similarity metrics based on deep networks. arXiv preprint arXiv:1602.02644, 2016. — [arXiv:1602.02644](https://arxiv.org/abs/1602.02644) find
  6. Goodfellow, Ian, Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron, and Bengio, Yoshua. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672–2680, 2014. find
  7. Grandvalet, Yves, Bengio, Yoshua, et al. Semi-supervised learning by entropy minimization. In NIPS, volume 17, pp. 529–536, 2004. find
  8. Hinton, Geoffrey E, Dayan, Peter, Frey, Brendan J, and Neal, Radford M. The “wake-sleep” algorithm for unsupervised neural networks. Science, 268(5214):1158, 1995. find
  9. Hu, Zhiting, Ma, Xuezhe, Liu, Zhengzhong, Hovy, Eduard, and Xing, Eric. Harnessing deep neural networks with logic rules. In ACL, 2016a. find
  10. Hu, Zhiting, Yang, Zichao, Salakhutdinov, Ruslan, and Xing, Eric P. Deep neural networks with massive learned knowledge. In EMNLP, 2016b. find
  11. Hu, Zhiting, Yang, Zichao, Salakhutdinov, Ruslan, and Xing, Eric P. On unifying deep generative models. arXiv preprint arXiv:1706.00550, 2017. — [arXiv:1706.00550](https://arxiv.org/abs/1706.00550) find
  12. Hu, Zhiting, Shi, Haoran, Yang, Zichao, Tan, Bowen, Zhao, Tiancheng, He, Junxian, Wang, Wentao, Yu, Xingjiang, Qin, Lianhui, Wang, Di, Ma, Xuezhe, Liu, Hector, Liang, Xiaodan, Zhu, Wanrong, Sachan, Devendra Singh, and Xing, Eric. Texar: A modularized, versatile, and extensible toolkit for text gener… find
  13. Kingma, Diederik P and Welling, Max. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013. find our card
  14. Kingma, Diederik P, Mohamed, Shakir, Rezende, Danilo Jimenez, and Welling, Max. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems, pp. 3581–3589, 2014. find
  15. Kusner, Matt and Hernndez-Lobato, Jos. GANs for sequences of discrete elements with the Gumbel-softmax distribution. arXiv preprint arXiv:1611.04051, 2016. — [arXiv:1611.04051](https://arxiv.org/abs/1611.04051) find
  16. Larsen, Anders Boesen Lindbo, Sønderby, Søren Kaae, and Winther, Ole. Autoencoding beyond pixels using a learned similarity metric. In ICML, 2016. find
  17. Mikolov, Tomas, Karafiát, Martin, Burget, Lukas, Cernockỳ, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In Interspeech, volume 2, pp. 3, 2010. find
  18. Odena, Augustus, Olah, Christopher, and Shlens, Jonathon. Conditional image synthesis with auxiliary classifier GANs. arXiv preprint arXiv:1610.09585, 2016. — [arXiv:1610.09585](https://arxiv.org/abs/1610.09585) find
  19. Radford, Alec, Metz, Luke, and Chintala, Soumith. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. find our card
  20. Reed, Scott, Lee, Honglak, Anguelov, Dragomir, Szegedy, Christian, Erhan, Dumitru, and Rabinovich, Andrew. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014. — [arXiv:1412.6596](https://arxiv.org/abs/1412.6596) find
  21. Siddharth, N., Paige, Brooks, Desmaison, Alban, Meent, Jan-Willem van de, Wood, Frank, Goodman, Noah D., Kohli, Pushmeet, and Torr, Philip H.S. Learning disentangled representations in deep generative models. 2017. find
  22. Socher, Richard, Perelygin, Alex, Wu, Jean Y, Chuang, Jason, Manning, Christopher D, Ng, Andrew Y, Potts, Christopher, et al. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), v… find
  23. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104–3112, 2014. find
  24. Taigman, Yaniv, Polyak, Adam, and Wolf, Lior. Unsupervised cross-domain image generation. In ICLR, 2017. find
  25. Tang, Shuai, Jin, Hailin, Fang, Chen, and Wang, Zhaowen. Unsupervised sentence representation learning with adversarial auto-encoder. 2016. van den Oord, Aaron, Kalchbrenner, Nal, and Kavukcuoglu, Koray. Pixel recurrent neural networks. In ICML, 2016. find
  26. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156– 3164, 2015. find our card
  27. Wen, Tsung-Hsien, Gasic, Milica, Mrksic, Nikola, Su, Pei-Hao, Vandyke, David, and Young, Steve. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. In EMNLP, 2015. find
  28. Wilson, Theresa, Wiebe, Janyce, and Hoffmann, Paul. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the conference on human language technology and empirical methods in natural language processing, pp. 347–354. Association for Computational Linguistics, 2005. find
  29. Yang, Zichao, Hu, Zhiting, Salakhutdinov, Ruslan, and Berg-Kirkpatrick, Taylor. Improved variational autoencoders for text modeling using dilated convolutions. In ICML, 2017. find
  30. Young, Steve, Gašić, Milica, Thomson, Blaise, and Williams, Jason D. POMDP-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101 (5):1160–1179, 2013. find
  31. Yu, Lantao, Zhang, Weinan, Wang, Jun, and Yu, Yong. SeqGAN: sequence generative adversarial nets with policy gradient. In AAAI, 2017. find
  32. Zhang, Yizhe, Gan, Zhe, and Carin, Lawrence. Generating text via adversarial training. In NIPS Workshop on Adversarial Training, 2016. find
  33. Zhou, Chunting and Neubig, Graham. Multi-space variational encoder-decoders for semi-supervised labeled sequence transduction. In ACL, 2017. find
  34. Zhu, Jun-Yan, Krähenbühl, Philipp, Shechtman, Eli, and Efros, Alexei A. Generative visual manipulation on the natural image manifold. In European Conference on Computer Vision, pp. 597–613. Springer, 2016. find