Elamur Research⌕
☰
home / papers / 2019 / Generating Sentences from Disentangled Syntactic and Semantic Spaces

Generating Sentences from Disentangled Syntactic and Semantic Spaces

Paper · 2019 · Yu Bao, Hao Zhou, Shujian Huang et al.

Summary

Standard variational autoencoders for text store all sentence information in one continuous latent vector. Syntax is therefore never represented explicitly and cannot be sampled or edited on its own. The authors present DSS-VAE, which divides a GRU encoder's sentence representation into two Gaussian latent variables: one for meaning and one for structure. The structural variable learns to predict a linearized constituency tree. ZPar supplies these parses during training only. The semantic variable learns to predict a bag of words. Adversarial objectives and an adversarial reconstruction loss keep each space from absorbing the other's content. Against a plain VAE, the model produces more fluent samples (lower perplexity) and higher reconstruction BLEU. It is also compared with the PRPN syntactic language model. Fixing the semantic vector through MAP inference while resampling the syntactic one gives unsupervised paraphrasing that beats both the VAE and the CGMH sampler. Pairing one sentence's meaning vector with another's syntax vector enables syntax transfer. The work matters because sentence structure becomes a controllable dimension of generation without any parallel corpus.

Key points

  • Architecture: two independent latent variables, z_sem and z_syn, each with a N(0, I) prior, are concatenated to initialize the decoder.
    • Syntax signal: a separate RNN predicts the linearized parse sequence, where opening and closing nodes such as NP and /NP have distinct embeddings.
    • Semantics signal: a bag-of-words prediction.
  • Paraphrase generation: DSS-VAE outperforms both the plain VAE and CGMH (Miao et al., 2019) on BLEU-ori and BLEU-ref at once.
  • Syntax-transfer benchmark: the authors built a test set of 1000 non-paraphrase sentence pairs from SNLI.
    • Semantics are scored with word-BLEU differences.
    • Syntax is scored with Tree Edit Distance (zss package).
  • Syntax-transfer results: the plain VAE essentially fails, with a geometric-mean score of 0.29.
    • Multi-task plus adversarial losses raise the score to 12.09.
    • The full model with both adversarial reconstruction losses reaches 15.30 (ΔTED 30.85, Δword-BLEU 7.59).
    • Transfer works better between structurally compatible sentences, for example ones of similar length.
  • Code and models: released at github.com/baoy-nlp/DSS-VAE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WaterlooWaterloo · Canada44 works · 12,783 citations🏢ByteDanceBeijing · China18 works · 3,890 citations🎓Nanjing UniversityNanjing · China28 works · 7,072 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xin-yu Dai and Jiajun Chen. 2019. Generating Sentences from Disentangled Syntactic and Semantic Spaces. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1602

@inproceedings{bao2019generating,
  title = {{Generating Sentences from Disentangled Syntactic and Semantic Spaces}},
  author = {Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xin-yu Dai and Jiajun Chen},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1602},
  url = {https://doi.org/10.18653/v1/p19-1602}
}

References

38 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. find
  2. Ciprian Chelba. 1997. A structured language model. In ACL, pages 498–500. find
  3. Huadong Chen, Shujian Huang, David Chiang, and Jiajun Chen. 2017. Improved neural machine translation with a syntax-aware encoder and decoder. In ACL, pages 1936–1945. our card
  4. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In EMNLP, pages 1724–1734. find
  5. Alexander Clark. 2001. Unsupervised induction of stochastic context-free grammars using distributional clustering. In Proceedings of the ACL 2001 Workshop on Computational Natural Language Learning. find
  6. Hanjun Dai, Yingtao Tian, Bo Dai, Steven Skiena, and Le Song. 2018. Syntax-directed variational autoencoder for structured data. In ICLR. find
  7. Jan Milan Deriu and Mark Cieliebak. 2018. Syntactic manipulation for generating more diverse and interesting texts. In Proceedings of the 11th International Conference on Natural Language Generation, pages 22–34. find
  8. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016. Recurrent neural network grammars. In NAACL, pages 199–209. find
  9. Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-sequence attentional neural machine translation. In ACL, pages 823–833. find
  10. Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In AAAI, pages 663–670. our card
  11. Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Ben-jamı́n Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. 2018. Automatic chemical design using a data-driven continuous representation… find
  12. Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. 2019. CGMH: Constrained sentence generation by Metropolis-Hastings sampling. In AAAI. find
  13. Ankush Gupta, Arvind Agarwal, Prawaan Singh, and Piyush Rai. 2018. A deep generative framework for paraphrase generation. In AAAI, pages 5149–5156. our card
  14. Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In ICML, pages 807–814. find
  15. Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In ICML, pages 1587– 1596. our card
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In ACL, pages 311–318. our card
  17. Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2018. Disentangled representation learning for text style transfer. arXiv preprint arXiv:1808.04339. — [arXiv:1808.04339](https://arxiv.org/abs/1808.04339) find
  18. Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. 2017. A hybrid convolutional variational autoencoder for text generation. In EMNLP, pages 627–637. find
  19. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  20. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In AAAI, pages 3295–3301. our card
  21. Diederik P Kingma and Max Welling. 2014. Autoencoding variational Bayes. In ICLR. find
  22. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A Smith. 2017. What do recurrent neural network grammars learn about syntax? In EACL, pages 1249–1258. find
  23. Matt J. Kusner, Brooks Paige, and José Miguel Hernández-Lobato. 2017. Grammar variational autoencoder. In ICML, pages 1945–1954. find
  24. Juncen Li, Robin Jia, He He, and Percy Liang. 2018a. Delete, Retrieve, Generate: a simple approach to sentiment and style transfer. In ACL, pages 1865– 1874. our card
  25. Junhui Li, Deyi Xiong, Zhaopeng Tu, Muhua Zhu, Min Zhang, and Guodong Zhou. 2017. Modeling source syntax for neural machine translation. In ACL, pages 688–697. find
  26. Juntao Li, Yan Song, Haisong Zhang, Dongmin Chen, Shuming Shi, Dongyan Zhao, and Rui Yan. 2018b. Generating classical chinese poems via conditional variational autoencoder and adversarial training. In EMNLP, pages 3890–3900. find
  27. Lemao Liu, Muhua Zhu, and Shuming Shi. 2018. Improving sequence-to-sequence constituency parsing. In AAAI, pages 4873–4880. find
  28. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH. find
  29. Yikang Shen, Zhouhan Lin, Chin-Wei Huang, and Aaron Courville. 2017. Neural language modeling by jointly learning syntax and lexicon. In ICLR. find
  30. Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. 2017. Adversarial discriminative domain adaptation. In CVPR, pages 7167–7176. find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998–6008. our card
  32. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In NIPS, pages 2773– 2781. find
  33. Rongxiang Weng, Shujian Huang, Zaixiang Zheng, XIN-YU DAI, and CHEN Jiajun. 2017. Neural machine translation with word predictions. In EMNLP, pages 136–145. find
  34. Kaizhong Zhang and Dennis Shasha. 1989. Simple fast algorithms for the editing distance between trees and related problems. SIAM journal on computing, 18(6):1245–1262. find
  35. Shuming Ma, Xu Sun, Yizhong Wang, and Junyang Lin. 2018. Bag-of-words as target for neural machine translation. In ACL, pages 332–338. find
  36. Junbo Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M. Rush, and Yann LeCun. 2018. Adversarially regularized autoencoders. In ICML, pages 5897– 5906. find
  37. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The Penn Treebank. Computational linguistics, 19(2):313–330. our card
  38. Hao Zhou, Zhaopeng Tu, Shujian Huang, Xiaohua Liu, Hang Li, and Jiajun Chen. 2017. Chunk-based bi-scale decoder for neural machine translation. In ACL, pages 580–586. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bao-2019-generating-sentences-disentangled-syntactic-semantic/