Elamur Research⌕
☰
home / papers / 2020 / Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space

Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space

Paper · 2020 · Chunyuan Li, Xiang Gao, Yuan Li et al.

Summary

Variational Autoencoders (VAEs) can learn compact sentence representations that allow controllable generation. However, language VAEs have usually been built on shallow networks such as two-layer LSTMs, which limits their capacity. They also suffer from KL vanishing. Optimus tackles this by building the first VAE at the scale of modern pre-trained language models. A BERT-based encoder maps a sentence into a latent vector, and a GPT-2 decoder generates text conditioned on that vector. The model is pre-trained on a large text corpus with sentence-level autoencoder and VAE objectives, where a β weight on the KL term controls regularization. It is then fine-tuned for downstream tasks. The experiments evaluate the model on:

  • modeling language
  • generating responses in dialog
  • transferring the style of text
  • understanding language when resources are scarce

Optimus sets new state-of-the-art results on VAE language modeling benchmarks and allows generation to be steered through latent vectors. When the encoder is frozen, its representations transfer better than BERT's. The work matters because it shows that deep generative models remain practical in the era of BERT and GPT-2.

Key points

  • Architecture: a BERT encoder is linked to a GPT-2 decoder through a latent space. Two schemes are proposed for injecting latent vectors into GPT-2 without re-training it.
  • Language modeling: large-scale pre-training substantially reduces KL vanishing. The model reaches state-of-the-art results on four VAE language modeling datasets and lower perplexity than GPT-2 on standard benchmarks.
  • Yelp sentiment classification: training sets ranged from 1 to 10K samples per class. In the feature-based setting, Optimus consistently beats BERT. With full fine-tuning, it adapts faster when labels are scarce and performs similarly to BERT with more data. tSNE plots show a smoother, more clearly clustered latent space.
  • GLUE (nine datasets): with frozen features, Optimus outperforms BERT, most clearly on MNLI, QQP and QNLI. Under full fine-tuning the two models are roughly equal.
  • Limitations: the authors consider the model under-trained and able to control only sentences of moderate length. Code and pre-trained models are available at github.com/ChunyuanLI/Optimus.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States240 works · 123,537 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang and Jianfeng Gao. 2020. Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.378

@inproceedings{li2020optimus,
  title = {{Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space}},
  author = {Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang and Jianfeng Gao},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.378},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.378}
}

References

66 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326. find
  2. Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. CONLL. our card
  3. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. our card
  4. Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. 2015. Importance weighted autoencoders. ICLR. find
  5. Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level Bleu. In Proceedings of the Ninth Workshop on Statistical Machine Translation. our card
  6. Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. 2018. Isolating sources of disentanglement in VAEs. NIPS. find
  7. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NIPS. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL. our card
  9. Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. 2018. Avoiding latent variable collapse with generative skip models. AISTATS. find
  10. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. NeurIPS. our card
  11. Yu Duan, Jiaxin Pei, Canwen Xu, and Chenliang Li. 2019. Pre-train and plug-in: Flexible conditional text generation with variational auto-encoders. arXiv preprint arXiv:1911.03882. find
  12. Xiang Gao, Yizhe Zhang, Sungjin Lee, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2019b. Structuring latent spaces for stylized response generation. EMNLP. find
  13. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. MIT press Cambridge. find
  14. Xiaodong Gu, Kyunghyun Cho, Jungwoo Ha, and Sunghun Kim. 2019. DialogWAE: Multimodal response generation with conditional wasserstein autoencoder. ICLR. find
  15. Suchin Gururangan, Tam Dang, Dallas Card, and Noah A Smith. 2019. Variational pretraining for semi-supervised text classification. arXiv preprint arXiv:1906.02242. find
  16. Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Lagging inference networks and posterior collapse in variational autoencoders. ICLR. find
  17. Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR. find
  18. Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. Neural computation. find
  19. Matthew D Hoffman and Matthew J Johnson. 2016. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS. find
  20. Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. ICML. our card
  21. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858. our card
  22. Yoon Kim, Sam Wiseman, Andrew C Miller, David Sontag, and Alexander M Rush. 2018a. Semi-amortized variational autoencoders. ICML. find
  23. Yoon Kim, Sam Wiseman, and Alexander M Rush. 2018b. A tutorial on deep latent variable models of natural language. arXiv preprint arXiv:1812.06834. — [arXiv:1812.06834](https://arxiv.org/abs/1812.06834) find
  24. Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. 2019. Implicit deep latent variable models for text generation. EMNLP. find
  25. Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. ICLR. find
  26. Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, Lawrence Carin, et al. 2019. Cyclical annealing schedule: A simple approach to mitigating KL vanishing. NAACL. find
  27. Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improved variational inference with inverse autoregressive flow. In NIPS. find
  28. Xiang Gao, Sungjin Lee, Yizhe Zhang, Chris Brockett, Michel Galley, Jianfeng Gao, and Bill Dolan. 2019a. Jointly optimizing diversity and relevance in neural response generation. NAACL. find
  29. Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In NIPS. find
  30. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In ICML. our card
  31. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. our card
  32. Bohan Li, Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick, and Yiming Yang. 2019. A surprisingly effective fix for deep latent variable modeling of text. EMNLP. find
  33. Chunyuan Li, Hao Liu, Changyou Chen, Yuchen Pu, Liqun Chen, Ricardo Henao, and Lawrence Carin. 2017a. ALICE: Towards understanding adversarial learning for joint distribution matching. In NIPS. find
  34. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. ACL. find
  35. Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017b. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547. our card
  36. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017c. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957. our card
  37. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  38. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. JMLR. find
  39. Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. 2016. Adversarial autoencoders. ICLR workshop. find
  40. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics. our card
  41. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In InterSpeech. find
  42. Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784. — [arXiv:1411.1784](https://arxiv.org/abs/1411.1784) find
  43. Yookoon Park, Jaemin Cho, and Gunhee Kim. 2018. A hierarchical latent structure for variational conversation modeling. arXiv preprint arXiv:1804.03424. find
  44. Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. 2020a. Soloist: Few-shot task-oriented dialog with a single pre-trained auto-regressive model. arXiv preprint arXiv:2005.05298. find
  45. Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. 2020b. Few-shot natural language generation for task-oriented dialog. arXiv preprint arXiv:2002.12328. our card
  46. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. our card
  47. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog. find
  48. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. our card
  49. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. ICML. our card
  50. Iulian Vlad Serban, Alessandro S., Yoshua Bengio, Aaron C Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In AAAI. our card
  51. Iulian Vlad Serban, A. Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In AAAI. our card
  52. Dinghan Shen, Asli Celikyilmaz, Yizhe Zhang, Liqun Chen, Xin Wang, Jianfeng Gao, and Lawrence Carin. 2019. Towards generating long and coherent text with multi-level latent variable models. ACL. find
  53. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In NIPS. our card
  54. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053. find
  55. Sandeep Subramanian, Sai Rajeswar Mudumba, Alessandro Sordoni, Adam Trischler, Aaron C Courville, and Chris Pal. 2018. Towards text generation with adversarially learned neural outlines. In NeurIPS. find
  56. Naftali Tishby, Fernando C Pereira, and William Bialek. 2000. The information bottleneck method. arXiv preprint physics/0004057. find
  57. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. our card
  58. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. ICLR. our card
  59. Tianming Wang and Xiaojun Wan. 2019. T-CVAE: Transformer-based conditioned variational autoencoder for story completion. In IJCAI. find
  60. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. NeurIPS. our card
  61. Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. ICML. find
  62. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Processing Systems. our card
  63. Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M Rush, and Yann LeCun. 2018. Adversarially regularized autoencoders. ICML. find
  64. Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. ACL. our card
  65. Qile Zhu, Wei Bi, Xiaojiang Liu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. 2020. A batch normalized inference network keeps the kl vanishing away. arXiv preprint arXiv:2004.12585. find
  66. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In ACM SIGIR. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2020-optimus-organizing-sentences-pre-trained/