Elamur Research
☰
home / papers / 2019 / Plug and Play Language Models: A Simple Approach to Controlled Text Generation

Plug and Play Language Models: A Simple Approach to Controlled Text Generation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1912.02164

Summary

Large pretrained Transformer language models write fluent text, but steering what they produce, such as the topic or sentiment, usually means changing the architecture or retraining on labelled data. This paper introduces the Plug and Play Language Model (PPLM). It keeps a pretrained, unconditional LM (GPT-2 345M) frozen and attaches small attribute models: either a hand-picked bag of words (BoW) or a one-layer classifier trained on the LM's hidden states. At each generation step, gradients from the attribute model change the LM's cached activations so the next tokens fit the target attribute. Extra terms keep the output close to the original distribution so it stays fluent. A strength setting controls how strong the push is, and several controllers can be combined. Automatic and human evaluations show that PPLM text matches the requested attribute much more often than plain GPT-2 and stays fluent. It performs about as well as, and often better than, CTRL and a GPT-2 fine-tuned with RL, without training the LM at all. This makes low-cost, modular control of text generation possible.

Key points

  • The attribute models are tiny: the sentiment discriminator has about 100,000× fewer parameters than GPT-2 345M, and the BoW controllers need no training at all.
  • Control was tested on 7 BoW-defined topics (e.g., Military, Space, Science, Politics) and on positive/negative sentiment. Topic and sentiment can also be combined.
  • The baselines were CTRL (a 1.6B-parameter model trained with fixed control codes), GPT-2 fine-tuned for positivity, weighted decoding and simple reranking.
  • For detoxification, PPLM follows the negative gradient of a toxicity classifier. With adversarial trigger prompts, the toxic output rate fell from 63.6% (318/500) to 4.6% (23/500).
  • The code is released at github.com/uber-research/PPLM. The same approach also supports story writing with a fixed structure.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

430 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 409 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1912.02164 · arXiv 1912.02164

@misc{dathathri2019plug,
  title = {{Plug and Play Language Models: A Simple Approach to Controlled Text Generation}},
  author = {Dathathri, Sumanth and Madotto, Andrea and Lan, Janice and Hung, Jane and Frank, Eric and Molino, Piero and Yosinski, Jason and Liu, Rosanne},
  year = {2019},
  doi = {10.48550/arxiv.1912.02164},
  eprint = {1912.02164},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1912.02164}
}

References

57 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kenn Adams. Improv encyclopedia story spine. http://improvencyclopedia.org/games/Story\_Spine.html. (accessed September 20, 2019). ↗
  2. Ashutosh Baheti, Alan Ritter, Jiwei Li, and Bill Dolan. Generating more interesting responses in neural conversation models with distributional constraints. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3970–3980, 2018. find
  3. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155, 2003. find
  4. Yun Chen, Victor OK Li, Kyunghyun Cho, and Samuel R Bowman. A stable and effective learning strategy for trainable greedy decoding. arXiv preprint arXiv:1804.07915, 2018. find
  5. Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. Style transformer: Unpaired text style transfer without disentangled latent representation. arXiv preprint arXiv:1905.05621, 2019a. find our card
  6. Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019b. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… find
  8. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. HotFlip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 31–36, Melbourne, Australia, July 2018. Association for Com… find our card
  9. Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 11–21, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D1… find our card
  10. Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833, 2018. find
  11. Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pp. 94–104, 2017. find our card
  12. Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. Hafez: an interactive poetry generation system. In Proceedings of ACL 2017, System Demonstrations, pp. 43–48, Vancouver, Canada, July 2017. Association for Computational Linguistics. URL https://www.aclweb. org/anthology/P17-4008. ↗
  13. Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor OK Li. Learning to translate in real-time with neural machine translation. arXiv preprint arXiv:1610.00388, 2016. find
  14. Jiatao Gu, Kyunghyun Cho, and Victor OK Li. Trainable greedy decoding for neural machine translation. arXiv preprint arXiv:1702.02429, 2017. find
  15. Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. CoRR, abs/1805.06087, 2018. URL http://arxiv. org/abs/1805.06087. find our card
  16. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. find
  17. Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. Controllable text generation. CoRR, abs/1703.00955, 2017. URL http://arxiv.org/abs/1703.00955. ↗ our card
  18. Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. 2016. find our card
  19. Jigsaw. Toxic comment classification challenge. https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/. Accessed: 2019-11-13. ↗
  20. Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. CTRL - A Conditional Transformer Language Model for Controllable Generation. arXiv preprint arXiv:1909, 2019. find our card
  21. Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. Controlling output length in neural encoder-decoders. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1328–1338, Austin, Texas, November 2016. Association for Computational… find
  22. Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc’Aurelio Ranzato, and Y-Lan Boureau. Multiple-attribute text rewriting. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=H1g2NhC5KQ. ↗
  23. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A Diversity-Promoting Objective Function for Neural Conversation Models. arXiv e-prints, art. arXiv:1510.03055, Oct 2015. find our card
  24. Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: A simple approach to sentiment and style transfer. CoRR, abs/1804.06437, 2018. URL http://arxiv.org/abs/1804.06437. ↗ our card
  25. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Languag… find
  26. Fuli Luo, Damai Dai, Pengcheng Yang, Tianyu Liu, Baobao Chang, Zhifang Sui, and Xu Sun. Learning to control the fine-grained sentiment for story ending generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6020–6026, 2019. find
  27. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150, Portland, Oregon, USA, J… find our card
  28. Christopher D Manning, Christopher D Manning, and Hinrich Schütze. Foundations of statistical natural language processing. MIT press, 1999. find
  29. Nicholas Metropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta H Teller, and Edward Teller. Equation of state calculations by fast computing machines. The journal of chemical physics, 21(6):1087–1092, 1953. find
  30. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. Facebook fair’s wmt19 news translation task submission. arXiv preprint arXiv:1907.06616, 2019. find our card
  31. Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015. find
  32. Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. find
  33. Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. Towards controllable story generation. In Proceedings of the First Workshop on Storytelling, pp. 43–49, 2018. find
  34. Martin Potthast, Tim Gollub, Kristof Komlossy, Sebastian Schuster, Matti Wiegmann, Erika Patricia Garces Fernandez, Matthias Hagen, and Benno Stein. Crowdsourcing a large corpus of clickbait on twitter. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 1498–1507,… find
  35. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018a. find
  36. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018b. find
  37. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019. find
  38. Gareth O Roberts and Jeffrey S Rosenthal. Optimal scaling of discrete approximations to langevin diffusions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 60(1): 255–268, 1998. find
  39. Gareth O Roberts, Richard L Tweedie, et al. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli, 2(4):341–363, 1996. find
  40. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. What makes a good conversation? How controllable attributes affect human judgments. arXiv e-prints, art. arXiv:1902.08654, Feb 2019. find our card
  41. Claude Elwood Shannon. A mathematical theory of communication. Bell system technical journal, 27(3):379–423, 1948. find
  42. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. Style transfer from non-parallel text by cross-alignment. CoRR, abs/1705.09655, 2017. URL http://arxiv.org/abs/1705.09655. ↗ our card
  43. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642, Se… find
  44. Felix Stahlberg, James Cross, and Veselin Stoyanov. Simple fusion: Return of the language model. arXiv preprint arXiv:1809.00125, 2018. find
  45. Nishant Subramani, Sam Bowman, and Kyunghyun Cho. Can unconditional language models recover arbitrary sentences? arXiv preprint arXiv:1907.04944, 2019. — [arXiv:1907.04944](https://arxiv.org/abs/1907.04944) find
  46. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6199, 2013. find our card
  47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 6000–6010, 2017. find
  48. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for nlp. arXiv preprint arXiv:1908.07125, 2019. find
  49. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992. find
  50. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Transformers: State-of-the-art natural language processing, 2019. find our card
  51. Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. Plan-and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7378–7385, 2019. find
  52. Kyra Yee, Nathan Ng, Yann N Dauphin, and Michael Auli. Simple and effective noisy channel modeling for neural machine translation. arXiv preprint arXiv:1908.05731, 2019. find
  53. Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Thirty-First AAAI Conference on Artificial Intelligence, 2017. find
  54. Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomas Kocisky. The neural noisy channel. arXiv preprint arXiv:1611.02554, 2016. — [arXiv:1611.02554](https://arxiv.org/abs/1611.02554) find
  55. Lei Yu, Laurent Sartran, Wojciech Stokowiec, Wang Ling, Lingpeng Kong, Phil Blunsom, and Chris Dyer. Putting machine translation in context with the noisy channel model. arXiv preprint arXiv:1910.00553, 2019. — [arXiv:1910.00553](https://arxiv.org/abs/1910.00553) find
  56. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. URL https://arxiv.org/abs/1909.08593. ↗
  57. S UPPLEMENTARY I NFORMATION FOR : P LUG AND P LAY L ANGUAGE M ODELS : A S IMPLE A PPROACH TO C ONTROLLED T EXT G ENERATION S6 E THICS OF C ONTROLLED L ANGUAGE M ODELS find