Elamur Research⌕
☰
home / papers / 2023 / Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning

Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning

Paper · 2023 · Chujie Zheng, Pei Ke, Zheng Zhang et al.

Summary

Large language models often produce outputs people do not want, such as toxic text or degenerate repetition. This paper presents CLICK, a training-based method that steers generation away from such attributes without changing the model architecture, so trained models can be used directly.

For each prompt, the base model samples several continuations, and a binary label function marks each one as positive or negative. CLICK then fine-tunes the model on two objectives: the usual language-modeling loss on positive data, and a max-margin loss that pushes the sequence likelihood of each negative continuation below that of a paired positive one.

Pairing follows a likelihood-ranking rule. Each negative is matched with the most likely positive whose likelihood is still below the negative's. This keeps the two samples comparable in fluency, so learning targets the controlled attribute.

The method was tested on language detoxification, sentiment steering, and repetition reduction. Automatic and human evaluations show it beats strong controllable-generation baselines, and ablations confirm the value of the pairing strategy. It is a simple, plug-and-play way to align generation with an attribute classifier.

Key points

  • Objective: total loss is L = L_LM + α·L_CL, where L_CL is a hinge loss on log-likelihoods with margin γ. Labels are kept binary to reduce classifier bias inherited from continuous scores.
  • Baselines: compared against Unlikelihood Training, Cringe loss, Quark, and Director. Decoding-time methods such as GeDi, DExperts, PPLM, and FUDGE are discussed as the alternative family.
  • Iterative training on the BAD test set: one more training round lowered toxicity probability from 0.084 to 0.056. The cost was fluency (output PPL rose from 6.48 to 7.71) and diversity (Dist-2/3 fell from 0.49/0.56 to 0.47/0.54).
  • Base model and data: BlenderBot produced only about 5 toxic continuations out of 20 on the BAD training set, which illustrates how scarce negatives tend to be.
  • Limitations and resources: CLICK relies on neural classifiers and may inherit their biases, for example overestimating toxicity when minority identities are mentioned. Code is available at github.com/chujiezheng/Click.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🔬State Key Laboratory of Intelligent Technology and SystemsBeijing · China23 works · 9,761 citations🔬Beijing National Research Center for Information Science and TechnologyBeijing · China28 works · 5,973 citations🔬Viện Trí tuệ nhân tạoHanoi · Vietnam3 works · 564 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Text generation · 202 works

Cite

Cite as Chujie Zheng, Pei Ke, Zheng Zhang and Minlie Huang. 2023. Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning. Findings of the Association for Computational Linguistics: ACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-acl.65

@inproceedings{zheng2023click,
  title = {{Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning}},
  author = {Chujie Zheng, Pei Ke, Zheng Zhang and Minlie Huang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.65},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.65}
}

References

41 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Leonard Adolphs, Tianyu Gao, Jing Xu, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. 2022. The cringe loss: Learning what language not to model. arXiv preprint arXiv:2211.05826. find
  2. Chenxin An, Jiangtao Feng, Kai Lv, Lingpeng Kong, Xipeng Qiu, and Xuanjing Huang. 2022. Cont: Contrastive neural text generation. In NeurIPS. find
  3. Kushal Arora, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. 2022. Director: Generator-classifiers for supervised language modeling. In AACL. find
  4. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In FAccT. find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In NeurIPS.
  6. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In ICML.
  7. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation. In ICLR.
  8. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  9. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In EMNLP.
  10. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of EMNLP.
  11. Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus. ↗
  12. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In CVPR.
  13. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In ICLR.
  14. Shaojie Jiang, Ruqing Zhang, Svitlana Vakulenko, and Maarten de Rijke. 2022. A simple contrastive learning objective for alleviating neural text degeneration. arXiv preprint arXiv:2205.02517. — [arXiv:2205.02517](https://arxiv.org/abs/2205.02517) find
  15. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
  16. Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021. Gedi: Generative discriminator guided sequence generation. In Findings of EMNLP.
  17. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016. A diversity-promoting objective function for neural conversation models. In NAACL.
  18. Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. 2021. Dexperts: Decoding-time controlled text generation with experts and anti-experts. In ACL.
  19. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  20. Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. Brio: Bringing order to abstractive summarization. In ACL.
  21. Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022. Quark: Controllable text generation with reinforced unlearning. In NeurIPS.
  22. Kris McGuffie and Alex Newhouse. 2020. The radicalization risks of gpt-3 and advanced neural language models. arXiv preprint arXiv:2009.06807. — [arXiv:2009.06807](https://arxiv.org/abs/2009.06807) find
  23. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In ICLR.
  24. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In NeurIPS.
  25. Jayshree Pandya. 2019. The dual-use dilemma of artificial intelligence. Cognitive World. find
  26. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. arXiv preprint arXiv:2202.03286.
  27. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap between neural text and human text using divergence frontiers. In NeurIPS.
  28. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog. find
  29. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric Michael Smith, Y.-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In EACL.
  30. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. find
  31. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In ICML. find
  32. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP.
  33. Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. In NeurIPS. find
  34. Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. 2022. On the safety of conversational models: Taxonomy, dataset, and benchmark. In Findings of ACL.
  35. Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, and Bryan Catanzaro. 2022. Exploring the limits of domain-adaptive training for detoxifying large-scale language models. In NeurIPS. find
  36. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019. Neural text generation with unlikelihood training. In ICLR.
  37. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2020. Transformers: State-of-the-art natural language processing. In EMNLP.
  38. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021. Bot-adversarial dialogue for safe conversational agents. In NAACL.
  39. Kevin Yang and Dan Klein. 2021. Fudge: Controlled text generation with future discriminators. In NAACL.
  40. Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. 2022. Calibrating sequence likelihood improves conditional language generation. arXiv preprint arXiv:2210.00045. — [arXiv:2210.00045](https://arxiv.org/abs/2210.00045) find
  41. Chujie Zheng, Sahand Sabour, Jiaxin Wen, Zheng Zhang, and Minlie Huang. 2023. Augesc: Dialogue augmentation with large language models for emotional support conversation. In Findings of ACL. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2023-click-controllable-text-generation-sequence/