Elamur Research
☰
home / papers / 2021 / DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts

DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.acl-long.522.pdf

Summary

Large pretrained language models can produce toxic or otherwise unwanted text. Retraining or finetuning them to fix this is becoming too expensive as models grow. The authors propose DExperts, which steers generation only at decoding time. At each step, the logits of a base LM are shifted by the difference between two smaller models: an "expert" tuned on text with the desired attribute and an "anti-expert" tuned on text with the unwanted one. The strength of this shift is set by a weight α. As a result, a token stays likely only when the expert favours it and the anti-expert does not. Candidates are restricted to the base model's nucleus, so no low-probability tail tokens are reintroduced.

The method is tested on detoxification and sentiment control, with GPT-2 Large as the base model. Toxicity (anti-)experts are trained on Jigsaw comments. DExperts beats DAPT, PPLM and GeDi on both automatic and human evaluations while keeping fluency and diversity. Because it needs only the base model's outputs, it also works with GPT-3 through an API. This makes cheap, customizable steering possible for very large models.

Key points

  • The (anti-)experts are GPT-2 Small, Medium and Large models finetuned on the Jigsaw Unintended Bias in Toxicity Classification data. This data contains about 160K toxic and about 1.4M non-toxic comments. Comments count as toxic if at least half of annotators flagged them, and as non-toxic if no annotator did.
  • Evaluation uses 10K non-toxic prompts from RealToxicityPrompts, with α = 2.0 and nucleus sampling (p = 0.9) for up to 20 tokens. Toxicity is scored with Perspective API.
  • Compared with plain GPT-2, DExperts (small) lowers average maximum toxicity from 0.527 to 0.302 and toxicity probability from 0.520 to 0.118. GeDi reaches only 0.363 and 0.217, with much worse perplexity (60.03 vs. 38.20).
  • An anti-expert-only variant, which reuses the base model as the expert, still outperforms all baselines (0.352 / 0.191). An effective anti-expert can be built from roughly 650 toxic comments.
  • Code is released at github.com/alisawuffles/DExperts. A preliminary stylistic-rewriting experiment is also reported.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

251 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 235 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.522

@inproceedings{liu2021dexperts,
  title = {{DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts}},
  author = {Liu, Alisa and Sap, Maarten and Lu, Ximing and Swayamdipta, Swabha and Bhagavatula, Chandra and Smith, Noah A. and Choi, Yejin},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.522},
  url = {https://doi.org/10.18653/v1/2021.acl-long.522}
}

References

43 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Confer-7 https://github.com/conversationai/ perspectiveapi Steven Bird and Edward Loper. 2004. NLTK: The natural… ↗
  2. Greg Brockman, Mira Murati, and Peter Welinder. 2020. OpenAI API. Blog post. find
  3. T. Brown, B. Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, G. Krüger, T. Henighan, R. Child, Aditya Ramesh, D. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen,… find
  4. Adam M Croom. 2013. How to do things with slurs: Studies in the way of derogatory words. In Language & communication. find
  5. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language models: A simple approach to controlled text generation. In Proceedings of the 2020 International Conference on Learning Representations (ICLR). find our card
  6. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online. find our card
  7. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (AIES). find
  8. Li Dong, Shaohan Huang, Furu Wei, Mirella Lapata, Ming Zhou, and Ke Xu. 2017. Learning to generate product reviews from attributes. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL). find
  9. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL). find
  10. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv. find
  11. Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation. find our card
  12. Susan T Fiske. 1993. Controlling other people: the impact of power on stereotyping. American Psychologist. find
  13. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics (EMNLP Findings). find our card
  14. Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Louis-Philippe Morency, and Stefan Scherer. 2017. Affect-LM: A neural language model for customizable affective text generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL). find
  15. Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. find
  16. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). find
  17. Geoffrey E. Hinton. 2002. Training products of experts by minimizing contrastive divergence. In Neural Computation. find
  18. Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  19. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In Proceedings of the Eighth International Conference on Learning Representations (ICLR). find
  20. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… find our card
  21. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find our card
  22. Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find our card
  23. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  24. Andrea Madotto, Etsuko Ishii, Zhaojiang Lin, Sumanth Dathathri, and Pascale Fung. 2020. Plug-and-play conversational models. In Findings of the Association for Computational Linguistics (EMNLP Findings). find
  25. Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). find
  26. Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in NLP models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  27. Kris McGuffie and Alex Newhouse. 2020. The radicalization risks of gpt-3 and advanced neural language models. arXiv. find
  28. Abigail Z. Jacobs and Hannah Wallach. 2021. Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT). find
  29. Shrimai Prabhumoye, Alan W Black, and Ruslan Salakhutdinov. 2020. Exploring controllable text generation techniques. In Proceedings of the 28th International Conference on Computational Linguistics (COLING). find
  30. Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. arXiv. find our card
  31. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Preprint. find
  32. Ben Krause, Akhilesh Deepak Gotmare, Bryan Mc-Cann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2020. GeDi: Generative discriminator guided sequence generation. arXiv. find our card
  33. Robin Lakoff. 1973. Language and woman’s place. Language in Society. find
  34. Jayshree Pandya. 2019. The dual-use dilemma of artificial intelligence. Forbes Magazine. find
  35. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Preprint. find
  36. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Pro… find our card
  37. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  38. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find our card
  39. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find our card
  40. Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2020. Towards Controllable Biases in Language Generation. In Findings of the Association for Computational Linguistics (EMNLP Findings). find
  41. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find our card
  42. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  43. Kevin Yang and Dan Klein. 2021. FUDGE: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find our card