Elamur Research⌕
☰
home / papers / 2020 / Recipes for Safety in Open-domain Chatbots

Recipes for Safety in Open-domain Chatbots

Paper · 2020 · Jing Xu, Da Ju, Margaret Li et al.

Summary

Chatbots trained on large collections of human conversation (here, Reddit-based pre-training) absorb toxic language and biases along with useful conversational skills. The authors, from Facebook AI Research, compare safety strategies for open-domain generative dialogue, starting from BlenderBot's BST 2.7B model. They test four families of methods: unsafe utterance detection with two-stage filtering, safer generation through data filtering, control and decoding, avoidance of sensitive topics, and gender bias mitigation. They evaluate each method on both safety and engagingness.

They make two new contributions. The first is Bot-Adversarial Dialogue (BAD), in which crowdworkers deliberately try to provoke models into unsafe replies, and the resulting data is used to train more robust classifiers. The second is "baked-in" safety, in which a classifier rewrites training targets into safe responses, so the generator learns safe behaviour and needs no classifier at deployment.

Both techniques beat their generative or two-stage counterparts on safety while keeping engagingness close to the state of the art. Even so, the safest model still fails on some adversarial inputs. This makes the work a practical reference for building safer, deployable chatbots.

Key points

  • The BAD dataset contains about 5k adversarial human–bot conversations (around 70k utterances). It was released together with the models in ParlAI.
  • The safety classifiers are Transformers with 256M and 622M parameters. They were trained on:
    • WTC (about 150k examples)
    • 8k offensive dialogue examples
    • 16k Build-it Break-it Fix-it examples
    • semi-supervised labels on pushshift.io Reddit and BST data
  • When an utterance is flagged, two-stage systems either give a canned safe reply or switch to a non sequitur. The non sequitur topic is drawn from 1,087 safe Wizard of Wikipedia topics.
  • The comparison models are DialoGPT and GPT2 (Large), measured against BST 2.7B (2 encoder and 24 decoder layers).
  • Human raters judged the best model safe 96.6% of the time on the BAD test set. The authors note several limitations:
    • The annotation is English-only and done by US-based annotators.
    • Offensiveness is defined by annotator consensus.
    • Norms of what counts as offensive change over time.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksBias, fairness & safety · 517 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Chatbots · 54 works

Cite

Cite as Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston and Emily Dinan. 2020. Recipes for Safety in Open-domain Chatbots. arXiv:2010.07079.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2010.07079 · arXiv 2010.07079

@misc{xu2020recipes,
  title = {{Recipes for Safety in Open-domain Chatbots}},
  author = {Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston and Emily Dinan},
  year = {2020},
  doi = {10.48550/arxiv.2010.07079},
  eprint = {2010.07079},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2010.07079}
}

References

71 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977.
  2. Betty van Aken, Julian Risch, Ralf Krestel, and Alexander Löser. 2018. Challenges for toxic comment classification: An in-depth error analysis. arXiv preprint arXiv:1809.07572.
  3. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. arXiv preprint arXiv:2001.08435. — [arXiv:2001.08435](https://arxiv.org/abs/2001.08435) find
  4. Prashanth Bhat and Ofra Klein. 2020. Covert hate speech: white nationalists and dog whistle communication on twitter. In Twitter, the Public Sphere, and the Chaos of Online Deliberation, pages 151–172. Springer. find
  5. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of" bias" in nlp. arXiv preprint arXiv:2005.14050.
  6. Tommaso Caselli, Valerio Basile, Jelena Mitrović, Inga Kartoziya, and Michael Granitzer. 2020. I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6193–6202.
  7. Hyojin Chin, Lebogang Wame Molefi, and Mun Yong Yi. 2020. Empathy is all you need: How a conversational agent should respond to verbal abuse. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–13. find
  8. Hyojin Chin and Mun Yong Yi. 2019. Should an agent be ignoring it? a study of verbal abuse types and conversational agents’ response styles. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–6. find
  9. Amanda Cercas Curry and Verena Rieser. 2019. A crowd-based evaluation of abuse response strategies in conversational agents. arXiv preprint arXiv:1909.04387. find
  10. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: a simple approach to controlled text generation. arXiv preprint arXiv:1912.02164.
  11. Antonella De Angeli and Sheryl Brahnam. 2008. I hate you! disinhibition with virtual partners. Interacting with computers, 20(3):302–310. find
  12. Antonella De Angeli and Rollo Carpenter. 2005. Stupid computer! abuse and social identities. In Proc. find
  13. INTERACT 2005 workshop Abuse: The darker side of Human-Computer Interaction, pages 19–25. find
  14. Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2019a. Queens are powerful too: Mitigating gender bias in dialogue generation. arXiv preprint arXiv:1911.03842.
  15. Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. 2020. Multi-dimensional gender bias classification. arXiv preprint arXiv:2005.00614. find
  16. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019b. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Na…
  17. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019c. Wizard of Wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations.
  18. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73. find
  19. Paula Cristina Teixeira Fortuna. 2017. Automatic detection of hate speech in text: an overview of the topic and dataset annotation with hierarchical classes. find
  20. Antigoni Maria Founta, Despoina Chatzakou, Nicolas Kourtellis, Jeremy Blackburn, Athena Vakali, and Ilias Leontiadis. 2019. A unified deep learning architecture for abuse detection. In Proceedings of the 10th ACM Conference on Web Science, pages 105– 114. find
  21. Sam Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462.
  22. Oguzhan Gencoglu. 2020. Cyberbullying detection with fairness constraints. arXiv preprint arXiv:2005.06625. — [arXiv:2005.06625](https://arxiv.org/abs/2005.06625) find
  23. Alon Halevy, Cristian Canton Ferrer, Hao Ma, Umut Ozertem, Patrick Pantel, Marzieh Saeidi, Fabrizio Silvestri, and Ves Stoyanov. 2020. Preserving integrity in online social networks. arXiv preprint arXiv:2009.10311. — [arXiv:2009.10311](https://arxiv.org/abs/2009.10311) find
  24. Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 123–129. find
  25. Jennifer Hill, W Randolph Ford, and Ingrid G Farreras. 2015. Real conversations with artificial intelligence: A comparison between human–human online conversations and human–chatbot conversations. Computers in human behavior, 49:245–250. find
  26. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
  27. Chandra Khatri, Behnam Hedayatnia, Rahul Goel, Anushree Venkatesh, Raefer Gabriel, and Arindam Mandal. 2018a. Detecting offensive content in open-domain conversations using two stage semi-supervision. CoRR, abs/1811.12900. — [arXiv:1811.12900](https://arxiv.org/abs/1811.12900) find
  28. Chandra Khatri, Behnam Hedayatnia, Anu Venkatesh, Jeff Nunn, Yi Pan, Qing Liu, Han Song, Anna Gottardi, Sanjeev Kwatra, Sanju Pancholi, et al. 2018b. Advancing the state of the art in open domain dialog systems through the alexa prize. arXiv preprint arXiv:1812.10757. — [arXiv:1812.10757](https://arxiv.org/abs/1812.10757) find
  29. Ben Krause, Akhilesh Deepak Gotmare, Bryan Mc-Cann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2020. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367.
  30. Klaus Krippendorff. 2004. Reliability in content analysis: Some common misconceptions and recommendations. Human communication research, 30(3):411–433. find
  31. Ritesh Kumar, Atul Kr. Ojha, Bornini Lahiri, Marcos Zampieri, Shervin Malmasi, Vanessa Murdock, and Daniel Kadar, editors. 2020. Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying. European Language Resources Association (ELRA), Marseille, France. find
  32. Margaret Li, Jason Weston, and Stephen Roller. 2019. ACUTE-EVAL: Improved dialogue evaluation with optimized questions and multi-turn comparisons. In NeurIPS workshop on Conversational AI. find
  33. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural L…
  34. Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. 2019. Does gender matter? towards fairness in dialogue systems. arXiv preprint arXiv:1910.10486.
  35. Haochen Liu, Zhiwei Wang, Tyler Derr, and Jiliang Tang. 2020. Chat as expected: Learning to manipulate black-box neural dialogue models. arXiv preprint arXiv:2005.13170. — [arXiv:2005.13170](https://arxiv.org/abs/2005.13170) find
  36. Catherine L Lortie and Matthieu J Guitton. 2011. Judgment of the humanness of an interlocutor is in the eye of the beholder. PLoS One, 6(9):e25085. find
  37. Rijul Magu, Kshitij Joshi, and Jiebo Luo. 2017. Detecting the hate code on social media. arXiv preprint arXiv:1703.05443. — [arXiv:1703.05443](https://arxiv.org/abs/1703.05443) find
  38. Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017a. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79–84. ACL.
  39. K.W Miller, Marty J Wolf, and F.S. Grodzinsky. 2017b. Why we should have seen that coming. ORBIT Journal, 1(2). find
  40. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial NLI: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599.
  41. Tong Niu and Mohit Bansal. 2018. Polite dialogue generation without parallel data. Transactions of the Association for Computational Linguistics, 6:373–389. find
  42. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038.
  43. Ashwin Paranjape, Abigail See, Kathleen Kenealy, Haojun Li, Amelia Hardy, Peng Qi, Kaushik Ram Sadagopan, Nguyet Minh Phu, Dilara Soylu, and Christopher D Manning. 2020. Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations. arXiv preprint arXiv:2008.12348. find
  44. Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. arXiv preprint arXiv:1808.07231.
  45. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304.
  46. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  47. Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, Eric King, Kate Bland, Amanda Wartick, Yi Pan, Han Song, Sk Jayadevan, Gene Hwang, and Art Pet-tigrue. 2017. Conversational AI: The science behind the Adlexa Pri… find
  48. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Com… find
  49. Julian Risch, Robin Ruff, and Ralf Krestel. 2020. Offensive language detection explained. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, pages 137–143, Marseille, France. European Language Resources Association (ELRA). find
  50. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al. 2020. Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004.13637.
  51. Cicero Nogueira dos Santos, Igor Melnyk, and Inkit Padhi. 2018. Fighting offensive language on social media with unsupervised text style transfer. arXiv preprint arXiv:1805.07685.
  52. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678.
  53. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International workshop on natural language processing for social media, pages 1–10.
  54. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 1702–1723. ACL.
  55. Kurt Shuster, Samuel Humeau, Antoine Bordes, and Jason Weston. 2018. Engaging image chat: Modeling personality in grounded dialogue. arXiv preprint arXiv:1811.00945. — [arXiv:1811.00945](https://arxiv.org/abs/1811.00945) find
  56. Eric Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020a. Can you put it all together: Evaluating conversational agents’ ability to blend skills. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. ACL.
  57. Eric Michael Smith, Diana Gonzalez-Rico, Emily Dinan, and Y-Lan Boureau. 2019. Zero-shot fine-grained style transfer: Leveraging distributed continuous style representations to transfer to unseen styles. arXiv preprint arXiv:1911.03914. find
  58. Eric Michael Smith, Diana Gonzalez-Rico, Emily Dinan, and Y-Lan Boureau. 2020b. Controlling style in generated dialogue. find
  59. Steve Durairaj Swamy, Anupam Jamatia, and Björn Gambäck. 2019. Studying generalisability across abusive language detection datasets. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 940–950. find
  60. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
  61. Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019. Challenges and frontiers in abusive content detection. In Proceedings of the Third Workshop on Abusive Language Online, page 80. Association for Computational Linguistics.
  62. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv:1908.07125. find
  63. Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. arXiv preprint arXiv:1705.09899.
  64. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In International Conference on Learning Representations.
  65. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, pages 1391–1399. ACM. find
  66. Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020. Demoting racial bias in hate speech detection. arXiv preprint arXiv:2005.12246. find
  67. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). arXiv preprint arXiv:1903.08983.
  68. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020). arXiv preprint arXiv:2006.07235. find
  69. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2204–2213. ACL.
  70. Yangjun Zhang, Pengjie Ren, and Maarten de Rijke. 2020. Detecting and classifying malevolent dialogue responses: Taxonomy, data and methodology. arXiv preprint arXiv:2008.09706. — [arXiv:2008.09706](https://arxiv.org/abs/2008.09706) find
  71. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. DialoGPT: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2020-recipes-safety-open-domain-chatbots/