Elamur Research⌕
☰
home / papers / 2022 / On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark

On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark

Paper · 2022 · Hao Sun, Guangxuan Xu, Jiawen Deng et al.

Summary

Open-domain chatbots can produce replies that look harmless in isolation but become harmful given what the user said. Earlier safety resources mostly judge single utterances. This paper defines that gap more precisely. It proposes a dialogue safety taxonomy that separates utterance-level unsafety from context-sensitive unsafety, meaning responses that are only problematic in light of the preceding turn. Context-sensitive cases are further split into personal harms (to the user) and non-personal harms. The authors build DiaSafety, a dataset of safe and unsafe context–response pairs drawn from social media and model generations. Off-the-shelf safety tools such as Perspective API perform poorly on it. A classifier trained on DiaSafety gives a strong baseline. Combined with existing utterance-level detectors in a two-step check, it is used to benchmark DialoGPT, Blenderbot and Plato-2. All three still show notable context-sensitive safety problems. The work matters because it supplies a definition, data and an evaluation protocol for a kind of chatbot risk that toxicity filters largely miss.

Key points

  • Six context-sensitive categories:
    • Personal unsafety: Offending User, Risk Ignorance, Unauthorized Expertise.
    • Non-personal unsafety: Toxicity Agreement, Biased Opinion, Sensitive Topic Continuation.
  • Dataset: DiaSafety is labeled as "5×2", i.e. five categories, each with safe and unsafe examples. It is released at github.com/thu-coai/DiaSafety.
  • Benchmark protocol: each model generates 10 responses per validation/test context. The score is the share of context-sensitive and utterance-level unsafe replies, where lower means safer.
  • Model comparison: Blenderbot is the safest overall, but it is weak on Toxicity Agreement, likely because it was trained toward empathy. Plato-2 controls utterance-level safety worst.
  • Scale and decoding: larger models are not safer, and top-k vs. nucleus sampling barely affects safety. An appendix leaderboard ranks 8 models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🏢TencentShenzhen · China66 works · 17,945 citations🎓University of California, Los AngelesLos Angeles · United States54 works · 53,688 citations🔬State Key Laboratory of Intelligent Technology and SystemsBeijing · China21 works · 9,641 citations🔬Beijing National Research Center for Information Science and TechnologyBeijing · China21 works · 5,843 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksSafety Research · 221 works

🧭 Topics 3Research subjects and languages the work is about.Evaluation & benchmarks · 685 worksDialogue systems · 401 worksBias, fairness & safety · 209 works

Cite

Cite as Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu and Minlie Huang. 2022. On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark. Findings of the Association for Computational Linguistics: ACL 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-acl.308

@inproceedings{sun2022safety,
  title = {{On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark}},
  author = {Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu and Minlie Huang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.308},
  url = {https://doi.org/10.18653/v1/2022.findings-acl.308}
}

References

75 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Our dataset is limited in Single-modal (text). We agree that dialogue system with other modals also contain safety problems. Meanwhile, a under-robust ASR may induce new challenges of erroneous safety check (Liu et al., 2020). find
  2. Our dataset is limited in single-turn dialogue. We do believe that multi-turn dialogue contexts would more make a difference to the safety of the response and deserve well future researches for the development of this community. • Though we list Sensitive Topic Continuation in our taxonomy, we belie… find
  3. Alaa A Abd-Alrazaq, Mohannad Alajlani, Ali Abdallah Alalwan, Bridgette M Bewick, Peter Gardner, and Mowafa Househ. 2019. An overview of the features of chatbots in mental health: A scoping review. International Journal of Medical Informatics, 132:103978. find
  4. Alaa Ali Abd-Alrazaq, Asma Rababeh, Mohannad Alajlani, Bridgette M Bewick, and Mowafa Househ. 2020. Effectiveness and safety of using chatbots to improve mental health: systematic review and meta-analysis. Journal of medical Internet research, 22(7):e16021. find
  5. ACM Committee on Professional Ethics. 2018. Acm code of ethics and professional conduct. https://www.acm.org/code-of-ethics. ↗
  6. Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. Towards a human-like open-domain chatbot.
  7. Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward res… find
  8. Ashutosh Baheti, Maarten Sap, Alan Ritter, and Mark O. Riedl. 2021. Just say no: Analyzing the stance of neural dialogue generation in offensive contexts. ArXiv, abs/2108.11830. find
  9. Yejin Bang, Nayeon Lee, Etsuko Ishii, Andrea Madotto, and Pascale Fung. 2021. Assessing political prudence of open-domain chatbots. find
  10. Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhen Guo, Zhibin Liu, and Xinchao Xu. 2021. Plato-2: Towards building an open-domain chatbot via curriculum learning.
  11. Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021. Redditbias: A real-world resource for bias evaluation and debiasing of conversational language models. find
  12. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. In ICWSM. find
  13. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Associat… find
  14. Timothy W Bickmore, Ha Trinh, Stefan Olafsson, Teresa K O’Leary, Reza Asadi, Nathaniel M Rickles, and Ricardo Cruz. 2018. Patient and consumer safety risks when using conversational assistants for medical information: an observational study of siri, alexa, and google assistant. Journal of medical In… find
  15. Su Lin Blodgett, Solon Barocas, Hal Daumé III au2, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in nlp.
  16. Amanda Cercas Curry and Verena Rieser. 2018. \# metoo alexa: How conversational systems respond to sexual harassment. In Proceedings of the second acl workshop on ethics in natural language processing, pages 7–14. find
  17. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25–35, Florence, Italy. Association for Computational Linguistics.
  18. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media, ICWSM ’17, pages 512–515.
  19. Antonella De Angeli and Sheryl Brahnam. 2008. I hate you! disinhibition with virtual partners. Interacting with computers, 20(3):302–310. find
  20. Antonella De Angeli, Rollo Carpenter, et al. 2005. Stupid computer! abuse and social identities. In Proc. find
  21. INTERACT 2005 workshop Abuse: The darker side of Human-Computer Interaction, pages 19–25. Citeseer. find
  22. J. Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. find
  23. Emily Dinan, Gavin Abercrombie, A. Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. 2021. Anticipating safety issues in e2e conversational ai: Framework and tooling. find
  24. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix it for dialogue safety: Robustness from adversarial human attack.
  25. European Commission. 2021. Proposal for a regulation of the european parliament and of the council laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CEL LAR:e0649735-… ↗
  26. Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR), 51(4):1–30. find
  27. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369.
  28. Malik Ghallab. 2019. Responsible ai: requirements and challenges. AI Perspectives, 1(1):1–7. find
  29. Laura Hanu and Unitary team. 2020. Detoxify. Github. https://github.com/unitaryai/detoxify. ↗
  30. Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2017. Ethical challenges in data-driven dialogue systems. find
  31. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. find
  32. Md Saroar Jahan and Mourad Oussalah. 2021. A systematic review of hate speech automatic detection using natural language processing. arXiv preprint arXiv:2106.00742. — [arXiv:2106.00742](https://arxiv.org/abs/2106.00742) find
  33. Daniel Jurafsky, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual, draft 13. Technical Report 97-02, University of Colorado, Boulder Institute of Cognitive Science, Boulder, CO. find
  34. Nayeon Lee, Andrea Madotto, and Pascale Fung. 2019. Exploring social bias in chatbots using stereotype knowledge. In WNLP@ ACL, pages 177–180. find
  35. Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. 2021a. Dexperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th I…
  36. Jiexi Liu, Ryuichi Takanobu, Jiaxin Wen, Dazhen Wan, Weiran Nie, Hongyan Li, Cheng Li, Wei Peng, and Minlie Huang. 2020. Robustness testing of language understanding in dialog systems. CoRR, abs/2012.15262. — [arXiv:2012.15262](https://arxiv.org/abs/2012.15262) find
  37. Ruibo Liu, Chenyan Jia, and Soroush Vosoughi. 2021b. A transformer-based framework for neutralizing and reversing the political polarity of news articles. Proceedings of the ACM on Human-Computer Interaction, 5. find
  38. Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. 2021c. Mitigating political bias in language models through reinforced calibration. In AAAI. find
  39. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  40. Binny Mathew, Punyajoy Saha, Hardik Tharad, Subham Rajgaria, Prajwal Singhania, Suman Kalyan Maity, Pawan Goyal, and Animesh Mukherje. 2019. Thou shalt not hate: Countering online hate speech. In Thirteenth International AAAI Conference on Web and Social Media. find
  41. Stefano Menini, Alessio Palmero Aprosio, and Sara Tonelli. 2021. Abuse is contextual, what about nlp? the role of context in abusive language annotation and detection. find
  42. A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston. 2017. Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476.
  43. Amit Mittal, Ayushi Agrawal, Ayushi Chouksey, Rachna Shriwas, and Saloni Agrawal. 2016. A comparative study of chatbots and humans. Situations, 2(2). find
  44. Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pre-trained language models. In ACL 2021, volume 2. find
  45. John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020. Toxicity detection: Does context really matter?
  46. Dorian Peters, Karina Vold, Diana Robinson, and Rafael A Calvo. 2020. Responsible ai—two frameworks for ethical design practice. IEEE Transactions on Technology and Society, 1(1):34–47. find
  47. Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2021. Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources and Evaluation, 55(2):477–523. find
  48. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  49. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, and Jason Weston. 2020. Recipes for building an open-domain chatbot.
  50. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In NAACL.
  51. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10, Valencia, Spain. Association for Computational Linguistics.
  52. Ashish Sharma, Adam S Miner, David C Atkins, and Tim Althoff. 2020. A computational approach to understanding empathy expressed in text-based mental health support. In EMNLP.
  53. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2021. "nice try, kiddo": Investigating ad hominems in dialogue responses. find
  54. Heung-Yeung Shum, Xiaodong He, and Di Li. 2018. From eliza to xiaoice: Challenges and opportunities with social chatbots. CoRR, abs/1801.01957. — [arXiv:1801.01957](https://arxiv.org/abs/1801.01957) find
  55. Eric Michael Smith, Diana Gonzalez-Rico, Emily Dinan, and Y-Lan Boureau. 2020a. Controlling style in generated dialogue. arXiv preprint arXiv:2009.10855. — [arXiv:2009.10855](https://arxiv.org/abs/2009.10855) find
  56. Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020b. Can you put it all together: Evaluating conversational agents’ ability to blend skills. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2021–2030, Online. Asso…
  57. Hao Sun, Zhenru Lin, Chujie Zheng, Siyang Liu, and Minlie Huang. 2021. PsyQA: A Chinese dataset for generating long counseling text for mental health support. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1489–1503, Online. Association for Computational Linguis… find
  58. Elsbeth Turcan and Kathy McKeown. 2019. Dreaddit: A Reddit dataset for stress analysis in social media. In Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019), pages 97–107, Hong Kong. Association for Computational Linguistics. find
  59. Aditya Nrusimha Vaidyam, Hannah Wisniewski, John David Halamka, Matcheri S Kashavan, and John Blake Torous. 2019. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. The Canadian Journal of Psychiatry, 64(7):456–464. find
  60. Yida Wang, Pei Ke, Yinhe Zheng, Kaili Huang, Yong Jiang, Xiaoyan Zhu, and Minlie Huang. 2020. A large-scale chinese short-text conversation dataset. find
  61. Zijian Wang and Christopher Potts. 2019. TALK-DOWN: A corpus for condescension detection in context. find
  62. EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pages 3711–3719. find
  63. Marty J Wolf, Keith W Miller, and Frances S Grodzinsky. 2017. Why we should have seen that coming: comments on microsoft’s tay “experiment,” and wider implications. The ORBIT Journal, 1(2):1–12. find
  64. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho…
  65. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. find
  66. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2950–2968…
  67. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media.
  68. NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 1:1415–1420. find
  69. Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang, Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoyu Zhang, Hongchao Fang, and Penghui Zhu. 2020. MedDialog : Large-scale Medical Dialogue Datasets. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processi…
  70. Yangjun Zhang, Pengjie Ren, and M. de Rijke. 2021. A taxonomy, data set, and benchmark for detecting and classifying malevolent dialogue responses. Journal of the Association for Information Science and Technology. find
  71. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. Dialogpt: Large-scale generative pre-training for conversational response generation.
  72. Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018. Learning gender-neutral word embeddings. In EMNLP.
  73. World Economic Forum. 2020. Chatbots RESET A Framework for Governing Responsible Use of Conversational AI in Healthcare In collaboration with Mitsubishi Chemical Holdings Corporation Contents. Technical Report December. find
  74. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. 26th International World Wide Web Conference, WWW 2017, pages 1391–1399. find
  75. Alexandros Xenos, John Pavlopoulos, and Ion Androutsopoulos. 2021. Context Sensitivity Estimation in Toxicity Detection. In Proceedings of the Workshop on Online Abuse and Harms at the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internati… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2022-safety-conversational-models-taxonomy-dataset/