Elamur Research⌕
☰
home / papers / 2023 / ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation

ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation

Summary

Most toxicity detectors are trained on social media text. Users of LLM chatbots behave differently: they phrase harmful intent as questions or instructions, and they sometimes use "jailbreaking" prompts that look benign but aim to make the model bypass its policies. To study this gap, the authors built ToxicChat, the first toxicity benchmark drawn from real user-AI conversations. It contains 10,166 first-turn user queries from the public Vicuna demo, labeled for toxicity and jailbreaking by four researchers.

To lower annotation cost, they used a human-AI collaborative scheme. Perspective API confidence scores were used to set aside inputs that were very likely non-toxic, which removed about 60% of the manual work. Disputed labels were resolved by discussion, and GPT-4 was used to re-check examples marked non-toxic.

Existing moderation APIs and detectors performed poorly on the benchmark. A roberta-base model fine-tuned on ToxicChat clearly beat models trained on much larger datasets from other domains. The work shows a real domain shift and provides a resource for safer chatbots.

Key points

  • Dataset contents: ToxicChat holds 10,166 examples collected from the Vicuna demo between March 30 and April 12, 2023. About 7.10% are toxic and 1.75% are jailbreaking. It is released at lmsys/toxic-chat on Hugging Face, with access gated by a request form.
  • Annotation quality and workload: On a 720-example trial set, inter-annotator agreement was 96.11%. Applying a Perspective API filter removed 4,668 low-score items from manual annotation, and on the trial set this filter missed only 1 of 48 toxic cases.
  • Off-the-shelf tools: F1 scores on the benchmark were 20.6 for OpenAI moderation, 5.3 for Perspective API, 11.6 for HateBERT, and 34.6 for ToxDectRoberta.
  • Training data comparison: roberta-base trained on ToxicChat reached an F1 of 72.1 and a jailbreaking recall of 83.5. The same model trained on a combination of other datasets (HSTA, Jigsaw, ToxiGen, RealToxicPrompts, ConvAbuse, and others) reached an F1 of only 42.7.
  • Using chatbot responses: Adding Vicuna's responses as features did not help significantly (F1 72.6 vs. 72.1). Using the responses alone made results worse.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

84 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 77 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.311

@inproceedings{lin2023toxicchat,
  title = {{ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation}},
  author = {Lin, Zi and Wang, Zihan and Tong, Yongqi and Wang, Yangkun and Guo, Yuxin and Wang, Yujia and Shang, Jingbo},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.311},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.311}
}

References

26 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ari Ball-Burack, Michelle Seng Ah Lee, Jennifer Cobbe, and Jatinder Singh. 2021. Differential tweetment: Mitigating racial dialect bias in harmful tweet detection. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 116–128. find
  2. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. find our card
  3. Tommaso Caselli, Valerio Basile, Jelena Mitrovic, and Michael Granitzer. 2020. Hatebert: Retraining BERT for abusive language detection in english. CoRR, abs/2010.12472. find our card
  4. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651. find our card
  5. Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. 2021. HateBERT: Retraining BERT for abusive language detection in English. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 17–25, Online. Association for Computational Linguistics. find our card
  6. Ian D Kivlichan, Zi Lin, Jeremiah Liu, and Lucy Vasserman. 2021. Measuring and improving model-moderator collaboration using uncertainty estimation. arXiv preprint arXiv:2107.04212. find
  7. Hyojin Chin and Mun Yong Yi. 2019. Should an agent be ignoring it? a study of verbal abuse types and conversational agents’ response styles. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–6. find
  8. Amanda Cercas Curry, Gavin Abercrombie, and Verena Rieser. 2021. Convabuse: Data, analysis, and benchmarks for nuanced detection in conversational AI. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic… find
  9. Amanda Cercas Curry and Verena Rieser. 2018. \# metoo alexa: how conversational systems respond to sexual harassment. In Proceedings of the second acl workshop on ethics in natural language processing, pages 7–14. find
  10. Amanda Cercas Curry and Verena Rieser. 2019. A crowd-based evaluation of abuse response strategies in conversational agents. arXiv preprint arXiv:1909.04387. find
  11. Paula Fortuna, Juan Soler, and Leo Wanner. 2020. Toxic, hateful, offensive or abusive? what are we really classifying? an empirical analysis of hate speech datasets. In Proceedings of the 12th language resources and evaluation conference, pages 6786– 6794. find our card
  12. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. ArXiv, abs/2009.11462. find our card
  13. Xiaochuang Han and Yulia Tsvetkov. 2020. Fortifying toxic speech detectors against veiled toxicity. arXiv preprint arXiv:2010.03154. find
  14. Camille Harris, Matan Halevy, Ayanna Howard, Amy Bruckman, and Diyi Yang. 2022. Exploring the role of grammar and word choice in bias toward african american english (aae) in hate speech classification. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 789–798. find
  15. Anna Koufakou, Endang Wahyu Pamungkas, Valerio Basile, and Viviana Patti. 2020. HurtBERT: Incorporating lexical features with BERT for the detection of abusive language. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 34–43, Online. Association for Computational Linguistics. find
  16. Stefano Leone. 2020. Rotten tomatoes movies and critic reviews dataset. find
  17. Xiaojuan Ma, Emily Yang, and Pascale Fung. 2019. Exploring perceived emotional intelligence of personality-driven virtual agents in handling user challenges. In The World Wide Web Conference, pages 1222–1233. find
  18. Debora Nozza, Claudia Volpetti, and Elisabetta Fersini. 2019. Unintended bias in misogyny detection. In IEEE/WIC/ACM International Conference on Web Intelligence, WI ’19, page 149–155, New York, NY, USA. Association for Computing Machinery. find
  19. John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020. Toxicity detection: Does context really matter? arXiv preprint arXiv:2006.00998. find our card
  20. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. find
  21. Ishan Sanjeev Upadhyay, KV Aditya Srivatsa, and Radhika Mamidi. 2022. Towards toxic positivity detection. In Proceedings of the tenth international workshop on natural language processing for social media, pages 75–82. find
  22. Kunze Wang, Dong Lu, Caren Han, Siqu Long, and Josiah Poon. 2020. Detect all abuse! toward universal abusive language detection models. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6366–6376, Barcelona, Spain (Online). International Committee on Computation… find
  23. Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL Student Research Workshop, pages 88–93, San Diego, California. Association for Computational Linguistics. find our card
  24. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75–86, Minneapolis,… find our card
  25. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. find
  26. Xuhui Zhou. 2021. Challenges in automated debiasing for toxic language detection. University of Washington. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lin-2023-toxicchat-unveiling-hidden-challenges-toxicity/