Elamur Research⌕
☰
home / papers / 2023 / Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity

Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity

Summary

Large language models are now used in real products, but there are few systematic, practice-oriented studies of their ethical hazards, particularly for recent conversational systems. This paper red-teams OpenAI's ChatGPT (the December 15, 2022 release) to find where it behaves harmfully. The authors first collected and manually examined 305,701 tweets about ChatGPT from December 2022. Using the LLM risk taxonomy of Weidinger et al., they grouped public concerns into four areas: bias, reliability, robustness and toxicity. Following HELM principles, they then tested ChatGPT zero-shot on established datasets for each area. Because those benchmarks miss many problems, they added hand-built case studies. On standard benchmarks, ChatGPT scores slightly above existing state-of-the-art models. The case studies, however, reveal concrete risks: biased generated code, susceptibility to prompt injection (jailbreaking) that produces unethical output, and misinformation spread through hallucination. The study shows that current benchmarks understate the harms of deployed LLMs, and it outlines the evaluations and design practices that responsible models will need.

Key points

  • The taxonomy of concerns comes from 305,701 ChatGPT-related tweets from December 2022. It yields four dimensions: Bias, Reliability, Robustness and Toxicity.
  • Benchmark selection follows HELM. Because existing datasets cover the risks only partially, the authors add qualitative case studies.
  • The main vulnerabilities found are prompt-injection jailbreaks, programming-related bias and hallucinated misinformation. Overall benchmark performance is marginally better than prior SOTA models.
  • The discussion covers emergent risks of scale and possible exhaustion of high-quality language data (a cited estimate puts this before 2026). It also addresses compute and carbon costs.
  • The authors list several limitations:
    • ChatGPT's hyperparameters are undisclosed.
    • The model changed across three versions within two months.
    • Prompts were designed intuitively.
    • Evaluated samples may have leaked into the proprietary training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

82 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 67 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.12867 · arXiv 2301.12867

@misc{zhuo2023red,
  title = {{Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity}},
  author = {Zhuo, Terry Yue and Huang, Yujin and Chen, Chunyang and Xing, Zhenchang},
  year = {2023},
  doi = {10.48550/arxiv.2301.12867},
  eprint = {2301.12867},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.12867}
}

References

71 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. (2021) Mum: A new ai milestone for understanding information. [Online]. Available: https://blog.google/ products/search/introducing-mum/ ↗
  2. (2021) Make every feature binary: A 135b parameter sparse neural network for massively improved search relevance. find
  3. N. team, M. R. Costa-jussà, J. Cross, O. cCelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Barrault, G. M. Gonzalez, P. Hansanti, J. Hoffman, S. Jarrett, K. R. Sadagopan, D. Rowe, S. L. Spruit, C. Tran… find
  4. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-… find
  5. M. Lee, P. Liang, and Q. Yang, “Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 2022, pp. 1–19. find
  6. E. Gibson, R. Futrell, S. P. Piantadosi, I. Dautriche, K. Mahowald, L. Bergen, and R. Levy, “How efficiency shapes human language,” Trends in cognitive sciences, vol. 23, no. 5, pp. 389–407, 2019. find
  7. A. Liptak, “Amazon’s alexa started ordering people dollhouses after hearing its name on tv,” The Verge, vol. 7, 2017. find
  8. Google home. [Online]. Available: https://home.google.com/ ↗
  9. M. J. Wolf, K. Miller, and F. S. Grodzinsky, “Why we should have seen that coming: comments on microsoft’s tay" experiment," and wider implications,” Acm Sigcas Computers and Society, vol. 47, no. 3, pp. 54–64, 2017. find
  10. N. Abdi, K. M. Ramokapane, and J. M. Such, “More than smart speakers: Security and privacy perceptions of smart home personal assistants.” in SOUPS@ USENIX Security Symposium, 2019. find
  11. J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young et al., “Scaling language models: Methods, analysis & insights from training gopher,” arXiv preprint arXiv:2112.11446, 2021. find
  12. Z. Jin, G. Chauhan, B. Tse, M. Sachan, and R. Mihalcea, “How good is nlp? a sober look at nlp tasks through the lens of social impact,” in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021, pp. 3099–3113. find
  13. T. Schuster, R. Schuster, D. J. Shah, and R. Barzilay, “The limitations of stylometry for detecting machine-generated fake news,” Computational Linguistics, vol. 46, no. 2, pp. 499–510, 2020. find
  14. P. P. Liang, C. Wu, L.-P. Morency, and R. Salakhutdinov, “Towards understanding and mitigating social biases in language models,” in International Conference on Machine Learning. PMLR, 2021, pp. 6565– 6576. find our card
  15. L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh et al., “Ethical and social risks of harm from language models,” arXiv preprint arXiv:2112.04359, 2021. find our card
  16. M. Nadeem, A. Bethke, and S. Reddy, “Stereoset: Measuring stereotypical bias in pretrained language models,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),… find our card
  17. H. R. Kirk, Y. Jun, F. Volpin, H. Iqbal, E. Benussi, F. Dreyer, A. Shtedritski, and Y. Asano, “Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models,” Advances in neural information processing systems, vol. 34, pp. 2611–2624, 2021. find
  18. N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. B. Brown, D. Song, U. Erlingsson et al., “Extracting training data from large language models.” in USENIX Security Symposium, vol. 6, 2021. find our card
  19. M. Wei and Z. Zhou, “Ai ethics issues in real world: Evidence from ai incident database,” arXiv preprint arXiv:2206.07635, 2022. — [arXiv:2206.07635](https://arxiv.org/abs/2206.07635) find
  20. E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, “Red teaming language models with language models,” arXiv preprint arXiv:2202.03286, 2022. find our card
  21. P. Henderson, K. Sinha, N. Angelard-Gontier, N. R. Ke, G. Fried, R. Lowe, and J. Pineau, “Ethical challenges in data-driven dialogue systems,” in Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, 2018, pp. 123–129. find
  22. L. Lucy and D. Bamman, “Gender and representation bias in gpt-3 generated stories,” in Proceedings of the Third Workshop on Narrative Understanding, 2021, pp. 48–55. find our card
  23. A. Abid, M. Farooqi, and J. Zou, “Persistent anti-muslim bias in large language models,” in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 2021, pp. 298–306. find our card
  24. W. M. Si, M. Backes, J. Blackburn, E. De Cristofaro, G. Stringhini, S. Zannettou, and Y. Zhang, “Why so toxic? measuring and triggering toxic behavior in open-domain chatbots,” in Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, 2022, pp. 2659–2673. find
  25. S. Roller, E. Dinan, N. Goyal, D. Ju, M. Williamson, Y. Liu, J. Xu, M. Ott, K. Shuster, E. M. Smith et al., “Recipes for building an open-domain chatbot,” arXiv preprint arXiv:2004.13637, 2020. find our card
  26. A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston, “Parlai: A dialog research software platform,” EMNLP 2017, p. 79, 2017. find our card
  27. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar et al., “Holistic evaluation of language models,” arXiv preprint arXiv:2211.09110, 2022. find
  28. D. Khurana, A. Koli, K. Khatter, and S. Singh, “Natural language processing: State of the art, current trends and challenges,” Multimedia tools and applications, pp. 1–32, 2022. find
  29. J. A. Goldstein, G. Sastry, M. Musser, R. DiResta, M. Gentzel, and K. Sedova, “Generative language models and automated influence operations: Emerging threats and potential mitigations,” arXiv preprint arXiv:2301.04246, 2023. — [arXiv:2301.04246](https://arxiv.org/abs/2301.04246) find
  30. M. Taddeo and L. Floridi, “How ai can be a force for good,” Science, vol. 361, no. 6404, pp. 751–752, 2018. find
  31. A. Jobin, M. Ienca, and E. Vayena, “The global landscape of ai ethics guidelines,” Nature Machine Intelligence, vol. 1, no. 9, pp. 389–399, 2019. find
  32. R. Higashinaka, M. Mizukami, H. Kawabata, E. Yamaguchi, N. Adachi, and J. Tomita, “Role play-based question-answering by real users for building chatbots with consistent personalities,” in Proceedings of the 19th annual sigdial meeting on discourse and dialogue, 2018, pp. 264–272. find
  33. M. Reisenbichler, T. Reutterer, D. A. Schweidel, and D. Dan, “Frontiers: Supporting content marketing with natural language generation,” Marketing Science, vol. 41, no. 3, pp. 441–452, 2022. find
  34. K. Bartz, C. Barr, and A. Aijaz, “Natural language generation for sponsored-search advertisements,” in Proceedings of the 9th ACM Conference on Electronic Commerce, 2008, pp. 1–9. find
  35. L. Cao, “Ai in finance: challenges, techniques, and opportunities,” ACM Computing Surveys (CSUR), vol. 55, no. 3, pp. 1–38, 2022. find
  36. L. Zhang, S. Wang, and B. Liu, “Deep learning for sentiment analysis: A survey,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 8, no. 4, p. e1253, 2018. find
  37. S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, and J. Gao, “Deep learning–based text classification: a comprehensive review,” ACM computing surveys (CSUR), vol. 54, no. 3, pp. 1–40, 2021. find
  38. Z. Abbasiantaeb and S. Momtazi, “Text-based question answering from information retrieval and deep neural network perspectives: A survey,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 11, no. 6, p. e1412, 2021. find
  39. Z. Talat, A. Névéol, S. Biderman, M. Clinciu, M. Dey, S. Longpre, S. Luccioni, M. Masoud, M. Mitchell, D. Radev et al., “You reap what you sow: On the challenges of bias evaluation under multilingual settings,” in Proceedings of BigScience Episode\# 5– Workshop on Challenges & Perspectives in Creat… find
  40. M. Alzantot, Y. S. Sharma, A. Elgohary, B.-J. Ho, M. Srivastava, and K.-W. Chang, “Generating natural language adversarial examples,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. find our card
  41. L. Li, R. Ma, Q. Guo, X. Xue, and X. Qiu, “Bert-attack: Adversarial attack against bert using bert,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 6193–6202. find our card
  42. K. McGuffie and A. Newhouse, “The radicalization risks of gpt-3 and advanced neural language models,” arXiv preprint arXiv:2009.06807, 2020. — [arXiv:2009.06807](https://arxiv.org/abs/2009.06807) find
  43. J. Jang, S. Ye, S. Yang, J. Shin, J. Han, G. Kim, S. J. Choi, and M. Seo, “Towards continual knowledge learning of language models,” arXiv preprint arXiv:2110.03215, 2021. find
  44. S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, “Realtoxicityprompts: Evaluating neural toxic degeneration in language models,” in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 3356–3369. find our card
  45. I. Solaiman and C. Dennison, “Process for adapting language models to society (palms) with values-targeted datasets,” Advances in Neural Information Processing Systems, vol. 34, pp. 5861–5873, 2021. find
  46. A. Parrish, A. Chen, N. Nangia, V. Padmakumar, J. Phang, J. Thompson, P. M. Htut, and S. Bowman, “Bbq: A hand-built bias benchmark for question answering,” in Findings of the Association for Computational Linguistics: ACL 2022, 2022, pp. 2086–2105. find our card
  47. J. Dhamala, T. Sun, V. Kumar, S. Krishna, Y. Pruksachatkun, K.-W. Chang, and R. Gupta, “Bold: Dataset and metrics for measuring biases in open-ended language generation,” in Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, 2021, pp. 862–872. find
  48. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020. find
  49. M. Post, “A call for clarity in reporting BLEU scores,” in Proceedings of the Third Conference on Machine Translation: Research Papers. Belgium, Brussels: Association for Computational Linguistics, Oct. 2018, pp. 186–191. [Online]. Available: https://www.aclweb.org/anthology/W18-6319 ↗ our card
  50. N. Goyal, C. Gao, V. Chaudhary, P.-J. Chen, G. Wenzek, D. Ju, S. Krishnan, M. Ranzato, F. Guzmán, and A. Fan, “The flores-101 evaluation benchmark for low-resource and multilingual machine translation,” Transactions of the Association for Computational Linguistics, vol. 10, pp. 522–538, 2022. find
  51. M. Popović, “chrf: character n-gram f-score for automatic mt evaluation,” in Proceedings of the tenth workshop on statistical machine translation, 2015, pp. 392–395. find our card
  52. H. Kim, Y. Yu, L. Jiang, X. Lu, D. Khashabi, G. Kim, Y. Choi, and M. Sap, “Prosocialdialog: A prosocial backbone for conversational agents,” arXiv preprint arXiv:2205.12688, 2022. find our card
  53. A. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” in Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, 2011, pp. 142–150. find our card
  54. C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo… find our card
  55. K. D. Dhole, V. Gangal, S. Gehrmann, A. Gupta, Z. Li, S. Mahamood, A. Mahendiran, S. Mille, A. Srivastava, S. Tan et al., “Nl-augmenter: A framework for task-sensitive natural language augmentation,” arXiv preprint arXiv:2112.02721, 2021. find our card
  56. M. Gardner, Y. Artzi, V. Basmov, J. Berant, B. Bogin, S. Chen, P. Dasigi, D. Dua, Y. Elazar, A. Gottumukkala et al., “Evaluating models’ local decision boundaries via contrast sets,” in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 1307–1323. find our card
  57. A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia et al., “Glm-130b: An open bilingual pre-trained model,” arXiv preprint arXiv:2210.02414, 2022. find our card
  58. (2022) Chatgpt has a handful of ethical constraints that are currently being tested. [Online]. Available: https://ordinary-times.com/2022/12/02/ chatgpt-has-a-handful-of-ethical-constraints-that-are-currently-b ↗
  59. T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 2381–2391. find
  60. S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 3214–3252. find our card
  61. (2022) Chatgpt: Optimizing language models for dialoguechatgpt: Optimizing language models for dialogue. [Online]. Available: https://openai.com/ blog/chatgpt/ ↗
  62. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” ACM Computing Surveys. find our card
  63. A. Gokaslan and V. Cohen, “Openwebtext corpus,” 2019. [Online]. Available: http://Skylion007.github.io/OpenWebTextCorpus ↗
  64. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485–5551, 2020. arXiv preprint arXiv:2209.03143, 2022. find our card
  65. A. Borji, “A categorical archive of chatgpt failures,” [79] P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, arXiv preprint arXiv:2302.03494, 2023. Y. Choi, and J. Gao, “Vinvl: Revisiting visual repre-[66] J. Armengol-Estapé, O. d. G. Bonet, and M. Melero, sentations in vision-language models,”… find
  66. (2022) Chatgpt is multilingual but to prompt for vision-language models,” International monocultural, and it’s learning your Journal of Computer Vision, vol. 130, no. 9, pp. 2337– values. [Online]. Available: https://jilltxt.net/ 2348, 2022. right-now-chatgpt-is-multilingual-but-monocultural-but-its… ↗
  67. Y. Goldberg. (2023) Some remarks on large language language models,” arXiv preprint arXiv:2301.10226, model. [Online]. Available: https://gist.github.com/ 2023. yoavg/59d174608e92e845c8994ac2e234c8a9 [83] X. He, Q. Xu, Y. Zeng, L. Lyu, F. Wu, J. Li, and ↗ our card
  68. E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. R. Jia, “Cater: Intellectual property protection on Manning, “Fast model editing at scale,” in Interna-text generation apis via conditional watermarks,” in tional Conference on Learning Representations. Advances in Neural Information Processing Sy… find
  69. N. D. Cao, W. Aziz, and I. Titov, “Editing factual [84] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, knowledge in language models,” in Conference on B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and Empirical Methods in Natural Language Processing, D. Amodei, “Scaling laws for neural langua… find
  70. C. Zhu, A. S. Rawat, M. Zaheer, S. Bhojanapalli, [85] A. Aghajanyan, L. Yu, A. Conneau, W.-N. Hsu, D. Li, F. X. Yu, and S. Kumar, “Modifying memories K. Hambardzumyan, S. Zhang, S. Roller, N. Goyal, in transformer models,” ArXiv, vol. abs/2012.00363, O. Levy, and L. Zettlemoyer, “Scaling laws for ge… find
  71. N. Kandpal, E. Wallace, and C. Raffel, “Deduplicating arXiv:2301.03728, 2023. training data mitigates privacy risks in language [86] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, models,” in International Conference on Machine S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, Learning. PMLR, 2022,… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhuo-2023-red-teaming-chatgpt-jailbreaking-bias/