Elamur Research⌕
☰
home / papers / 2023 / Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs

Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs

Paper · 2023 · Yuxia Wang, Haonan Li, Xudong Han et al.

Summary

Open-source LLMs often ship without strong safety mechanisms, and earlier harmful-prompt collections were either private, narrow in scope, or lacked labeled model responses. This work addresses that gap with Do-Not-Answer, a publicly released benchmark for testing whether text-only LLMs decline requests they should not fulfil. The authors extend the risk framework of Weidinger et al. (2021) into a three-level taxonomy and gather at least ten prompts per fine-grained category. Six models were run on every prompt, and their replies were manually annotated, both for harmfulness and for the kind of response given (e.g., refusing or complying directly). LLaMA-2 proved the most cautious system and ChatGLM2 the least. The authors also fine-tuned compact classifiers, such as Longformer and BERT, on the annotations. These classifiers judged safety almost as well as GPT-4 used as an evaluator, and they produced nearly the same model ranking. Cheap, reproducible safety auditing therefore becomes feasible without relying on a commercial API.

Key points

  • The taxonomy has 5 risk areas, 12 harm types and 60 specific risk types, which together yield 939 prompts. All prompts are risky by design.

  • More than 5,000 responses were hand-labeled, drawn from GPT-4, ChatGPT, Claude, LLaMA-2, ChatGLM2 and Vicuna. By human judgment, 99.7% of LLaMA-2's responses were harmless, against 90.9% for ChatGLM2.

  • In harmful-response detection, both GPT-4 and a fine-tuned Longformer (under 600M parameters) exceed 98% accuracy. The automatic rankings match the human ranking except that ChatGPT and Claude swap places.

  • Ablations show two effects:

    • Supplying the instruction together with the response helps the classifiers, especially for action classification.
    • Longformer's longer context improves on BERT mainly for categories that require reading the full answer.

  • Limitations:

    • The dataset contains no benign prompts, so over-refusing models cannot be detected.
    • Evaluation covers only English, single-turn, zero-shot use.
    • Annotation is single-label.

    Data and code are available at github.com/Libr-AI/do-not-answer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia60 works · 21,199 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksSocial Sciences · 744 worksSafety Research · 221 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksBias, fairness & safety · 209 works

Cite

Cite as Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov and Timothy Baldwin. 2023. Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs. arXiv:2308.13387.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.13387 · arXiv 2308.13387

@misc{wang2023do,
  title = {{Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs}},
  author = {Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov and Timothy Baldwin},
  year = {2023},
  doi = {10.48550/arxiv.2308.13387},
  eprint = {2308.13387},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.13387}
}

References

26 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150.
  2. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in ChatGPT: Analyzing persona-assigned language models. CoRR, abs/2304.05335.
  3. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. BOLD: dataset and metrics for measuring biases in open-ended language generation. In FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toron… find
  4. Emilio Ferrara. 2023. Should ChatGPT be biased? challenges and risks of bias in large language models. CoRR, abs/2304.03738. find
  5. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858.
  6. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguisti…
  7. Xudong Han, Timothy Baldwin, and Trevor Cohn. 2023a. Everybody needs good neighbours: An unsupervised locality-based method for bias mitigation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. find
  8. Xudong Han, Timothy Baldwin, and Trevor Cohn. 2023b. Fair enough: Standardizing evaluation and model selection for fairness research in NLP. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, p… find
  9. Xudong Han, Aili Shen, Trevor Cohn, Timothy Baldwin, and Lea Frermann. 2022. Systematic evaluation of predictive fairness. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language… find
  10. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:…
  11. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. BeaverTails: Towards improved safety alignment of LLM via a human-preference dataset. arXiv preprint arXiv:2307.04657.
  12. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… find
  13. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
  14. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. find
  15. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Min…
  16. OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.
  17. Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul F. Christiano, and Allan Dafoe. 2023. Model evaluation for extreme risks. CoRR, abs/2305.15324. find
  18. Shivashankar Subramanian, Xudong Han, Timothy Baldwin, Trevor Cohn, and Lea Frermann. 2021. Evaluating debiasing techniques for intersectional biases. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic… find
  19. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  20. Sander Van Der Linden. 2022. Misinformation: susceptibility, spread, and interventions to immunize the public. Nature Medicine, 28(3):460–467. find
  21. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does LLM safety training fail? arXiv preprint arXiv:2307.02483.
  22. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903.
  23. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational…
  24. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean L…
  25. Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2023. FLASK: fine-grained la…
  26. Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023. Exploring AI ethics of ChatGPT: A diagnostic analysis. arXiv preprint arXiv:2301.12867.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2023-answer-dataset-evaluating-safeguards-llms/