📄 SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety

Paper · 2024 · Paul Rottger, Fabio Pernisi, Bertie Vidgen et al.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

🏛️ Institutions

🎓Bocconi UniversityMilan · Italy28 works · 3,077 citations

🧭 Fields, topics and concepts

Cite

Cite as Paul Rottger, Fabio Pernisi, Bertie Vidgen and Dirk Hovy. 2024. SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety. arXiv:2404.05399.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.05399 · arXiv 2404.05399

@misc{rottger2024safetyprompts,
  title = {{SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety}},
  author = {Paul Rottger, Fabio Pernisi, Bertie Vidgen and Dirk Hovy},
  year = {2024},
  doi = {10.48550/arxiv.2404.05399},
  eprint = {2404.05399},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.05399}
}

References

156 references, 67 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aakanksha; Ahmadian, A.; Ermis, B.; Goldfarb-Tarrant, S.; Kreutzer, J.; Fadaee, M.; and Hooker, S. 2024. The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Proceedings of the 2024 Conference on Empirical Metho…
2. Abdin, M.; Jacobs, S. A.; Awan, A. A.; Aneja, J.; Awadallah, A.; Awadalla, H.; Bach, N.; Bahree, A.; Bakhtiari, A.; Bao, J.; Behl, H.; Benhaim, A.; Bilenko, M.; Bjorck, J.; Bubeck, S.; Cai, Q.; Cai, M.; Mendes, C. C. T.; Chen, W.; Chaudhary, V.; Chen, D.; Chen, D.; Chen, Y.-C.; Chen, Y.-L.; Chopra,… — Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
3. Ahn, J.; and Oh, A. 2021. Mitigating Language-Dependent Ethnic Bias in BERT. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 533–549. Online and Punta Cana, Dominican Republic: Association for Co… find
4. Dices dataset: Diversity in conversational ai evaluation for safety. Advances in Neural Information Processing Systems, 36. Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; Das-Sarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; et al. 2022. Training a helpful and harmless assistant wit… find
5. Bajaj, D.; Lei, Y.; Tong, J.; and Huang, R. 2024. Evaluating Gender Bias of LLMs in Making Morality Judgements. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Findings of the Association for Computational Linguistics: EMNLP 2024, 15804–15818. Miami, Florida, USA: Association for Computational… find
6. Beeching, E.; Fourrier, C.; Habib, N.; Han, S.; Lambert, N.; Rajani, N.; Sanseviero, O.; Tunstall, L.; and Wolf, T. 2023. Open LLM Leaderboard. https://huggingface.co/ spaces/HuggingFaceH4/open llm leaderboard. ↗
7. Bender, E. M. 2011. On achieving and evaluating language-independence in NLP. Linguistic Issues in Language Technology, 6. find
8. Bhardwaj, R.; Do, D. A.; and Poria, S. 2024. Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volum…
9. Bhardwaj, R.; and Poria, S. 2023. Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment. arXiv:2308.09662.
10. Bhatt, M.; Chennabasappa, S.; Nikolaidis, C.; Wan, S.; Evtimov, I.; Gabi, D.; Song, D.; Ahmad, F.; Aschermann, C.; Fontana, L.; et al. 2023. Purple llama cyberseceval: A secure coding benchmark for language models. arXiv preprint arXiv:2312.04724. — [arXiv:2312.04724](https://arxiv.org/abs/2312.04724) find
11. Bhatt, S.; Dev, S.; Talukdar, P.; Dave, S.; and Prabhakaran, V. 2022. Re-contextualizing Fairness in NLP: The Case of India. In He, Y.; Ji, H.; Li, S.; Liu, Y.; and Chang, C.-H., eds., Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and… find
12. Bianchi, F.; Kalluri, P.; Durmus, E.; Ladhak, F.; Cheng, M.; Nozza, D.; Hashimoto, T.; Jurafsky, D.; Zou, J.; and Caliskan, A. 2023. Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and… find
13. Bianchi, F.; Suzgun, M.; Attanasio, G.; Rottger, P.; Jurafsky, D.; Hashimoto, T.; and Zou, J. 2024. Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions. In The Twelfth International Conference on Learning Representations.
14. Bolukbasi, T.; Chang, K.-W.; Zou, J. Y.; Saligrama, V.; and Kalai, A. T. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29.
15. Brahman, F.; Kumar, S.; Balachandran, V.; Dasigi, P.; Pyatkin, V.; Ravichander, A.; Wiegreffe, S.; Dziri, N.; Chandu, K.; Hessel, J.; Tsvetkov, Y.; Smith, N. A.; Choi, Y.; and Hajishirzi, H. 2024. The Art of Saying No: Contextual Noncompliance in Language Models. In The Thirty-eight Conference on Ne…
16. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877– 1901.
17. Caliskan, A.; Bryson, J. J.; and Narayanan, A. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334): 183–186. Carlini, N.; Nasr, M.; Choquette-Choo, C. A.; Jagielski, M.; Gao, I.; Koh, P. W.; Ippolito, D.; Tramèr, F.; and Schmidt, L. 2023. Are al… find
18. Cercas Curry, A.; Abercrombie, G.; and Rieser, V. 2021. ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Detection in Conversational AI. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 7388–7403. Online and Punta Cana, Dominican Republic: Association f…
19. Chen, Y.; Arunasalam, A.; and Celik, Z. B. 2023. Can Large Language Models Provide Security & Privacy Advice? Measuring the Ability of LLMs to Refute Misconceptions. In Proceedings of the 39th Annual Computer Security Applications Conference, ACSAC ’23, 366–378. New York, NY, USA: Association for Co… find
20. Chiang, W.-L.; Zheng, L.; Sheng, Y.; Angelopoulos, A. N.; Li, T.; Li, D.; Zhang, H.; Zhu, B.; Jordan, M.; Gonzalez, J. E.; and Stoica, I. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132.
21. Cui, J.; Chiang, W.-L.; Stoica, I.; and Hsieh, C.-J. 2024. OR-Bench: An Over-Refusal Benchmark for Large Language Models. arXiv preprint arXiv:2405.20947.
22. Cui, S.; Zhang, Z.; Chen, Y.; Zhang, W.; Liu, T.; Wang, S.; and Liu, T. 2023. Fft: Towards harmlessness evaluation and analysis for llms with factuality, fairness, toxicity. arXiv preprint arXiv:2311.18580. — [arXiv:2311.18580](https://arxiv.org/abs/2311.18580) find
23. DeepMind, G. 2024. Gemma 2: Improving Open Language Models at a Practical Size. find
24. Dhamala, J.; Sun, T.; Kumar, V.; Krishna, S.; Pruksachatkun, Y.; Chang, K.-W.; and Gupta, R. 2021. BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, 862–872. New York,… find
25. Dinan, E.; Abercrombie, G.; Bergman, A.; Spruit, S.; Hovy, D.; Boureau, Y.-L.; and Rieser, V. 2022. SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 411… find
26. Dinan, E.; Humeau, S.; Chintagunta, B.; and Weston, J. 2019. Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Languag…
27. Dua, D.; Wang, Y.; Dasigi, P.; Stanovsky, G.; Singh, S.; and Gardner, M. 2019. DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. In Burstein, J.; Doran, C.; and Solorio, T., eds., Proceedings of the 2019 Conference of the North American Chapter of the Association…
28. Durmus, E.; Nguyen, K.; Liao, T.; Schiefer, N.; Askell, A.; Bakhtin, A.; Chen, C.; Hatfield-Dodds, Z.; Hernandez, D.; Joseph, N.; Lovitt, L.; McCandlish, S.; Sikder, O.; Tamkin, A.; Thamkul, J.; Kaplan, J.; Clark, J.; and Ganguli, D. 2024. Towards Measuring the Representation of Subjective Global Op…
29. Emelin, D.; Le Bras, R.; Hwang, J. D.; Forbes, M.; and Choi, Y. 2021. Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language… find
30. Esiobu, D.; Tan, X.; Hosseini, S.; Ung, M.; Zhang, Y.; Fernandes, J.; Dwivedi-Yu, J.; Presani, E.; Williams, A.; and Smith, E. 2023. ROBBIE: Robust Bias Evaluation of Large Generative Language Models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Metho… find
31. Felkner, V.; Chang, H.-C. H.; Jang, E.; and May, J. 2023. WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Models. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume… find
32. Forbes, M.; Hwang, J. D.; Shwartz, V.; Sap, M.; and Choi, Y. 2020. Social Chemistry 101: Learning to Reason about Social and Moral Norms. In Webber, B.; Cohn, T.; He, Y.; and Liu, Y., eds., Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 653– 670. Onli… find
33. Ganguli, D.; Lovitt, L.; Kernion, J.; Askell, A.; Bai, Y.; Kadavath, S.; Mann, B.; Perez, E.; Schiefer, N.; Ndousse, K.; et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858.
34. Gao, L.; Tow, J.; Biderman, S.; Black, S.; DiPofi, A.; Foster, C.; Golding, L.; Hsu, J.; McDonell, K.; Muennighoff, N.; Phang, J.; Reynolds, L.; Tang, E.; Thite, A.; Wang, B.; Wang, K.; and Zou, A. 2021. A framework for few-shot language model evaluation. In Zenodo. find
35. Zenodo. Garg, N.; Schiebinger, L.; Jurafsky, D.; and Zou, J. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16): E3635–E3644. find
36. Gehman, S.; Gururangan, S.; Sap, M.; Choi, Y.; and Smith, N. A. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020, 3356– 3369. Online: Association for Computational Linguistics.
37. Ghanim, M. A.; Almohaimeed, S.; Zheng, M.; Solihin, Y.; and Lou, Q. 2024. Jailbreaking LLMs with Arabic Transliteration and Arabizi. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 18584– 18600. Miami, Flor…
38. Ghosh, S.; Varshney, P.; Galinkin, E.; and Parisien, C. 2024. AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts. arXiv preprint arXiv:2404.05993. — [arXiv:2404.05993](https://arxiv.org/abs/2404.05993) find
39. Guo, Y.; Cui, G.; Yuan, L.; Ding, N.; Sun, Z.; Sun, B.; Chen, H.; Xie, R.; Zhou, J.; Lin, Y.; Liu, Z.; and Sun, M. 2024. Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Proceedings of the 2024 Conference on Em… find
40. Gupta, P.; Yau, L. Q.; Low, H. H.; Lee, I.-S.; Lim, H. M.; Teoh, Y. X.; Hng, K. J.; Liew, D. W.; Bhardwaj, R.; Bhardwaj, R.; and Poria, S. 2024a. WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Models. In Hernandez Farias, D. I.; Hope, T.; and Li, M., eds., Proceedings of th… find
41. Hall, S. M.; Abrantes, F. G.; Zhu, H.; Sodunke, G.; Shtedritski, A.; and Kirk, H. R. 2023. VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
42. Han, S.; Rao, K.; Ettinger, A.; Jiang, L.; Lin, B. Y.; Lambert, N.; Choi, Y.; and Dziri, N. 2024a. WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
43. Han, T.; Kumar, A.; Agarwal, C.; and Lakkaraju, H. 2024b. MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Hansson, S.; Mavromatakis, K.; Adesam, Y.; Bouma, G.;… find
44. Hartvigsen, T.; Gabriel, S.; Palangi, H.; Sap, M.; Ray, D.; and Kamar, E. 2022. ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 330…
45. Hendrycks, D.; Burns, C.; Basart, S.; Critch, A.; Li, J.; Song, D.; and Steinhardt, J. 2021. Aligning AI With Shared Human Values. In International Conference on Learning Representations.
46. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020. Measuring Massive Multi-task Language Understanding. In International Conference on Learning Representations. find
47. Hendrycks, D.; Mazeika, M.; Zou, A.; Patel, S.; Zhu, C.; Navarro, J.; Song, D.; Li, B.; and Steinhardt, J. 2022. What Would Jiminy Cricket Do? Towards Agents That Behave Morally. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). Holtermann,… find
48. Huang, J.; Shao, H.; and Chang, K. C.-C. 2022. Are Large Pre-Trained Language Models Leaking Your Personal Information? In Goldberg, Y.; Kozareva, Z.; and Zhang, Y., eds., Findings of the Association for Computational Linguistics: EMNLP 2022, 2038–2047. Abu Dhabi, United Arab Emirates: Association f… find
49. Huang, K.; Liu, X.; Guo, Q.; Sun, T.; Sun, J.; Wang, Y.; Zhou, Z.; Wang, Y.; Teng, Y.; Qiu, X.; Wang, Y.; and Lin, D. 2024a. Flames: Benchmarking Value Alignment of LLMs in Chinese. In Duh, K.; Gomez, H.; and Bethard, S., eds., Proceedings of the 2024 Conference of the North American Chapter of the… find
50. Huang, Y.; Gupta, S.; Xia, M.; Li, K.; and Chen, D. 2024b. Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation. In The Twelfth International Conference on Learning Representations.
51. Huang, Y.; and Xiong, D. 2024. CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Models. In Calzolari, N.; Kan, M.-Y.; Hoste, V.; Lenci, A.; Sakti, S.; and Xue, N., eds., Proceedings of the 2024 Joint International Conference on Computational Linguistics,… find
52. Torino, Italia: ELRA and ICCL. Jha, A.; Mostafazadeh Davani, A.; Reddy, C. K.; Dave, S.; Prabhakaran, V.; and Dev, S. 2023. SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st An… find
53. Ji, J.; Liu, M.; Dai, J.; Pan, X.; Zhang, C.; Bian, C.; Chen, B.; Sun, R.; Wang, Y.; and Yang, Y. 2023. BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
54. Jiang, L.; Rao, K.; Han, S.; Ettinger, A.; Brahman, F.; Kumar, S.; Mireshghallah, N.; Lu, X.; Sap, M.; Choi, Y.; and Dziri, N. 2024. WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models. In The Thirty-eighth Annual Conference on Neural Information Processing Sys…
55. Jin, J.; Kim, J.; Lee, N.; Yoo, H.; Oh, A.; and Lee, H. 2024a. KoBBQ: Korean Bias Benchmark for Question Answering. Transactions of the Association for Computational Linguistics, 12: 507–524. Jin, Z.; Kleiman-Weiner, M.; Piatti, G.; Levine, S.; Liu, J.; Adauto, F. G.; Ortu, F.; Strausz, A.; Sachan,…
56. Joshi, P.; Santy, S.; Budhiraja, A.; Bali, K.; and Choudhury, M. 2020. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 6282–6293. Online: Association for Computational Linguistics.
57. Kaiyom, F.; Ahmed, A.; Mai, Y.; Klyman, K.; Bommasani, R.; and Liang, P. 2024. HELM Safety: Towards Standardized Safety Evaluations of Language Models. Stanford Center for Research on Foundation Models. find
58. Kim, H.; Yu, Y.; Jiang, L.; Lu, X.; Khashabi, D.; Kim, G.; Choi, Y.; and Sap, M. 2022. ProsocialDialog: A Prosocial Backbone for Conversational Agents. In Goldberg, Y.; Kozareva, Z.; and Zhang, Y., eds., Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 4005–402…
59. Kirk, H. R.; Jun, Y.; Volpin, F.; Iqbal, H.; Benussi, E.; Dreyer, F.; Shtedritski, A.; and Asano, Y. 2021. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. Advances in neural information processing systems, 34: 2611–2624. Kirk, H… find
60. Koenecke, A.; Nam, A.; Lake, E.; Nudell, J.; Quartey, M.; Mengesha, Z.; Toups, C.; Rickford, J. R.; Jurafsky, D.; and Goel, S. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14): 7684–7689. find
61. Kour, G.; Zalmanovici, M.; Zwerdling, N.; Goldbraich, E.; Fandina, O.; Anaby Tavor, A.; Raz, O.; and Farchi, E. 2023. Unveiling Safety Vulnerabilities of Large Language Models. In Gehrmann, S.; Wang, A.; Sedoc, J.; Clark, E.; Dhole, K.; Chandu, K. R.; Santus, E.; and Sedghamiz, H., eds., Proceedings… find
62. LakeraAI. 2023a. Gandalf Prompt Injection. In Hugging Face Dataset. find
63. LakeraAI. 2023b. Mosscap Prompt Injection. In Hugging Face Dataset. find
64. Lambert, N.; Pyatkin, V.; Morrison, J.; Miranda, L.; Lin, B. Y.; Chandu, K.; Dziri, N.; Kumar, S.; Zick, T.; Choi, Y.; Smith, N. A.; and Hajishirzi, H. 2024. Reward-Bench: Evaluating Reward Models for Language Modeling. arXiv:2403.13787. find
65. Lee, J.; Kim, M.; Kim, S.; Kim, J.; Won, S.; Lee, H.; and Choi, E. 2024. KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computational Linguistics: ACL 2024, 11177–11213. Bangkok, Thaila… find
66. Levy, S.; Adler, W.; Karver, T. S.; Dredze, M.; and Kaufman, M. R. 2024. Gender Bias in Decision-Making with Large Language Models: A Study of Relationship Conflicts. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Findings of the Association for Computational Linguistics: EMNLP 2024, 5777–580… find
67. Levy, S.; Allaway, E.; Subbiah, M.; Chilton, L.; Patton, D.; McKeown, K.; and Wang, W. Y. 2022. SafeText: A Benchmark for Exploring Physical Safety in Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2407–2421. Abu Dhabi, United Arab Emirate… find
68. Li, L.; Dong, B.; Wang, R.; Hu, X.; Zuo, W.; Lin, D.; Qiao, Y.; and Shao, J. 2024a. SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computational Linguistics: ACL 2024, 3923–3…
69. Li, N.; Pan, A.; Gopal, A.; Yue, S.; Berrios, D.; Gatti, A.; Li, J. D.; Dombrowski, A.-K.; Goel, S.; Mukobi, G.; Helm-Burger, N.; Lababidi, R.; Justen, L.; Liu, A. B.; Chen, M.; Barrass, I.; Zhang, O.; Zhu, X.; Tamirisa, R.; Bharathi, B.; Herbert-Voss, A.; Breuer, C. B.; Zou, A.; Mazeika, M.; Wang,… — The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
70. Li, T.; Khashabi, D.; Khot, T.; Sabharwal, A.; and Srikumar, V. 2020. UNQOVERing Stereotyping Biases via Underspecified Questions. In Cohn, T.; He, Y.; and Liu, Y., eds., Findings of the Association for Computational Linguistics: EMNLP 2020, 3475–3489. Online: Association for Computational Linguisti…
71. Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; Zhang, Y.; Narayanan, D.; Wu, Y.; Kumar, A.; et al. 2023. Holistic Evaluation of Language Models. Transactions on Machine Learning Research. find
72. Liang, P. P.; Wu, C.; Morency, L.-P.; and Salakhutdinov, R. 2021. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning, 6565–6576.
73. PMLR. Lin, S.; Hilton, J.; and Evans, O. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Muresan, S.; Nakov, P.; and Villavicencio, A., eds., Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3214–3252. Dublin, Ireland:…
74. Lin, Z.; Wang, Z.; Tong, Y.; Wang, Y.; Guo, Y.; Wang, Y.; and Shang, J. 2023. ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation. In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP 2023, 4694–470…
75. Liu, C.; Zhao, F.; Qing, L.; Kang, Y.; Sun, C.; Kuang, K.; and Wu, F. 2023. A chinese prompt attack dataset for llms with evil content. arXiv preprint arXiv:2309.11830. — [arXiv:2309.11830](https://arxiv.org/abs/2309.11830) find
76. Lourie, N.; Le Bras, R.; and Choi, Y. 2021. Scruples: A corpus of community ethical judgments on 32,000 real-life anecdotes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 13470–13479. find
77. Luccioni, S.; Akiki, C.; Mitchell, M.; and Jernite, Y. 2024. Stable bias: Evaluating societal representations in diffusion models. Advances in Neural Information Processing Systems, 36. find
78. Manerba, M.; Stanczak, K.; Guidotti, R.; and Augenstein, I. 2024. Social Bias Probing: Fairness Benchmarking for Language Models. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 14653–14671. Miami, Florida,… find
79. Mazeika, M.; Phan, L.; Yin, X.; Zou, A.; Wang, Z.; Mu, N.; Sakhaee, E.; Li, N.; Basart, S.; Li, B.; Forsyth, D.; and Hendrycks, D. 2024. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. In Forty-first International Conference on Machine Learning.
80. Mazeika, M.; Zou, A.; Mu, N.; Phan, L.; Wang, Z.; Yu, C.; Khoja, A.; Jiang, F.; O’Gara, A.; Sakhaee, E.; Xiang, Z.; Rajabi, A.; Hendrycks, D.; Poovendran, R.; Li, B.; and Forsyth, D. 2023. find
81. TDC 2023 (LLM Edition): The Trojan Detection Challenge. In NeurIPS Competition Track. find
82. Meta. 2024. The Llama 3 Herd of Models.
83. Meyer, J.; Rauchenstein, L.; Eisenberg, J. D.; and Howell, N. 2020. Artie Bias Corpus: An Open Dataset for Detecting Demographic Bias in Speech Applications. In Calzolari, N.; Béchet, F.; Blache, P.; Choukri, K.; Cieri, C.; Declerck, T.; Goggi, S.; Isahara, H.; Maegaard, B.; Mariani, J.; Mazo, H.;…
84. Mireshghallah, N.; Kim, H.; Zhou, X.; Tsvetkov, Y.; Sap, M.; Shokri, R.; and Choi, Y. 2024. Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory. In The Twelfth International Conference on Learning Representations. find
85. Mou, Y.; Zhang, S.; and Ye, W. 2024. SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
86. Mu, N.; Chen, S.; Wang, Z.; Chen, S.; Karamardian, D.; Aljeraisy, L.; Alomair, B.; Hendrycks, D.; and Wagner, D. 2024. Can LLMs Follow Simple Rules? arXiv:2311.04235. — [arXiv:2311.04235](https://arxiv.org/abs/2311.04235) find
87. Nadeem, M.; Bethke, A.; and Reddy, S. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Zong, C.; Xia, F.; Li, W.; and Navigli, R., eds., Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference…
88. Nangia, N.; Vania, C.; Bhalerao, R.; and Bowman, S. R. 2020. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In Webber, B.; Cohn, T.; He, Y.; and Liu, Y., eds., Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 195…
89. Nozza, D.; Bianchi, F.; and Hovy, D. 2021. HONEST: Measuring Hurtful Sentence Completion in Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2398–2406. Online: Association for Computati…
90. Ontocord.AI. 2023. OIG-Moderation. In Hugging Face Dataset. find
91. Ouyang, S.; Wang, S.; Liu, Y.; Zhong, M.; Jiao, Y.; Iter, D.; Pryzant, R.; Zhu, C.; Ji, H.; and Han, J. 2023. The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods… find
92. Pan, A.; Chan, J. S.; Zou, A.; Li, N.; Basart, S.; Woodside, T.; Zhang, H.; Emmons, S.; and Hendrycks, D. 2023. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In International Conference on Machine Learning, 26837– 26867. find
93. PMLR. Parrish, A.; Chen, A.; Nangia, N.; Padmakumar, V.; Phang, J.; Thompson, J.; Htut, P. M.; and Bowman, S. 2022. BBQ: A hand-built bias benchmark for question answering. In Muresan, S.; Nakov, P.; and Villavicencio, A., eds., Findings of the Association for Computational Linguistics: ACL 2022, 20…
94. Parrish, A.; Kirk, H. R.; Quaye, J.; Rastogi, C.; Bartolo, M.; Inel, O.; Ciro, J.; Mosquera, R.; Howard, A.; Cukierski, W.; et al. 2023. Adversarial Nibbler: A Data-Centric Challenge for Improving the Safety of Text-to-Image Models. arXiv preprint arXiv:2305.14384. — [arXiv:2305.14384](https://arxiv.org/abs/2305.14384) find
95. Perez, E.; Ringer, S.; Lukosiute, K.; Nguyen, K.; Chen, E.; Heiner, S.; Pettit, C.; Olsson, C.; Kundu, S.; Kadavath, S.; Jones, A.; Chen, A.; Mann, B.; Israel, B.; Seethor, B.; McKinnon, C.; Olah, C.; Yan, D.; Amodei, D.; Amodei, D.; Drain, D.; Li, D.; Tran-Johnson, E.; Khundadze, G.; Kernion, J.; L… — Discovering Language Model Behaviors with Model-Written Evaluations
96. Pikuliak, M.; Oresko, S.; Hrckova, A.; and Simko, M. 2024. Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Findings of the Association for Computational Linguistics: EMNLP 2024, 3060–3083. Mi… find
97. Pistilli, G.; Leidinger, A.; Jernite, Y.; Kasirzadeh, A.; Luccioni, A. S.; and Mitchell, M. 2024. CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 7(1): 1132–1144. find
98. Qi, X.; Zeng, Y.; Xie, T.; Chen, P.-Y.; Jia, R.; Mittal, P.; and Henderson, P. 2024. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! In The Twelfth International Conference on Learning Representations.
99. Qiu, H.; Zhang, S.; Li, A.; He, H.; and Lan, Z. 2023. Latent jailbreak: A benchmark for evaluating text safety and output robustness of large language models. arXiv preprint arXiv:2307.08487. — [arXiv:2307.08487](https://arxiv.org/abs/2307.08487) find
100. Radharapu, B.; Robinson, K.; Aroyo, L.; and Lahoti, P. 2023. AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications. In Wang, M.; and Zitouni, I., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, 380–39… find
101. Rashkin, H.; Smith, E. M.; Li, M.; and Boureau, Y.-L. 2019. Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. In Korhonen, A.; Traum, D.; and Màrquez, L., eds., Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 5370– 5381. Floren… find
102. Reimao, R.; and Tzerpos, V. 2019. For: A dataset for synthetic speech detection. In 2019 International Conference on Speech Technology and Human-Computer Dialogue (SpeD), 1–10. find
103. IEEE. Reuel, A.; Hardy, A.; Smith, C.; Lamparth, M.; Hardy, M.; and Kochenderfer, M. 2024. BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
104. Ricker, J.; Damm, S.; Holz, T.; and Fischer, A. 2022. Towards the detection of diffusion model deepfakes. arXiv preprint arXiv:2210.14571. Röttger, P.; Kirk, H.; Vidgen, B.; Attanasio, G.; Bianchi, F.; and Hovy, D. 2024. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large La…
105. Rudinger, R.; Naradowsky, J.; Leonard, B.; and Van Durme, B. 2018. Gender Bias in Coreference Resolution. In Walker, M.; Ji, H.; and Stent, A., eds., Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume…
106. Santurkar, S.; Durmus, E.; Ladhak, F.; Lee, C.; Liang, P.; and Hashimoto, T. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML’23.
107. JMLR.org. Scherrer, N.; Shi, C.; Feder, A.; and Blei, D. 2023. Evaluating the Moral Beliefs Encoded in LLMs. In Thirty-seventh Conference on Neural Information Processing Systems.
108. Schulhoff, S.; Pinto, J.; Khan, A.; Bouchard, L.-F.; Si, C.; Anati, S.; Tagliabue, V.; Kost, A.; Carnahan, C.; and Boyd-Graber, J. 2023. Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs Through a Global Prompt Hacking Competition. In Bouamor, H.; Pino, J.; and Bali, K., e… find
109. Schwemmer, C.; Knight, C.; Bello-Pardo, E. D.; Oklobdzija, S.; Schoonvelde, M.; and Lockhart, J. W. 2020. Diagnosing gender bias in image recognition systems. Socius, 6: 2378023120967171. find
110. Shaikh, O.; Zhang, H.; Held, W.; Bernstein, M.; and Yang, D. 2023. On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics…
111. Sharma, M.; Tong, M.; Korbak, T.; Duvenaud, D.; Askell, A.; Bowman, S. R.; DURMUS, E.; Hatfield-Dodds, Z.; Johnston, S. R.; Kravec, S. M.; Maxwell, T.; McCandlish, S.; Ndousse, K.; Rausch, O.; Schiefer, N.; Yan, D.; Zhang, M.; and Perez, E. 2024. Towards Understanding Sycophancy in Language Models.…
112. Shen, X.; Chen, Z.; Backes, M.; Shen, Y.; and Zhang, Y. 2024. ” do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 1671–1685.
113. Sheng, E.; Chang, K.-W.; Natarajan, P.; and Peng, N. 2019. The Woman Worked as a Babysitter: On Biases in Language Generation. In Inui, K.; Jiang, J.; Ng, V.; and Wan, X., eds., Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Con…
114. Shi, C.; Wang, X.; Ge, Q.; Gao, S.; Yang, X.; Gui, T.; Zhang, Q.; Huang, X.; Zhao, X.; and Lin, D. 2024. Navigating the overkill in large language models. arXiv preprint arXiv:2401.17633. find
115. Siddiq, M. L.; and Santos, J. C. S. 2022. SecurityEval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security, MSR4P&S 2022, 29–33. Ne… find
116. Smith, E. M.; Hall, M.; Kambadur, M.; Presani, E.; and Williams, A. 2022. “I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset. In Goldberg, Y.; Kozareva, Z.; and Zhang, Y., eds., Proceedings of the 2022 Conference on Empirical Methods in Natural Languag…
117. Sotnikova, A.; Cao, Y. T.; Daumé III, H.; and Rudinger, R. 2021. Analyzing Stereotypes in Generative Text Inference Tasks. In Zong, C.; Xia, F.; Li, W.; and Navigli, R., eds., Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 4052–4065. Online: Association for Computationa… find
118. Souly, A.; Lu, Q.; Bowen, D.; Trinh, T.; Hsieh, E.; Pandey, S.; Abbeel, P.; Svegliato, J.; Emmons, S.; Watkins, O.; and Toyer, S. 2024. A StrongREJECT for Empty Jailbreaks. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models. find
119. Srivastava, A.; Rastogi, A.; Rao, A.; Shoeb, A. A. M.; Abid, A.; Fisch, A.; Brown, A. R.; Santoro, A.; Gupta, A.; Garriga-Alonso, A.; et al. 2023. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.
120. Sun, D.; Abzaliev, A.; Kotek, H.; Klein, C.; Xiu, Z.; and Williams, J. 2023a. DELPHI: Data for Evaluating LLMs’ Performance in Handling Controversial Issues. In Wang, M.; and Zitouni, I., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, 82… find
121. Sun, H.; Xu, G.; Deng, J.; Cheng, J.; Zheng, C.; Zhou, H.; Peng, N.; Zhu, X.; and Huang, M. 2022. On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark. In Muresan, S.; Nakov, P.; and Villavicencio, A., eds., Findings of the Association for Computational Linguistics: ACL 2022, 390…
122. Sun, H.; Zhang, Z.; Deng, J.; Cheng, J.; and Huang, M. 2023b. Safety assessment of chinese large language models. arXiv preprint arXiv:2304.10436.
123. Sun, L.; Huang, Y.; Wang, H.; Wu, S.; Zhang, Q.; Gao, C.; Huang, Y.; Lyu, W.; Zhang, Y.; Li, X.; et al. 2024. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561.
124. Tamkin, A.; Askell, A.; Lovitt, L.; Durmus, E.; Joseph, N.; Kravec, S.; Nguyen, K.; Kaplan, J.; and Ganguli, D. 2023a. Evaluating and mitigating discrimination in language model decisions. arXiv preprint arXiv:2312.03689. find
125. Tamkin, A.; Askell, A.; Lovitt, L.; Durmus, E.; Joseph, N.; Kravec, S.; Nguyen, K.; Kaplan, J.; and Ganguli, D. 2023b. Evaluating and mitigating discrimination in language model decisions. arXiv preprint arXiv:2312.03689. Tan, X. E.; Hansanti, P.; Wood, C.; Yu, B.; Ropers, C.; and Costa-jussà, M. R… find
126. Tedeschi, S.; Friedrich, F.; Schramowski, P.; Kersting, K.; Navigli, R.; Nguyen, H.; and Li, B. 2024. ALERT: A Comprehensive Benchmark for Assessing Large Language Models’ Safety through Red Teaming. arXiv preprint arXiv:2404.08676. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.;…
127. Toyer, S.; Watkins, O.; Mendes, E.; Svegliato, J.; Bailey, L.; Wang, T.; Ong, I.; Elmaaroufi, K.; Abbeel, P.; Darrell, T.; Ritter, A.; and Russell, S. 2023. Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Follow… find
128. Ung, M.; Xu, J.; and Boureau, Y.-L. 2022. SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures. In Muresan, S.; Nakov, P.; and Villavicencio, A., eds., Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 6462–648…
129. Wang, B.; Chen, W.; Pei, H.; Xie, C.; Kang, M.; Zhang, C.; Xu, C.; Xiong, Z.; Dutta, R.; Schaeffer, R.; et al. 2024a. DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models. Advances in Neural Information Processing Systems, 36.
130. Wang, W.; Tu, Z.; Chen, C.; Yuan, Y.; Huang, J.-t.; Jiao, W.; and Lyu, M. 2024b. All Languages Matter: On the Multilingual Safety of LLMs. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computational Linguistics: ACL 2024, 5865–5877. Bangkok, Thailand: Association… find
131. Wang, Y.; Li, H.; Han, X.; Nakov, P.; and Baldwin, T. 2024c. Do-Not-Answer: Evaluating Safeguards in LLMs. In Graham, Y.; and Purver, M., eds., Findings of the Association for Computational Linguistics: EACL 2024, 896–911. St. Julian’s, Malta: Association for Computational Linguistics.
132. Wei, A.; Haghtalab, N.; and Steinhardt, J. 2023. Jailbroken: How Does LLM Safety Training Fail? arXiv preprint arXiv:2307.02483.
133. Wolfe, R.; Yang, Y.; Howe, B.; and Caliskan, A. 2023. Contrastive Language-Vision AI Models Pretrained on Web-Scraped Multimodal Data Exhibit Sexual Objectification Bias. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’23, 1174–1185. New York, NY, USA:… find
134. Xie, T.; Qi, X.; Zeng, Y.; Huang, Y.; Sehwag, U. M.; Huang, K.; He, L.; Wei, B.; Li, D.; Sheng, Y.; et al. 2024. Sorry-bench: Systematically evaluating large language model safety refusal behaviors. arXiv preprint arXiv:2406.14598.
135. Xu, G.; Liu, J.; Yan, M.; Xu, H.; Si, J.; Zhou, Z.; Yi, P.; Gao, X.; Sang, J.; Zhang, R.; et al. 2023. Cvalues: Measuring the values of chinese large language models from safety to responsibility. arXiv preprint arXiv:2307.09705.
136. Xu, J.; Ju, D.; Li, M.; Boureau, Y.-L.; Weston, J.; and Dinan, E. 2021. Bot-Adversarial Dialogue for Safe Conversational Agents. In Toutanova, K.; Rumshisky, A.; Zettlemoyer, L.; Hakkani-Tur, D.; Beltagy, I.; Bethard, S.; Cotterell, R.; Chakraborty, T.; and Zhou, Y., eds., Proceedings of the 2021 Co…
137. Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; Dong, G.; Wei, H.; Lin, H.; Tang, J.; Wang, J.; Yang, J.; Tu, J.; Zhang, J.; Ma, J.; Yang, J.; Xu, J.; Zhou, J.; Bai, J.; He, J.; Lin, J.; Dang, K.; Lu, K.; Chen, K.; Yang, K.; Li, M.; Xue, M.; Ni, N.; Zhan… find
138. Yu, E.; Li, J.; Liao, M.; Wang, S.; Zuchen, G.; Mi, F.; and Hong, L. 2024a. CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,… find
139. Yu, J.; Li, L.; and Lan, Z. 2024a. Beyond Binary Classification: A Fine-grained Safety Dataset for Large Language Models. find
140. IEEE Access. Yu, J.; Li, L.; and Lan, Z. 2024b. Beyond Binary Classification: A Fine-Grained Safety Dataset for Large Language Models. IEEE Access, 12: 64717–64726. find
141. Yu, J.; Lin, X.; Yu, Z.; and Xing, X. 2024b. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253.
142. Yu, L.; Leng, Y.; Huang, Y.; Wu, S.; Liu, H.; Ji, X.; Zhao, J.; Song, J.; Cui, T.; Cheng, X.; Liutao, L.; and Xiong, D. 2024c. CMoralEval: A Moral Evaluation Benchmark for Chinese Large Language Models. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computational… find
143. Yuan, X.; Li, J.; Wang, D.; Chen, Y.; Mao, X.; Huang, L.; Xue, H.; Wang, W.; Ren, K.; and Wang, J. 2024. S-Eval: Automatic and Adaptive Test Generation for Benchmarking Safety Evaluation of Large Language Models. arXiv preprint arXiv:2405.14191. — [arXiv:2405.14191](https://arxiv.org/abs/2405.14191) find
144. Zhang, M.; Pan, X.; and Yang, M. 2023. Jade: A linguistics-based safety evaluation platform for llm. arXiv preprint arXiv:2311.00286. — [arXiv:2311.00286](https://arxiv.org/abs/2311.00286) find
145. Zhang, W.; Lei, X.; Liu, Z.; An, M.; Yang, B.; Zhao, K.; Wang, K.; and Lian, S. 2024a. CHiSafetyBench: A Chinese Hierarchical Safety Benchmark for Large Language Models. arXiv preprint arXiv:2406.10311. — [arXiv:2406.10311](https://arxiv.org/abs/2406.10311) find
146. Zhang, Y.; Mai, Y.; Roberts, J. S.; Bommasani, R.; Dubois, Y.; and Liang, P. 2024b. HELM Instruct: A Multidimensional Instruction Following Evaluation Framework with Absolute Ratings. find
147. Zhang, Z.; Lei, L.; Wu, L.; Sun, R.; Huang, Y.; Long, C.; Liu, X.; Lei, X.; Tang, J.; and Huang, M. 2024c. SafetyBench: Evaluating the Safety of Large Language Models. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Proceedings of the 62nd Annual Meeting of the Association for Computational Lingu…
148. Zhao, D.; Wang, A.; and Russakovsky, O. 2021. Understanding and evaluating racial biases in image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 14830–14840. find
149. Zhao, J.; Fang, M.; Shi, Z.; Li, Y.; Chen, L.; and Pechenizkiy, M. 2023. CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… find
150. Zhao, J.; Wang, T.; Yatskar, M.; Ordonez, V.; and Chang, K.-W. 2018. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In Walker, M.; Ji, H.; and Stent, A., eds., Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:…
151. Zhao, W.; Mondal, D.; Tandon, N.; Dillion, D.; Gray, K.; and Gu, Y. 2024a. WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models. In Calzolari, N.; Kan, M.-Y.; Hoste, V.; Lenci, A.; Sakti, S.; and Xue, N., eds., Proceedings of the 2024 Joint Internat… find
152. Torino, Italia: ELRA and ICCL. Zhao, W.; Ren, X.; Hessel, J.; Cardie, C.; Choi, Y.; and Deng, Y. 2024b. (InThe)WildChat: 570K ChatGPT Interaction Logs In The Wild. In The Twelfth International Conference on Learning Representations. find
153. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Li, T.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Li, Z.; Lin, Z.; Xing, E.; Gonzalez, J. E.; Stoica, I.; and Zhang, H. 2024. LMSYS-1M: A Large-Scale Real-World LLM Conversation Dataset. In The Twelfth International Conference on Learning Representations. find
154. Zhou, J.; Deng, J.; Mi, F.; Li, Y.; Wang, Y.; Huang, M.; Jiang, X.; Liu, Q.; and Meng, H. 2022. Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark. In Goldberg, Y.; Kozareva, Z.; and Zhang, Y., eds., Findings of the Association for Computational Linguistics: EMNLP 2… find
155. Ziems, C.; Yu, J.; Wang, Y.-C.; Halevy, A.; and Yang, D. 2022. The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems. In Muresan, S.; Nakov, P.; and Villavicencio, A., eds., Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)…
156. Zou, A.; Wang, Z.; Carlini, N.; Nasr, M.; Kolter, J. Z.; and Fredrikson, M. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

Updated 2 November 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rottger-2024-safetyprompts-systematic-review-open-datasets/