Elamur Research⌕
☰
home / papers / 2024 / On Prompt-Driven Safeguarding for Large Language Models

On Prompt-Driven Safeguarding for Large Language Models

Paper · 2024 · Chujie Zheng, Fan Yin, Hao Zhou et al.

Summary

Safety prompts are system-level instructions placed before user input. They are a common, lightweight way to make chat LLMs decline harmful requests, but it has been unclear why they work, which makes them hard to improve automatically. The authors examine this through the models' internal representations. They built a small, controlled set of paired harmful and harmless "How to X" queries and projected the last-token hidden states of eight open 7B chat models with PCA. Two findings emerge. First, models already separate harmful from harmless queries without any safety prompt. Second, safety prompts mainly push all queries toward a region where refusal is more likely, which also causes needless refusals of benign requests. Building on this, they propose DRO (Directed Representation Optimization). DRO keeps the model frozen and treats the safety prompt as trainable continuous embeddings. It learns to shift harmful queries toward an estimated refusal direction and harmless ones away from it, with a regularizer that keeps representations from degenerating. DRO clearly strengthens human-written prompts on jailbreak and out-of-domain test sets while preserving general ability.

Key points

  • Synthetic data: 100 harmful and 100 harmless queries were generated with gpt-3.5-turbo. Each pair shares the same verb, the set was manually checked, and lengths were matched (about 14.0 vs. 13.8 LLaMA tokens on average).
  • Models: eight 7B chat models were tested: llama-2-chat, codellama-instruct, vicuna-v1.5, orca-2, mistral-instruct-v0.1/0.2 and openchat-3.5.
  • Evaluation setup:
    • Three safety prompts were compared: the official LLaMA-2 prompt, the official Mistral prompt, and a shortened LLaMA-2 variant.
    • Each query was sampled 20 times (top-p 0.9).
    • Refusals of harmful queries were judged with LlamaGuard; refusals of harmless queries were detected by string matching.
  • Main result: with the optimized LLaMA-2 prompt, compliance with harmful AdvBench queries fell from 10.3% to 1.4%. General quality on AlpacaEval was not degraded.
  • Venue and code: the paper appeared at ICML 2024, and the code is available at github.com/chujiezheng/LLM-Safeguard.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🏢TencentShenzhen · China163 works · 21,032 citations🎓University of California, Los AngelesLos Angeles · United States99 works · 56,330 citations🔬Beijing National Research Center for Information Science and TechnologyBeijing · China33 works · 6,706 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksBias, fairness & safety · 587 works

Cite

Cite as Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang and Nanyun Peng. 2024. On Prompt-Driven Safeguarding for Large Language Models. arXiv:2401.18018.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.18018 · arXiv 2401.18018

@misc{zheng2024prompt,
  title = {{On Prompt-Driven Safeguarding for Large Language Models}},
  author = {Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang and Nanyun Peng},
  year = {2024},
  doi = {10.48550/arxiv.2401.18018},
  eprint = {2401.18018},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.18018}
}

References

38 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adolphs, L., Gao, T., Xu, J., Shuster, K., Sukhbaatar, S., and Weston, J. The CRINGE loss: Learning what language not to model. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023. URL https://aclanthology.org/2023.acl-long.493. ↗
  2. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. find
  3. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b.
  4. Bhatt, M., Chennabasappa, S., Nikolaidis, C., Wan, S., Evtimov, I., Gabi, D., Song, D., Ahmad, F., Aschermann, C., Fontana, L., et al. Purple llama cyberseceval: A secure coding benchmark for language models. arXiv preprint arXiv:2312.04724, 2023. — [arXiv:2312.04724](https://arxiv.org/abs/2312.04724) find
  5. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/. ↗
  6. Deng, M., Wang, J., Hsieh, C.-P., Wang, Y., Guo, H., Shu, T., Song, M., Xing, E., and Hu, Z. RLPrompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022. URL https://aclanthology.org/2022.emn… ↗
  7. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rygGQyrFvH. ↗
  8. Huang, Y., Gupta, S., Xia, M., Li, K., and Chen, D. Catastrophic jailbreak of open-source LLMs via exploiting generation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=r42tSSCHPh. ↗
  9. Ivison, H., Wang, Y., Pyatkin, V., Lambert, N., Peters, M., Dasigi, P., Jang, J., Wadden, D., Smith, N. A., Beltagy, I., et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702, 2023.
  10. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. find
  11. Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021. URL https://aclanthology.org/2021.emnlp-main.243. ↗
  12. Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023. ↗
  13. Li, X., Yu, P., Zhou, C., Schick, T., Zettlemoyer, L., Levy, O., Weston, J., and Lewis, M. Self-alignment with instruction backtranslation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=1oijHJBRsT. ↗
  14. Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computat… find
  15. Liu, X., Xu, N., Chen, M., and Xiao, C. AutoDAN: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=7Jwpw4qKkb. ↗
  16. Mitra, A., Del Corro, L., Mahajan, S., Codas, A., Simoes, C., Agarwal, S., Chen, X., Razdaibiedina, A., Jones, E., Aggarwal, K., et al. Orca 2: Teaching small language models how to reason. arXiv preprint arXiv:2311.11045, 2023.
  17. OpenAI. Gpt-4 technical report. arXiv:2303.08774, 2023. arXiv preprint Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023.
  18. Shaikh, O., Zhang, H., Held, W., Bernstein, M., and Yang, D. On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), July 2023. URL https://aclanthol…
  19. Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. Towards Controllable Biases in Language Generation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3239–3254, November 2020. doi: 10.18653/v1/2020.findings-emnlp.291. URL https://aclanthology.org/2020.findings-emnlp… ↗
  20. Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. URL https://aclanthology.org/2020.emn… ↗
  21. Sun, H., Xu, G., Deng, J., Cheng, J., Zheng, C., Zhou, H., Peng, N., Zhu, X., and Huang, M. On the safety of conversational models: Taxonomy, dataset, and benchmark. In Findings of the Association for Computational Linguistics: ACL 2022, 2022. URL https://aclanthology.org/2022.findings-acl.308. ↗
  22. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  23. Wang, G., Cheng, S., Zhan, X., Li, X., Song, S., and Liu, Y. Openchat: Advancing open-source language models with mixed-quality data. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=AOJyfhWYHf. ↗
  24. Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How does LLM safety training fail? In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=jA235JGM09. ↗
  25. Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021.
  26. Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., and Weston, J. Neural text generation with unlikelihood training. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SJeYe0NtvH. ↗
  27. Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., and Dinan, E. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079, 2020.
  28. Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023.
  29. Zeng, Y., Lin, H., Zhang, J., Yang, D., Jia, R., and Shi, W. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373, 2024.
  30. Zhao, J., Khashabi, D., Khot, T., Sabharwal, A., and Chang, K.-W. Ethical-advice taker: Do language models understand natural language interventions? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4158–4164, August 2021. doi: 10.18653/v1/2021.findings-acl.364. URL… find
  31. Zheng, C. Chat templates for huggingface large language models. https://github.com/chujiezheng/chat_templates, 2023. ↗
  32. Zheng, C. and Huang, M. Exploring prompt-based few-shot learning for grounded dialog generation. arXiv preprint arXiv:2109.06513, 2021. — [arXiv:2109.06513](https://arxiv.org/abs/2109.06513) find
  33. Zheng, C., Ke, P., Zhang, Z., and Huang, M. Click: Controllable text generation with sequence likelihood contrastive learning. In Findings of the Association for Computational Linguistics: ACL 2023, 2023. URL https://aclanthology.org/2023.findings-acl.65. ↗
  34. Zheng, C., Wang, Z., Ji, H., Huang, M., and Peng, N. Weak-to-strong extrapolation expedites alignment. arXiv preprint arXiv:2404.16792, 2024a. — [arXiv:2404.16792](https://arxiv.org/abs/2404.16792) find
  35. Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2024b. URL https://openreview.net/forum?id=shr9PXz7T0. ↗
  36. Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=92gvk82DE-. ↗
  37. Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.
  38. A. Open-Source Models and Data Used in This Work llama-2-chat codellama-instruct vicuna-v1.5 orca-2 mistral-instruct-v0.1 mistral-instruct-v0.2 openchat-3.5 openchat-3.5-1210 LlamaGuard https://huggingface.co/meta-llama/Llama-2-7b-chat-hf https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf htt… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2024-prompt-driven-safeguarding-large-language/