Elamur Research⌕
☰
home / papers / 2023 / Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment

Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment

Paper · 2023 · Rishabh Bhardwaj and Soujanya Poria

Summary

Large language models can still be pushed into producing dangerous content. Older safety probes, such as appending Chain-of-Thought cues to harmful questions, are now mostly refused by current models. The authors present Red-Eval, a red-teaming benchmark based on Chain of Utterances (CoU) prompting. The prompt stages a dialogue between a malicious Red-LM and an overly compliant Base-LM, gives a worked demonstration, and asks the tested model to write the next Base-LM turn, starting with its "internal thoughts". This attack succeeds on roughly 65% of harmful queries for GPT-4, 73% for ChatGPT, and over 86% for open-source LLaMA-based models. The second contribution, Red-Instruct, uses ChatGPT to build HarmfulQA: harmful questions paired with safe (blue) and jailbroken harmful (red) conversations. Vicuna-7B is then fine-tuned with Safe-Align, which trains on blue responses and optionally first pushes the model away from red ones through gradient ascent. The resulting model, Starling, resists red-teaming better and improves on HHH, while TruthfulQA, MMLU and BBH scores stay near the baseline.

Key points

  • HarmfulQA: 1,960 harmful questions across 10 topics with sub-topics, plus 9,536 blue conversations (~66K turns) and 7,356 red conversations (~52K turns), all generated with ChatGPT. It is released on Hugging Face as declare-lab/HarmfulQA, and the model as declare-lab/starling-7B.
  • Red-Eval attack success:
    • On DangerousQA (200 questions), success is 0.367 for GPT-4 and 0.736 for ChatGPT.
    • On HarmfulQA, it is 0.452 and 0.702 respectively.
    • Standard and CoT prompting stay near zero for both systems.
  • Two Safe-Align variants:
    • Strategy-A trains only on blue data.
    • Strategy-B first applies gradient ascent on red responses, then trains on blue data.
    • Strategy-B proved unstable: with more than 200 steps, generation quality and MMLU collapsed, which required a learning rate of 1e-5 and mixing in ShareGPT data.
  • Starling (blue-red) on HHH: the average HHH score rises by more than 6%, with harmlessness up over 12% and helpfulness up over 9%, at a cost of about 5% in honesty.
  • Utility (Table 6):
    • TruthfulQA improves from 46.99 for Vicuna-7B to 49.60 for Starling (blue-red) and 48.90 for Starling (blue).
    • MMLU dips slightly, from 47.18 to about 46.5–46.7.
    • BBH stays around 33.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Singapore University of Technology and DesignSingapore · Singapore54 works · 11,534 citations

Also on the byline: Gender Studies.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksBias, fairness & safety · 587 works

Cite

Cite as Rishabh Bhardwaj and Soujanya Poria. 2023. Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment. arXiv:2308.09662.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.09662 · arXiv 2308.09662

@misc{bhardwaj2023red,
  title = {{Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment}},
  author = {Rishabh Bhardwaj and Soujanya Poria},
  year = {2023},
  doi = {10.48550/arxiv.2308.09662},
  eprint = {2308.09662},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.09662}
}

References

27 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris… find
  2. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/… ↗
  3. Jiawen Deng, Hao Sun, Zhexin Zhang, Jiale Cheng, and Minlie Huang. Recent advances towards safe, responsible, and moral dialogue systems: A survey. arXiv preprint arXiv:2302.09270, 2023. — [arXiv:2302.09270](https://arxiv.org/abs/2302.09270) find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  5. Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459, 2023.
  6. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. arXiv preprint arXiv:2008.02275, 2020.
  7. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  8. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  9. Firuz Kamalov and Ikhlaas Gurrib. A new era of artificial intelligence in education: A multifaceted revolution. CoRR, abs/2305.18303, 2023. — [arXiv:2305.18303](https://arxiv.org/abs/2305.18303) find
  10. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022.
  11. Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021.
  12. Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Languages are rewards: Hindsight finetuning using human feedback. arXiv preprint arXiv:2302.02676, 2023.
  13. Oded Nov, Nina Singh, and Devin M. Mann. Putting chatgpt’s medical advice to the (turing) test. CoRR, abs/2301.10035, 2023. — [arXiv:2301.10035](https://arxiv.org/abs/2301.10035) find
  14. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193, 2021.
  15. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301, 2018.
  16. Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning. arXiv preprint arXiv:2212.08061, 2022.
  17. Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning. arXiv preprint arXiv:2212.08061, 2022.
  18. Irene Solaiman and Christy Dennison. Process for adapting language models to society (palms) with values-targeted datasets. Advances in Neural Information Processing Systems, 34:5861– 5873, 2021.
  19. Guijin Son, Hanearl Jung, Moonjeong Hahm, Keonju Na, and Sol Jin. Beyond classification: Financial reasoning in state-of-the-art language models. CoRR, abs/2305.01505, 2023. find
  20. Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision, 2023.
  21. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
  22. Ruixiang Tang, Xiaotian Han, Xiaoqian Jiang, and Xia Hu. Does synthetic data generation of llms help clinical text mining? arXiv preprint arXiv:2303.04360, 2023. — [arXiv:2303.04360](https://arxiv.org/abs/2303.04360) find
  23. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, 2023.
  24. Dietrich Trautmann, Alina Petrova, and Frank Schilder. Legal prompt engineering for multilingual legal judgement prediction. CoRR, abs/2212.02199, 2022. find
  25. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large l… — A Survey of Large Language Models
  26. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023.
  27. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bhardwaj-2023-red-teaming-large-language-models/