Elamur Research
☰
home / papers / 2023 / Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations

Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2312.06674

Summary

Existing moderation services such as Perspective API, the OpenAI Content Moderation API and Azure Content Safety are poorly suited to guarding chatbots. They judge user and assistant text the same way, enforce a single fixed policy, are available only through APIs, and run on small transformer backbones. Meta's GenAI team addresses this with Llama Guard, a Llama2-7b model instruction-tuned to work as a safety classifier for both user prompts and model responses. The authors define a six-category risk taxonomy with guidelines and build a small, carefully labeled dataset seeded from Anthropic's harmlessness preference data. They frame moderation as an instruction task: the input contains the policy, which side of the dialogue to judge, the conversation, and the required output format. On ToxicChat, Llama Guard outperforms all compared baselines in AUPRC. On the OpenAI Moderation Evaluation set, its zero-shot results match the OpenAI API, and adding in-context examples lets it surpass that API. Since the policy sits in the prompt and the weights are public, developers can adapt the model to their own taxonomies at low cost.

Key points

  • Six risk categories make up the taxonomy: Violence & Hate; Sexual Content; Guns & Illegal Weapons; the category for Regulated or Controlled Substances; the category for Suicide & Self Harm; and Criminal Planning, which applies only when nothing else fits.
  • The model first outputs a single SentencePiece token, "safe" or "unsafe". For unsafe content it then lists the violated category codes (e.g., O1). The probability of the first token serves as a classifier score, which supports binary, multi-label and one-vs-all classification.
  • Prompt and response classification are handled by one model, switched only by changing the instruction wording. The authors describe this as the first explicit separation of the two moderation tasks.
  • New policies can be applied without retraining. Zero-shot prompting uses category names or descriptions, and few-shot prompting adds 2–4 in-context examples per category, including hard negative safe examples.
  • Further fine-tuning on ToxicChat is more data-efficient when starting from Llama Guard than from base Llama2-7b. Limitations include English-only fine-tuning data, imperfect policy coverage, and exposure to prompt injection. Weights are released through the PurpleLlama repository.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning · Computing & GPUs

Cited by

314 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 305 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.06674 · arXiv 2312.06674

@misc{inan2023llama,
  title = {{Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations}},
  author = {Inan, Hakan and Upasani, Kartikeya and Chi, Jianfeng and Rungta, Rashi and Iyer, Krithika and Mao, Yuning and Tontchev, Michael and Hu, Qing and Fuller, Brian and Testuggine, Davide and Khabsa, Madian},
  year = {2023},
  doi = {10.48550/arxiv.2312.06674},
  eprint = {2312.06674},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.06674}
}

References

29 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. find our card
  2. Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop… find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. find
  4. Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers, 2023. find
  5. Amanda Cercas Curry and Verena Rieser. \#MeToo Alexa: How conversational systems respond to sexual harassment. In Mark Alfano, Dirk Hovy, Margaret Mitchell, and Michael Strube, editors, Proceedings of the Second ACL Workshop on Ethics in Natural Language Processing, pages 7–14, New Orleans, Louisiana… find
  6. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models, 2023. find our card
  7. Ona de Gibert, Naiara Perez, Aitor Garcı́a-Pablos, and Montse Cuadros. Hate Speech Dataset from a White Supremacy Forum. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 11–20, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18… find our card
  8. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. Build it break it fix it for dialogue safety: Robustness from adversarial human attack, 2019. find our card
  9. Emily Dinan, Gavin Abercrombie, A. Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. Anticipating safety issues in e2e conversational ai: Framework and tooling, 2021. find
  10. Rishav Hada, Sohi Sudhir, Pushkar Mishra, Helen Yannakoudakis, Saif M. Mohammad, and Ekaterina Shutova. Ruddit: Norms of offensiveness for English Reddit comments. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Compu… find
  11. Alon Halevy, Cristian Canton-Ferrer, Hao Ma, Umut Ozertem, Patrick Pantel, Marzieh Saeidi, Fabrizio Silvestri, and Ves Stoyanov. Preserving integrity in online social networks. Communications of the ACM, 65(2):92–98, 2022. find
  12. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… find
  13. Google Jigsaw. Perspective api. https://www.perspectiveapi.com/, 2017. ↗
  14. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes, 2021. find our card
  15. Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, 2018. find our card
  16. Nayeon Lee, Andrea Madotto, and Pascale Fung. Exploring social bias in chatbots using stereotype knowledge. In Amittai Axelrod, Diyi Yang, Rossana Cunha, Samira Shaikh, and Zeerak Waseem, editors, Proceedings of the 2019 Workshop on Widening NLP, pages 177–180, Florence, Italy, August 2019. Associat… — [ACL W19-3655](https://aclanthology.org/W19-3655/) find
  17. Alyssa Lees, Vinh Q. Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. A new generation of perspective api: Efficient multilingual character-level transformers, 2022. find
  18. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… find
  19. Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation, 2023. find
  20. Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15009–15018, 20… find our card
  21. Meta. Llama 2 responsible use guide. https://ai.meta.com/static-resource/responsible-use-guide/, 2023. ↗
  22. OpenAI. Gpt-4 technical report, 2023. find our card
  23. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. find our card
  24. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. find
  25. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2022a. find
  26. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022b. find our card
  27. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023. find
  28. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In Jonathan May, Ekaterina Shutova, Aurelie Herbelot, Xiaodan Zhu, Marianna Apidianaki, and Saif M. Moham… find our card
  29. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023. find