Elamur Research⌕
☰
home / papers / 2025 / Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment

Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment

Paper · 2025 · Melissa Kazemi Rad, Huy Nghiem, Andy Luo et al.

Summary

Conversational assistants built on LLMs need an input filter that catches fraudulent requests, illegal-activity requests and jailbreak attempts before they reach the main agent. A plain prompted LLM-as-a-Judge often reasons poorly, misses instructions buried in long prompts and breaks the required output format. The authors, from Capital One and the University of Maryland, built a small balanced training set of 400 queries and a test set of about 6,800. Harmful examples come from public jailbreak and malicious-prompt collections, and safe queries were generated synthetically. Each training query was paired with manually checked Chain-of-Thought answers, some preferred and some rejected. Four open models were first adapted with LoRA-based SFT and then aligned with DPO or KTO, so that they give a binary verdict with a short justification. SFT accounts for most of the improvement, and preference alignment adds small further gains. The best model, a DPO-tuned Llama3 8B Instruct, clearly beats existing guardrail models. The study shows that cheap, low-data tuning can produce a reliable, parseable safety judge.

Key points

  • Data: the positive class draws on AdvBench, MaliciousInstruct, Forbidden Question Set and Jailbreak Prompt Set. Negatives are synthetic everyday queries. The split is 400 training and roughly 6,800 test examples, balanced 50/50.
  • Models and methods: the study tests Mistral 7B Instruct v2, Mixtral 8x7B Instruct v1, Llama2 13B Chat and Llama3 8B Instruct in zero-shot mode. Each model is compared across its base version, SFT with LoRA, SFT + DPO and SFT + KTO. An #END stop token was added to curb repetitive outputs.
  • Gains from SFT and alignment: SFT raised F1 by up to 227% and attack detection ratio (ADR) by up to 344% over the base models. KTO usually edged out DPO, except on Llama3. False positive rate rose slightly, for example from 0.1% to 1.6% for KTO-tuned Llama3.
  • Output format: the share of malformed answers fell for every model. For Llama3 8B Instruct it dropped from 16.8% to 0.3% after DPO.
  • Comparison with existing guardrails: Llama3-DPO scored F1 96.1, ADR 93.3 and FPR 0.8, against LlamaGuard-2 at 69.2 / 54.2 / 2.2. On jailbreak-only positives, ProtectAI DeBERTaV3 reached ADR 71.2. PromptGuard reached a high ADR of 98.2, but it flagged nearly everything, with an FPR of 99.8.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations🏢Capital OneTysons Corner · United States6 works · 16 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksBias, fairness & safety · 517 works

Cite

Cite as Melissa Kazemi Rad, Huy Nghiem, Andy Luo, Sahil Wadhwa, Mohammad Sorower and Stephen Rawls. 2025. Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment. arXiv:2501.13080.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.13080 · arXiv 2501.13080

@misc{rad2025refining,
  title = {{Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment}},
  author = {Melissa Kazemi Rad, Huy Nghiem, Andy Luo, Sahil Wadhwa, Mohammad Sorower and Stephen Rawls},
  year = {2025},
  doi = {10.48550/arxiv.2501.13080},
  eprint = {2501.13080},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.13080}
}

References

51 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdali, S.; Anarfi, R.; Barberan, C.; and He, J. 2024. Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices. arXiv:2403.12503. — [arXiv:2403.12503](https://arxiv.org/abs/2403.12503) find
  2. Abedu, S.; Abdellatif, A.; and Shihab, E. 2024. LLM-Based Chatbots for Mining Software Repositories: Challenges and Opportunities. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, 201–210. find
  3. Adlakha, V.; BehnamGhader, P.; Lu, X. H.; Meade, N.; and Reddy, S. 2024. Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering. arXiv:2307.16877. Almazrouei, E.; Alobeidli, H.; Alshamsi, A.; Cappelli, A.; Cojocaru, R.; Debbah, M.; Étienne Goffinet; Hesslow,…
  4. Anil, C.; Durmus, E.; Sharma, M.; Benton, J.; Kundu, S.; Batson, J.; Rimsky, N.; Tong, M.; Mu, J.; Ford, D.; Mosconi, F.; Agrawal, R.; Schaeffer, R.; Bashkansky, N.; Svenningsen, S.; Lambert, M.; Radhakrishnan, A.; et al. 2024. find
  5. Many-shot Jailbreaking. Biswas, A.; and Talukdar, W. 2023. Guardrails for trust, safety, and ethical development and deployment of Large Language Models (LLM). Journal of Science & Technology, 4(6): 55–82. find
  6. Chao, P.; Robey, A.; Dobriban, E.; Hassani, H.; Pappas, G. J.; and Wong, E. 2024. Jailbreaking Black Box Large Language Models in Twenty Queries. arXiv:2310.08419.
  7. Chen, C.; and Shu, K. 2023. Combating misinformation in the age of llms: Opportunities and challenges. find
  8. AI Magazine. Dam, S. K.; Hong, C. S.; Qiao, Y.; and Zhang, C. 2024. A Complete Survey on LLM-based AI Chatbots. arXiv:2406.16937. — [arXiv:2406.16937](https://arxiv.org/abs/2406.16937) find
  9. Dong, Y.; Mu, R.; Jin, G.; Qi, Y.; Hu, J.; Zhao, X.; Meng, J.; Ruan, W.; and Huang, X. 2024. Building Guardrails for Large Language Models. arXiv:2402.01822. — [arXiv:2402.01822](https://arxiv.org/abs/2402.01822) find
  10. Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; Goyal, A.; Hartshorn, A.; Yang, A.; Mitra, A.; et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783.
  11. Es, S.; James, J.; Espinosa-Anke, L.; and Schockaert, S. 2023. RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217.
  12. Ethayarajh, K.; Xu, W.; Muennighoff, N.; Jurafsky, D.; and Kiela, D. 2024. KTO: Model Alignment as Prospect Theoretic Optimization. arXiv:2402.01306. find
  13. Hadi, M. U.; Qureshi, R.; Shah, A.; Irfan, M.; Zafar, A.; Shaikh, M. B.; Akhtar, N.; Wu, J.; Mirjalili, S.; et al. 2023. Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects. find
  14. Authorea Preprints. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
  15. Huang, Y.; Gupta, S.; Xia, M.; Li, K.; and Chen, D. 2023. Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation. arXiv:2310.06987.
  16. Inan, H.; Upasani, K.; Chi, J.; Rungta, R.; Iyer, K.; Mao, Y.; Tontchev, M.; Hu, Q.; Fuller, B.; Testuggine, D.; and Khabsa, M. 2023. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. arXiv:2312.06674. Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; d…
  17. Jones, E.; Dragan, A.; Raghunathan, A.; and Steinhardt, J. 2023. Automatically Auditing Large Language Models via Discrete Optimization. arXiv:2303.04381. Li, J.; Sun, S.; Yuan, W.; Fan, R.-Z.; hai zhao; and Liu, P. 2024. Generative Judge for Evaluating Alignment. In The Twelfth International Confer…
  18. Li, X.; Zhou, Z.; Zhu, J.; Yao, J.; Liu, T.; and Han, B. 2023. DeepInception: Hypnotize Large Language Model to Be Jailbreaker. arXiv:2311.03191.
  19. Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P. 2023. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172.
  20. Liu, X.; Yu, Z.; Zhang, Y.; Zhang, N.; and Xiao, C. 2024. Automatic and Universal Prompt Injection Attacks against Large Language Models. arXiv:2403.04957. — [arXiv:2403.04957](https://arxiv.org/abs/2403.04957) find
  21. Luo, Y.; Yang, Z.; Meng, F.; Li, Y.; Zhou, J.; and Zhang, Y. 2024. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning. arXiv:2308.08747. find
  22. Manakul, P.; Liusie, A.; and Gales, M. J. F. 2023. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. arXiv:2303.08896. find
  23. Mecklenburg, N.; Lin, Y.; Li, X.; Holstein, D.; Nunes, L.; Malvar, S.; Silva, B.; Chandra, R.; Aski, V.; Yannam, P. K. R.; Aktas, T.; and Hendry, T. 2024. Injecting New Knowledge into Large Language Models via Supervised Fine-Tuning. arXiv:2404.00213. — [arXiv:2404.00213](https://arxiv.org/abs/2404.00213) find
  24. Mehrotra, A.; Zampetakis, M.; Kassianik, P.; Nelson, B.; Anderson, H.; Singer, Y.; and Karbasi, A. 2024. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. arXiv:2312.02119. Min, S.; Krishna, K.; Lyu, X.; Lewis, M.; tau Yih, W.; Koh, P. W.; Iyyer, M.; Zettlemoyer, L.; and Hajishirzi, H. 202…
  25. Nghiem, H.; Prindle, J.; Zhao, J.; and au2, H. D. I. 2024. ”You Gotta be a Doctor, Lin”: An Investigation of Name-Based Bias of Large Language Models in Employment Recommendations. arXiv:2406.12232. find
  26. OpenAI; Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; Avila, R.; Babuschkin, I.; Balaji, S.; Balcom, V.; et al. 2024. GPT-4 Technical Report. arXiv:2303.08774. OpenAI Usage Policies. ???? OpenAI. Usage policies. http…
  27. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training language models to follow instruction…
  28. ProtectAI.com. 2023. Fine-Tuned DeBERTa-v3 for Prompt Injection Detection. find
  29. Rafailov, R.; Sharma, A.; Mitchell, E.; Ermon, S.; Manning, C. D.; and Finn, C. 2024. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290.
  30. Rebedea, T.; Dinu, R.; Sreedhar, M.; Parisien, C.; and Cohen, J. 2023. NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails. arXiv:2310.10501. find
  31. Ren, W.; Li, X.; Wang, L.; Zhao, T.; and Qin, W. 2024. Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning. arXiv:2402.18865. — [arXiv:2402.18865](https://arxiv.org/abs/2402.18865) find
  32. Schimanski, T.; Ni, J.; Kraus, M.; Ash, E.; and Leippold, M. 2024. Towards Faithful and Robust LLM Specialists for Evidence-Based Question-Answering. arXiv:2402.08277. find
  33. Shah, R.; Feuillade-Montixi, Q.; Pour, S.; Tagade, A.; Casper, S.; and Rando, J. 2023. Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation. arXiv:2311.03348. — [arXiv:2311.03348](https://arxiv.org/abs/2311.03348) find
  34. Shen, X.; Chen, Z.; Backes, M.; Shen, Y.; and Zhang, Y. 2024. ”Do Anything Now”: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. arXiv:2308.03825.
  35. Team, L. 2024. Meta Llama Guard 2. https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL CARD.md. ↗
  36. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; Bikel, D.; Blecher, L.; Ferrer, C. C.; Chen, M.; Cucurull, G.; Esiobu, D.; Fernandes, J.; Fu, J.; Fu, W.; Fuller, B.; Gao, C.; Goswami, V.; Goyal, N.; et al. 2023. Llama 2:…
  37. Tversky, A.; and Kahneman, D. 1992. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5: 297–323. find
  38. Wan, Y.; Pu, G.; Sun, J.; Garimella, A.; Chang, K.-W.; and Peng, N. 2023. ”Kelly is a Warm Person, Joseph is a Role Model”: Gender Biases in LLM-Generated Reference Letters. arXiv:2310.09219.
  39. Wang, Y.; and Singh, L. 2023. Adding guardrails to advanced chatbots. arXiv:2306.07500. — [arXiv:2306.07500](https://arxiv.org/abs/2306.07500) find
  40. Wei, A.; Haghtalab, N.; and Steinhardt, J. 2024. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36.
  41. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Chi, E. H.; Le, Q.; and Zhou, D. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. CoRR, abs/2201.11903.
  42. Wei, Z.; Wang, Y.; Li, A.; Mo, Y.; and Wang, Y. 2023. Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations. arXiv:2310.06387.
  43. Wen, Y.; Jain, N.; Kirchenbauer, J.; Goldblum, M.; Geiping, J.; and Goldstein, T. 2023. Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery. arXiv:2302.03668. find
  44. Xu, Z.; Liu, Y.; Deng, G.; Li, Y.; and Picek, S. 2024. A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models. arXiv:2402.13457. find
  45. Yang, H.; Zhao, Y.; Wu, Y.; Wang, S.; Zheng, T.; Zhang, H.; Ma, Z.; Che, W.; and Qin, B. 2024. Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey. arXiv:2406.08068. — [arXiv:2406.08068](https://arxiv.org/abs/2406.08068) find
  46. Yi, S.; Liu, Y.; Sun, Z.; Cong, T.; He, X.; Song, J.; Xu, K.; and Li, Q. 2024. Jailbreak Attacks and Defenses Against Large Language Models: A Survey. arXiv:2407.04295.
  47. Zhan, T.; Shi, C.; Shi, Y.; Li, H.; and Lin, Y. 2024. Optimization Techniques for Sentiment Analysis Based on LLM (GPT-3). arXiv:2405.09770. — [arXiv:2405.09770](https://arxiv.org/abs/2405.09770) find
  48. Zhang, Z.; Yang, J.; Ke, P.; Mi, F.; Wang, H.; and Huang, M. 2024. Defending large language models against jailbreaking attacks through goal prioritization. In ACL.
  49. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.
  50. Zhu, S.; Zhang, R.; An, B.; Wu, G.; Barrow, J.; Wang, Z.; Huang, F.; Nenkova, A.; and Sun, T. 2023. AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models. arXiv:2310.15140. find
  51. Zou, A.; Wang, Z.; Carlini, N.; Nasr, M.; Kolter, J. Z.; and Fredrikson, M. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rad-2025-refining-input-guardrails-enhancing-llm/