Elamur Research⌕
☰
home / papers / 2026 / Detection of Adversarial Prompts with Model Predictive Entropy

Detection of Adversarial Prompts with Model Predictive Entropy

Paper · 2026 · Franziska Rubenbauer, Sebastian Steindl, Patrick Levi et al.

Summary

Safety-aligned LLMs can still be jailbroken. Many existing defenses depend on knowing the attack type in advance, on retraining, or on a separate guard model. The authors present SENTRY (Semantic ENtropy-based attack Recognition sYstem), an attack-agnostic detector that looks at how uncertain the model is when it starts answering. They adapt TokenSAR, a weighted token-entropy score from Duan et al. that was first built for hallucination detection in Q&A. TokenSAR scales each token's entropy by its semantic relevance and normalizes the result, so that answers of different lengths can be compared.

Only the first ten generated tokens are needed. A prompt is flagged when its score passes a threshold. The approach was tested on Llama, Vicuna, Falcon, DeepSeek and Mistral, using jailbreaks built with the h4rm3l framework. Successful adversarial prompts produced clearly higher TokenSAR values than both benign and plainly malicious prompts. Detection quality was comparable to Llama Guard 3 1B, at lower cost and with no extra training. This makes SENTRY a cheap supplementary safeguard for models that are already aligned.

Key points

  • Three prompt categories: benign, malicious and adversarial. Jailbreak success was decided by keyword search on outputs capped at 10 new tokens. Detection was scored with F1 for the adversarial and basic classes.
  • Threshold setting: thresholds come from interquartile means over 50 prompts, and the appendix reports that these generalize. Token importance relies on semantic similarity (spaCy embeddings).
  • Baseline: SENTRY was compared with Llama Guard 3 1B. The authors claim advantages in compute, language flexibility and scalability, with no retraining required when the base model is updated.
  • Limitations:
    • It works only for safety-aligned models; Vicuna does not show the entropy gap.
    • Falcon3 7B Instruct had too few successful attacks to evaluate the threshold reliably.
    • Only smaller open-source models were tested.
    • Logits are required, although the OpenAI API exposes logprobs.
    • It assumes attackers do not deliberately minimize entropy.
  • Code: available on GitHub at FranziskaRubenbauer/SENTRY.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Ostbayerische Technische Hochschule Amberg-Weiden.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 587 works

Cite

Cite as Franziska Rubenbauer, Sebastian Steindl, Patrick Levi, Daniel Loebenberger and Ulrich Schäfer. 2026. Detection of Adversarial Prompts with Model Predictive Entropy. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.103

@inproceedings{rubenbauer2026detection,
  title = {{Detection of Adversarial Prompts with Model Predictive Entropy}},
  author = {Franziska Rubenbauer, Sebastian Steindl, Patrick Levi, Daniel Loebenberger and Ulrich Schäfer},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.103},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.103}
}

References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI@Meta. 2024. Llama 3 model card. find
  2. Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. 2023. Uncertainty in Natural Language Generation: From Theory to Applications. arXiv preprint. and Madian Khabsa. 2023. Llama Guard: LLM-based Input-Ou…
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, and 12 others. 202… find
  4. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022.… find
  5. Sarah Ball, Frauke Kreuter, and Nina Panickssery. 2024. Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models. arXiv preprint.
  6. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2023. Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. arXiv preprint. find
  7. Moussa Koulako Bala Doumbouya, Ananjan Nandi, Gabriel Poesia, Davide Ghilardi, Anna Goldie, Federico Bianchi, Dan Jurafsky, and Christopher D. Manning. 2024. h4rm3l: A Dynamic Benchmark of Composable Jailbreak Attacks for LLM Safety Assessment. arXiv preprint. find
  8. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. arXiv preprint.
  9. Hanyu Duan, Yi Yang, and Kar Yan Tam. 2024a. Do LLMs Know about Hallucination? An Empirical Investigation of LLM’s Hidden States. arXiv preprint. find
  10. Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024b. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Co… find
  11. Sebastian Glorin. 2023. Do ChatGPT and Other AI Chatbots Pose a Cybersecurity Risk?: An Exploratory Study. International Journal of Security and Privacy in Pervasive Computing (IJSPPC), 15(1):1–11. Publisher: IGI Global Scientific Publishing. find
  12. Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. 2024. COLD-attack: jailbreaking LLMs with stealthiness and controllability. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of ICML’24, pages 16974– 17002, Vienna, Austria. JMLR.org. find
  13. Maanak Gupta, CharanKumar Akiri, Kshitiz Aryal, Eli Parker, and Lopamudra Praharaj. 2023. From Chat-GPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy. IEEE Access, PP:1–1. find
  14. HuggingFaceH4. 2024. HuggingFaceH4/cai-conversation-harmless · Datasets at Hugging Face. find
  15. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. 2024. LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B. arXiv preprint. find
  16. AI @ Meta Llama Team. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  17. Andrey Malinin and Mark Gales. 2021. Uncertainty Estimation in Autoregressive StructuredPrediction. arXiv preprint. find
  18. OpenAI. 2025. API Reference - OpenAI API. find
  19. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022a. Trai… — Training language models to follow instructions with human feedback
  20. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022b. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  21. Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational L…
  22. Sebastian Steindl, Ulrich Schäfer, Bernd Ludwig, and Patrick Levi. 2024. Linguistic obfuscation attacks and large language model uncertainty. In Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024), pages 35–40, St Julians, Malta. Association for Computational Linguistics. find
  23. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  24. Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. 2024. Jailbreak Attacks and Defenses Against Large Language Models: A Survey. arXiv preprint.
  25. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.
  26. Yukai Zhou, Jian Lou, Zhijie Huang, Zhan Qin, Sibei Yang, and Wenjie Wang. 2025. Don’t say no: Jailbreaking LLM by suppressing refusal. In Findings of the Association for Computational Linguistics: ACL 2025, pages 25224–25249, Vienna, Austria. Association for Computational Linguistics. find
  27. Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2023. AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models. arXiv preprint.
  28. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv preprint.
  29. Xiaotian Zou, Yongkang Chen, and Ke Li. 2024. Is the System Message Really Important to Jailbreaks in Large Language Models? arXiv preprint. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rubenbauer-2026-detection-adversarial-prompts-model-predictive/