Elamur Research⌕
☰
home / papers / 2026 / False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize

False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize

Paper · 2026 · Cheng Wang, Zeming Wei, Qin Liu et al.

Summary

The paper asks whether lightweight probes trained on LLM hidden states actually detect harmful intent, or only appear to. Earlier work proposed such probes as safety filters. The authors test this with an SVM trained on the last-layer, last-token representation of Gemma-3-it, Llama-3.1-Instruct and Qwen2.5-Instruct models. They train on benign Alpaca paired with BeaverTailsEval or ForbiddenQuestions, then test on unseen sets: Dolly, HarmBench and AdvBench.

In-domain accuracy is close to perfect, but it collapses under distribution shift. Three studies probe why:

  • n-gram Naive Bayes baselines reach comparable scores;
  • probes misclassify "cleaned" prompts, where harmful content is swapped for benign content but the structure is kept;
  • probes are thrown off by benign text containing alarming vocabulary.

The authors conclude that probes latch onto instructional phrasing and trigger words rather than harmfulness itself. Yet the same LLMs, asked directly in zero-shot mode, classify safety almost perfectly. This suggests the weakness lies in the probing method, not in the models' understanding. The findings warn that probe-based detectors give a false sense of security and call for better evaluation protocols.

Key points

  • In-distribution probe accuracy exceeds 98% for every model and dataset pair. On OOD data it falls by 15–99 points, and some settings drop to near zero (e.g., Gemma-3-27b-it trained with ForbiddenQuestions scores 0.0% on HarmBench).
  • Models range from 4B to 72B parameters (Gemma-3, Llama-3.1, Qwen2.5). Base and instruction-tuned versions generalize about equally poorly.
  • Replacing the SVM with Logistic Regression or an MLP gives the same picture: 99.0% in-domain, but only about 23–30% on semantically cleaned data.
  • Zero-shot prompting of the LLMs reaches roughly 96.5–100% accuracy on Alpaca, Dolly, AdvBench and HarmBench.
  • In PCA plots of hidden states, malicious and cleaned prompts cluster together, while OOD data forms separate clusters. Code is released on GitHub (WangCheng0116/Why-Probe-Fails).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States162 works · 35,836 citations🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓National University of SingaporeSingapore · Singapore180 works · 34,304 citations🎓University of California, DavisDavis · United States28 works · 41,916 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Bias, fairness & safety · 517 worksInterpretability · 130 works

Cite

Cite as Cheng Wang, Zeming Wei, Qin Liu, Wenxuan Zhou and Muhao Chen. 2026. False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1300

@inproceedings{wang2026false,
  title = {{False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize}},
  author = {Cheng Wang, Zeming Wei, Qin Liu, Wenxuan Zhou and Muhao Chen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1300},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1300}
}

References

62 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Deema Alnuhait, Neeraja Kirtane, Muhammad Khalifa, and Hao Peng. 2024. Factcheckmate: Preemptively detecting and mitigating hallucinations in lms. arXiv preprint arXiv:2410.02899. find
  2. Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. 2024. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932. find
  3. Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when it’s lying.
  4. Tyler A Chang, Zhuowen Tu, and Benjamin K Bergen. 2022. The geometry of multilingual language model representations. arXiv preprint arXiv:2205.10964.
  5. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models.
  6. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. find
  7. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning, 20(3):273–297.
  8. Weilong Dong, Peiguang Li, Yu Tian, Xinyi Zeng, Fengdi Li, and Sirui Wang. 2025. Feature-aware malicious output detection and mitigation. arXiv preprint arXiv:2504.09191. — [arXiv:2504.09191](https://arxiv.org/abs/2504.09191) find
  9. Shaona Ghosh, Amrita Bhattacharjee, Yftah Ziser, and Christopher Parisien. 2025a. Safesteer: Interpretable safety steering with refusal-evasion in llms. arXiv preprint arXiv:2506.04250. — [arXiv:2506.04250](https://arxiv.org/abs/2506.04250) find
  10. Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, and Christopher Parisien. 2025b. Aegis2. 0: A diverse ai safety dataset and risks taxonomy for alignment of llm guardrails. arXiv preprint arXiv:2501.09004. — AEGIS2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails
  11. Weiyang Guo, Zesheng Shi, Liye Zhao, Jiayuan Ma, Zeen Zhu, Junxian He, Min Zhang, and Jing Li. 2026a. E3-tir: Enhanced experience exploitation for tool-integrated reasoning. find
  12. Weiyang Guo, Zesheng Shi, Zeen Zhu, Yuan Zhou, Min Zhang, and Jing Li. 2026b. Backdoors in rlvr: Jailbreak backdoors in llms from verifiable reward.
  13. Rima Hazra, Sayan Layek, Somnath Banerjee, and Soujanya Poria. 2024. Safety arithmetic: A framework for test-time safety alignment of language models by steering parameters and activations. find
  14. Xiangdong Hu, Yangyang Jiang, Qin Hu, and Xiaojun Jia. 2026. Gambit: A gamified jailbreak framework for multimodal large language models.
  15. Peichun Hua, Hao Li, Shanghao Shi, Zhiyuan Yu, and Ning Zhang. 2025. Rethinking jailbreak detection of large vision language models with representational contrastive scoring.
  16. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. 2023. Catastrophic jailbreak of open-source llms via exploiting generation. arXiv preprint arXiv:2310.06987.
  17. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and…
  18. Haibo Jin, Leyang Hu, Xinuo Li, Peiyan Zhang, Chonghan Chen, Jun Zhuang, and Haohan Wang. 2024. Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models. find
  19. Tianjie Ju, Weiwei Sun, Wei Du, Xinwei Yuan, Zhaochun Ren, and Gongshen Liu. 2024. How large language models encode context knowledge? a layer-wise probing study. arXiv preprint arXiv:2402.16061. find
  20. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural…
  21. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. find
  22. Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto Yamada, Hui Liu, and Jiliang Tang. 2024. Towards understanding jailbreak attacks in llms: A representation space analysis. arXiv preprint arXiv:2406.10794. find
  23. Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024a. Automatic and universal prompt injection attacks against large language models. find
  24. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2024b. Prompt injection attack against llm-integrated applications.
  25. Yue Liu, Shengfang Zhai, Mingzhe Du, Yulin Chen, Tri Cao, Hongcheng Gao, Cheng Wang, Xinfeng Li, Kun Wang, Junfeng Fang, Jiaheng Zhang, and Bryan Hooi. 2025a. Guardreasoner-vl: Safeguarding vlms via reinforced reasoning. find
  26. Yule Liu, Zhen Sun, Xinlei He, and Xinyi Huang. 2024c. Quantized delta weight is safety keeper. arXiv preprint arXiv:2411.19530. — [arXiv:2411.19530](https://arxiv.org/abs/2411.19530) find
  27. Yule Liu, Heyi Zhang, Jinyi Zheng, Zhen Sun, Zifan Peng, Tianshuo Cong, Yilong Yang, Xinlei He, and Zhuo Ma. 2025b. Grpo privacy is at risk: A membership inference attack against reinforcement learning with verifiable rewards. arXiv preprint arXiv:2511.14045. — [arXiv:2511.14045](https://arxiv.org/abs/2511.14045) find
  28. Zhenhua Liu, Tong Zhu, Chuanyuan Tan, Haonan Lu, Bing Liu, and Wenliang Chen. 2024d. Probing language models for pre-training data detection. arXiv preprint arXiv:2406.01333. find
  29. Zixuan Liu, Xiaolin Sun, and Zizhan Zheng. 2024e. Enhancing llm safety via constrained direct preference optimization. arXiv preprint arXiv:2403.02475. — [arXiv:2403.02475](https://arxiv.org/abs/2403.02475) find
  30. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.
  31. Meta. 2024. The llama 3 herd of models.
  32. OpenAI. 2024. Gpt-4o system card.
  33. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  34. Cheng Qian, Hainan Zhang, Lei Sha, and Zhiming Zheng. 2025. Hsf: Defending against jailbreak attacks with hidden state filtering. In Companion Proceedings of the ACM on Web Conference 2025, pages 2078–2087. find
  35. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… find
  36. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model.
  37. Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2023. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263.
  38. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. “Do Anything Now”: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM.
  39. Dan Shi, Tianhao Shen, Yufei Huang, Zhigen Li, Yongqi Leng, Renren Jin, Chuang Liu, Xinwei Wu, Zishan Guo, Linhao Yu, Ling Shi, Bojian Jiang, and Deyi Xiong. 2024a. Large language model safety: A holistic survey. find
  40. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024b. Detecting pretraining data from large language models. find
  41. Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. 2025. Layer by layer: Uncovering hidden representations in language models. arXiv preprint arXiv:2502.02013. find
  42. Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. 2024. A strongreject for empty jailbreaks. find
  43. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention is all you need. find
  45. Cheng Wang, Yue Liu, Baolong Bi, Duzhen Zhang, Zhong-Zhi Li, Yingwei Ma, Yufei He, Shengju Yu, Xinfeng Li, Junfeng Fang, et al. 2025a. Safety in large reasoning models: A survey. arXiv preprint arXiv:2504.17704. find
  46. Cheng Wang, Yiwei Wang, Yujun Cai, and Bryan Hooi. 2025b. Tricking retrievers with influential tokens: An efficient black-box corpus poisoning attack. find
  47. Cheng Wang, Yiwei Wang, Bryan Hooi, Yujun Cai, Nanyun Peng, and Kai-Wei Chang. 2025c. Con-recall: Detecting pre-training data in llms via contrastive decoding. find
  48. Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, Liang Lin, Zhihao Xu, Haolang Lu, Xinye Cao, Xinyun Zhou, Weifei Jin, Fanci Meng, Shicheng Xu, Junyuan Mao, Yu Wang, Hao Wu, Minghe Wang, Fan Zhang, Junfeng Fang, Wenjie Qu, Yue Liu… find
  49. Xi Wang, Songlei Jian, Shasha Li, Xiaopeng Li, Zhaoye Li, Bin Ji, Baosheng Wang, and Jie Yu. 2026. Jpu: Bridging jailbreak defense and unlearning via on-policy path rectification. arXiv preprint arXiv:2601.03005. find
  50. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368.
  51. Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. 2023. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387.
  52. Zeming Wei, Chengcan Wu, and Meng Sun. 2025. Rega: Representation-guided abstraction for model-based safeguarding of llms. arXiv preprint arXiv:2506.01770. — [arXiv:2506.01770](https://arxiv.org/abs/2506.01770) find
  53. Xiqiao Xiong, Ouxiang Li, Zhuo Liu, Moxin Li, Wentao Shi, Fengbin Zhu, Qifan Wang, and Fuli Feng. 2026. Trojail: Trajectory-level optimization for multi-turn large language model jailbreaks with process rewards.
  54. Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. 2024. Jailbreak attacks and defenses against large language models: A survey.
  55. Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, et al. 2024. Shieldgemma: Generative ai content moderation based on gemma. arXiv preprint arXiv:2407.21772. — [arXiv:2407.21772](https://arxiv.org/abs/2407.21772) find
  56. Yihao Zhang, Zeming Wei, Jun Sun, and Meng Sun. 2024. Adversarial representation engineering: A general model editing framework for large language models. Advances in Neural Information Processing Systems, 37:126243–126264. find
  57. Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. 2024. On prompt-driven safeguarding for large language models. arXiv preprint arXiv:2401.18018. find
  58. Zexuan Zhong, Ziqing Huang, Alexander Wettig, and Danqi Chen. 2023. Poisoning retrieval corpora by injecting adversarial passages. find
  59. Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. 2024. How alignment and jailbreak work: Explain llm safety through intermediate hidden states. arXiv preprint arXiv:2406.05644. find
  60. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models.
  61. Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. 2024. Poisonedrag: Knowledge corruption attacks to retrieval-augmented generation of large language models. find
  62. Dataset Name AdvBench ForbiddenQuestions BeaverTailsEval JailbreakBench StrongReject MaliciousInstruct HarmBench Dataset Name Alpaca Dolly SimpleQA NaturalQuestions XSTest Malicious Dataset HuggingFace Path walledai/AdvBench walledai/ForbiddenQuestions walledai/BeaverTailsEval walledai/JailbreakBenc… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-false-sense-security-why-probing/