Elamur Research⌕
☰
home / papers / 2026 / StealthGraph: Exposing Domain-Specific Risks in LLMs through Knowledge-Graph-Guided Harmful Prompt Generation

StealthGraph: Exposing Domain-Specific Risks in LLMs through Knowledge-Graph-Guided Harmful Prompt Generation

Paper · 2026 · Huawei Zheng, Xinqi Jiang, Sen Yang et al.

Summary

LLMs used in specialized fields such as medicine or finance need safety tests, but domain-specific harmful-prompt collections are scarce. Most are built by hand or contain blatant requests that current models refuse easily. The authors present StealthGraph, an automatic two-stage pipeline that produces covert harmful prompts grounded in domain knowledge.

First, a domain subgraph is pulled from Wikidata via SPARQL, starting from chosen root entities and pruned by Wikipedia sitelink counts. Its entities, combined with harm-category examples, guide an LLM to write explicit harmful prompts, which are then screened for toxicity and fluency. Second, these prompts are disguised through two routes: plain indirect rephrasing, and rewriting informed by "domain-context cards" that summarize neighbouring graph entities. A rewrite is kept only if it preserves meaning, stays fluent and hides its intent.

Across medicine, finance, law and education, the data is more challenging than general benchmarks. Alignment on it improves robustness to covert prompts, with general-capability loss similar to that from existing datasets. This supports more realistic red-teaming of specialized assistants.

Key points

  • The knowledge graph is built from Wikidata using four relations: P31, P279, P361 and P527. Entities are kept only if their sitelink count is at least a threshold T. Few-shot seeds come from JailbreakBench (10 harm categories, 100 exemplars).
  • Across the four domains, the obfuscation success rate (OSR) ranges from 29.03% to 42.82%, with medicine the hardest to disguise. Harmfulness, scored by Granite-Guardian-3.1-8B, stays between 95.34% and 97.85%.
  • Self-BLEU is 54.42–59.53 for all KG-guided prompts and falls to 23.24–28.08 for the successfully obfuscated subset, indicating good diversity.
  • On MMLU, the base model scores 49.75. After alignment it scores 43.41 (SG-Origin) and 42.68 (SG-Implicit), close to AdvBench (43.59) and Do-Not-Answer (43.01).
  • In the ablation, two-strategy rewriting matches direct rewriting at κ=10 but achieves higher OSR at larger budgets, especially κ=30. Code and datasets are available at github.com/ZJUIDG-AIVA/StealthGraph.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksBias, fairness & safety · 587 worksKnowledge graphs & ontologies · 211 works

Cite

Cite as Huawei Zheng, Xinqi Jiang, Sen Yang, Shouling Ji, Yingcai Wu and Dazhen Deng. 2026. StealthGraph: Exposing Domain-Specific Risks in LLMs through Knowledge-Graph-Guided Harmful Prompt Generation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.295

@inproceedings{zheng2026stealthgraph,
  title = {{StealthGraph: Exposing Domain-Specific Risks in LLMs through Knowledge-Graph-Guided Harmful Prompt Generation}},
  author = {Huawei Zheng, Xinqi Jiang, Sen Yang, Shouling Ji, Yingcai Wu and Dazhen Deng},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.295},
  url = {https://doi.org/10.18653/v1/2026.acl-long.295}
}

References

60 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishabh Bhardwaj, Duc Anh Do, and Soujanya Poria. 2024. Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14138–14149.
  2. Rishabh Bhardwaj and Soujanya Poria. 2023. Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment. Preprint, arXiv:2308.09662.
  3. Dana Brin, Vera Sorin, Eli Konen, Girish Nadkarni, Benjamin S Glicksberg, and Eyal Klang. 2024. How GPT models perform on the United States medical licensing examination: a systematic review. Discover Applied Sciences, 6(10):500. find
  4. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. 2024. JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models. In…
  5. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2023. Jailbreaking Black Box Large Language Models in Twenty Queries. Preprint, arXiv:2310.08419.
  6. DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Preprint, arXiv:2501.12948.
  7. Moussa Koulako Bala Doumbouya, Ananjan Nandi, Gabriel Poesia, Davide Ghilardi, Anna Goldie, Federico Bianchi, Dan Jurafsky, and Christopher Manning. 2025. h4rm3l: A Language for Composable Jailbreak Attack Synthesis. In International Conference on Learning Representations, pages 57253– 57286. find
  8. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2024. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. Preprint, arXiv:2404.16130. find
  9. Sebastian Gehrmann, Claire Huang, Xian Teng, Sergei Yurovski, Arjun Bhorkar, Naveen Thomas, John Doucette, David Rosenberg, Mark Dredze, and David Rabinowitz. 2025. Understanding and Mitigating Risks of Generative AI in Financial Services. In Proceedings of the 2025 ACM Conference on Fairness, Accou… find
  10. Gemini Team. 2025. Gemini: A Family of Highly Capable Multimodal Models. Preprint, arXiv:2312.11805.
  11. Gemma Team and Google DeepMind. 2024. Gemma: Open Models Based on Gemini Research and Technology. Preprint, arXiv:2403.08295. find
  12. GLM-5-Team. 2026. GLM-5: from Vibe Coding to Agentic Engineering. Preprint, arXiv:2602.15763.
  13. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security,…
  14. Xu Guo and Yiqiang Chen. 2024. Generative AI for Synthetic Data Generation: Methods, Challenges and the Future. Preprint, arXiv:2403.04190. — [arXiv:2403.04190](https://arxiv.org/abs/2403.04190) find
  15. Tessa Han, Aounon Kumar, Chirag Agarwal, and Himabindu Lakkaraju. 2024. MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models. In Advances in Neural Information Processing Systems, pages 33423–33454. find
  16. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations.
  17. Yuting Huang, Chengyuan Liu, Yifeng Feng, Yiquan Wu, Chao Wu, Fei Wu, and Kun Kuang. 2025. Rewrite to Jailbreak: Discover Learnable and Transferable Implicit Harmfulness Instruction. In Findings of the Association for Computational Linguistics: ACL 2025, pages 3669–3690. and Madian Khabsa. 2023. Lla…
  18. Jiabao Ji, Bairu Hou, Alexander Robey, George J. Pappas, Hamed Hassani, Yang Zhang, Eric Wong, and Shiyu Chang. 2025. Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th… find
  19. Xiaojun Jia, Tianyu Pang, Chao Du, Yihao Huang, Jindong Gu, Yang Liu, Xiaochun Cao, and Min Lin. 2025. Improved Techniques for Optimization-Based Jailbreaking on Large Language Models. In International Conference on Learning Representations, pages 6337–6358. find
  20. Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. 2024. WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models. In Advances in Neural Information Process…
  21. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2024. GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 382(2270):20230254. find
  22. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Proc…
  23. Jiahui Li, Yongchang Hao, Haoyu Xu, Xing Wang, and Yu Hong. 2025. Exploiting the Index Gradients for Optimization-Based Jailbreaking on Large Language Models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4535–4547. find
  24. Xiaoxia Li, Siyuan Liang, Jiyi Zhang, Han Fang, Aishan Liu, and Ee-Chien Chang. 2024a. Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMs. Preprint, arXiv:2402.14872. — [arXiv:2402.14872](https://arxiv.org/abs/2402.14872) find
  25. Zheng Hui, Yijiang River Dong, Ehsan Shareghi, and Nigel Collier. 2025. TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law. Preprint, arXiv:2507.21134. find
  26. Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. 2024b. DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLMs Jailbreakers. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13891–13913. find
  27. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. 2023. ToxicChat: Unveiling Hidden Challenges of Toxicity Detection…
  28. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. In International Conference on Learning Representations, pages 56174–56194.
  29. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-EVAL: NLG Evaluation using GPT-4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522.
  30. Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. 2024. JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks. In First Conference on Language Modeling.
  31. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. In Proceedings of the 41st International Con…
  32. Jeremy McHugh, Kristina Šekrst, and Jon Cefalu. 2025. Prompt Injection 2.0: Hybrid AI Threats. Preprint, arXiv:2507.13169. — [arXiv:2507.13169](https://arxiv.org/abs/2507.13169) find
  33. Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2024. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. In Advances in Neural Information Processing Systems, pages 61065–61105.
  34. Meta. 2024. The Llama 3 Herd of Models. Preprint, arXiv:2407.21783.
  35. Junjie Mu, Zonghao Ying, Zhekui Fan, Zonglei Jing, Yaoyuan Zhang, Zhengmin Yu, Wenxin Zhang, Quanchen Zou, and Xiangzheng Zhang. 2025. Mask-GCG: Are All Tokens in Adversarial Suffixes Necessary for Jailbreak Attacks? Preprint, arXiv:2509.06350. — [arXiv:2509.06350](https://arxiv.org/abs/2509.06350) find
  36. OpenAI. 2023. GPT-4 Technical Report. Preprint, arXiv:2303.08774.
  37. OpenAI. 2024. GPT-4o System Card. arXiv:2410.21276.
  38. Preprint, Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchia, Subhajit Chaudhury, Tejaswini Pedapati, Pierre Dognin, Keerthiram Murugesan, Erik Miehling, Martín Santillán Cooper, Kieran Fraser, Giulio Zizzo, Muhammad Zaid Hameed, Mark Purcell, Michael Desmond, Qian Pan, Inge Vejsbjerg, Elizabeth… find
  39. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2025. Safety Alignment Should be Made More Than Just a Few Tokens Deep. In International Conference on Learning Representations, pages 54911–54941.
  40. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! In International Conference on Learning Representations, pages 30988–31043.
  41. Qwen Team. 2025. Qwen3 Technical Report. Preprint, arXiv:2505.09388.
  42. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  43. Shagoto Rahman and Ian Harris. 2025. Summary the Savior: Harmful Keyword and Query-based Summarization for LLM Jailbreak Defense. In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), pages 266–275. find
  44. Sippo Rossi, Alisia Marianne Michel, Raghava Rao Mukkamala, and Jason Bennett Thatcher. 2024. An Early Categorization of Prompt Injection Attacks on Large Language Models. Preprint, arXiv:2402.00898. — [arXiv:2402.00898](https://arxiv.org/abs/2402.00898) find
  45. Paul Röttger, Fabio Pernisi, Bertie Vidgen, and Dirk Hovy. 2025. SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 27617– 27627.
  46. Yonadav Shavit, Sandhini Agarwal, Miles Brundage, Steven Adler, Cullen O’Keefe, Rosie Campbell, Teddy Lee, Pamela Mishkin, Tyna Eloundou, Alan Hickey, Katarina Slama, Lama Ahmad, Paul McMillan, Alex Beutel, Alexandre Passos, and David G. Robinson. 2023. Practices for Governing Agentic AI Systems. Re… find
  47. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. “Do Anything Now”: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, page 1671–1685.
  48. Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. 2024. A StrongREJECT for Empty Jailbreaks. In Advances in Neural Information Processing Systems, pages 125416–125440. find
  49. Yuting Tan, Xuying Li, Zhuo Li, Huizhen Shu, and Peikang Hu. 2025. The Resurgence of GCG Adversarial Attacks on Large Language Models. Preprint, arXiv:2509.00391. — [arXiv:2509.00391](https://arxiv.org/abs/2509.00391) find
  50. Yihong Tang, Bo Wang, Xu Wang, Dongming Zhao, Jing Liu, Ruifang He, and Yuexian Hou. 2025. Role-Break: Character Hallucination as a Jailbreak Attack in Role-Playing Systems. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7386–7402. find
  51. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A Benchmarking Platform for Text Generation Models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, page 1097–1100.
  52. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. Preprint, arXiv:2307.15043.
  53. Shangqing Tu, Zhuoran Pan, Wenxuan Wang, Zhexin Zhang, Yuliang Sun, Jifan Yu, Hongning Wang, Lei Hou, and Juanzi Li. 2025. Knowledge-to-Jailbreak: Investigating Knowledge-driven Jailbreaking Attacks for Large Language Models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery an… find
  54. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. In Advances in Neural Information Processing Systems, pages 5776– 5788. find
  55. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2024. Do-Not-Answer: Evaluating Safeguards in LLMs. In Findings of the Association for Computational Linguistics: EACL 2024, pages 896–911.
  56. Jim Webber. 2012. A programmatic introduction to Neo4j. In Proceedings of the 3rd Annual Conference on Systems, Programming, and Applications: Software for Humanity, page 217–218. find
  57. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How Does LLM Safety Training Fail? In Advances in Neural Information Processing Systems, pages 80079–80110.
  58. Zixuan Weng, Xiaolong Jin, Jinyuan Jia, and Xiangyu Zhang. 2025. Foot-In-The-Door: A Multi-turn Jailbreak for LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1939–1950. find
  59. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, pages 46…
  60. Muyang Zheng, Yuanzhi Yao, Changting Lin, Cai-hong Kai, Yanxiang Chen, and Zhiquan Liu. 2025. MIST: Jailbreaking Black-box Large Language Models via Iterative Semantic Tuning. Preprint, arXiv:2506.16792. — [arXiv:2506.16792](https://arxiv.org/abs/2506.16792) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2026-stealthgraph-exposing-domain-specific-risks/