Elamur Research⌕
☰
home / papers / 2024 / Security and Privacy Challenges of Large Language Models: A Survey

Security and Privacy Challenges of Large Language Models: A Survey

Summary

Large language models now power text generation, translation, question answering and code writing. Their scale and reliance on loosely curated web data, however, expose them to many attacks. This survey organizes the security and privacy threats facing LLMs and the countermeasures proposed so far. The authors start with a short account of how LLMs are built, trained and fine-tuned, and identify which pipeline components create weak points. They then group threats into security attacks, such as jailbreaking, prompt injection, backdoors and data poisoning, and privacy attacks, such as PII leakage, membership inference and gradient leakage. They also note where the two categories overlap. For each class, the paper reviews example attacks, their impact and possible defenses. It also covers risks that arise in specific application sectors. The authors conclude that LLMs remain highly vulnerable and that few defenses are designed for them specifically. Methods built for smaller LMs might carry over, but they need careful evaluation and adaptation. This matters for any team that deploys LLMs where personal data is handled or legal compliance is required.

Key points

  • The paper is a 51-page survey (arXiv:2402.00888, prepared for J. ACM). Table 2 compares it with roughly twenty earlier surveys and empirical studies, including work on SmoothLLM, PII leakage and adversarial ML taxonomies.
  • Covered attacks include jailbreaking (e.g., DAN, PAIR), prompt injection, backdoors (e.g., CBA, BGMAttack), data poisoning, PII leakage, MIA and gradient attacks (TAG).
  • The authors connect unintended data leakage to privacy laws: HIPAA, GDPR and CCPA.
  • Application-specific risks are discussed for misinformation and hallucination, cybercrime, transportation, healthcare, education, governance and science.
  • Open problems include explainable AI (XAI) as a double-edged tool. Methods like LIME and SHAP, or XAI-aware model extraction (XaMEA), can make it easier to attack models. Open-source LLMs may be especially exposed, and this has been studied very little for LLMs compared with DNNs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

65 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 62 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.00888 · arXiv 2402.00888

@misc{das2024security,
  title = {{Security and Privacy Challenges of Large Language Models: A Survey}},
  author = {Das, Badhan Chandra and Amini, M. Hadi and Wu, Yanzhao},
  year = {2024},
  doi = {10.48550/arxiv.2402.00888},
  eprint = {2402.00888},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.00888}
}

References

334 references, 58 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 30dascoding. 2024. Unraveling the Mysteries of Anomaly Detection: The Power of Explainable AI. Available Online: https://30dayscoding.com/blog/explainable-ai-in-anomaly-detection [Accessed on October 31, 2024]. ↗
  2. Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. 308–318. find
  3. David Leslie & Sandra Wachter Abeba Birhane, Atoosa Kasirzadeh. 2023. Science in the age of large language models. doi.org/10.1038/s42254-023-00581-4 (2023). — [doi:10.1038/s42254-023-00581-4](https://doi.org/10.1038/s42254-023-00581-4) find
  4. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023). find our card
  5. Hojjat Aghakhani, Wei Dai, Andre Manoel, Xavier Fernandes, Anant Kharkar, Christopher Kruegel, Giovanni Vigna, David Evans, Ben Zorn, and Robert Sim. 2023. TrojanPuzzle: Covertly Poisoning Code-Suggestion Models. arXiv preprint arXiv:2301.02344 (2023). — [arXiv:2301.02344](https://arxiv.org/abs/2301.02344) find
  6. Protect AI. 2024. LLM Guard - The Security Toolkit for LLM Interactions. Available Online: https://llm-guard.com/ [Accessed on October 11, 2024]. ↗
  7. Alex Albert. 2023. Jailbreak Chat. Available Online: https://www.jailbreakchat.com [Accessed on January 28, 2024]. ↗
  8. Tarek Ali. 2024. Next-generation intrusion detection systems with LLMs: real-time anomaly detection, explainable AI, and adaptive data generation. Master’s thesis. T. Ali. find
  9. Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132 (2023). find our card
  10. Thimira Amaratunga. 2023. Threats, Opportunities, and Misconception s. In Understanding Large Language Models: Learning Their Underlying Concepts and Technologies. Springer, 131–148. find
  11. Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. 2021. Large-scale differentially private BERT. arXiv preprint arXiv:2108.01624 (2021). find
  12. Yoshinori Aono, Takuya Hayashi, Lihua Wang, Shiho Moriai, et al. 2017. Privacy-preserving deep learning via additively homomorphic encryption. IEEE transactions on information forensics and security 13, 5 (2017), 1333–1345. find
  13. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022). find
  14. Mislav Balunovic, Dimitar Dimitrov, Nikola Jovanović, and Martin Vechev. 2022. LAMP: Extracting text from gradients with language model priors. Advances in Neural Information Processing Systems 35 (2022), 7641–7654. find
  15. Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, and Jaehoon Amir Safavi. 2017. Mitigating poisoning attacks on machine learning models: A data provenance based approach. In Proceedings of the 10th ACM workshop on artificial intelligence and security. 103–110. find
  16. Marcel Binz, Stephan Alaniz, Adina Roskies, Balazs Aczel, Carl T Bergstrom, Colin Allen, Daniel Schad, Dirk Wulff, Jevin D West, Qiong Zhang, et al. 2023. How should the advent of large language models affect the practice of science? arXiv preprint arXiv:2312.03759 (2023). — [arXiv:2312.03759](https://arxiv.org/abs/2312.03759) find
  17. ROB BONTA. 2023. California Consumer Privacy Act (CCPA). Available Online: https://oag.ca.gov/privacy/ccpa. [Accessed on January 28, 2024]. ↗
  18. Samuel R Bowman. 2023. Eight things to know about large language models. arXiv preprint arXiv:2304.00612 (2023). — [arXiv:2304.00612](https://arxiv.org/abs/2304.00612) find
  19. Hezekiah J Branch, Jonathan Rodriguez Cefalu, Jeremy McHugh, Leyla Hujer, Aditya Bahl, Daniel del Castillo Iglesias, Ron Heichman, and Ramesh Darwishi. 2022. Evaluating the susceptibility of pre-trained language models via handcrafted adversarial examples. arXiv preprint arXiv:2209.02128 (2022). — [arXiv:2209.02128](https://arxiv.org/abs/2209.02128) find
  20. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. find
  21. Xiangrui Cai, Haidong Xu, Sihan Xu, Ying Zhang, et al. 2022. BadPrompt: Backdoor attacks on continuous prompts. Advances in Neural Information Processing Systems 35 (2022), 37068–37080. find
  22. Erik Cambria, Lorenzo Malandri, Fabio Mercorio, Navid Nobani, and Andrea Seveso. 2024. XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Models. arXiv preprint arXiv:2407.15248 (2024). — [arXiv:2407.15248](https://arxiv.org/abs/2407.15248) find
  23. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21). 2633–2650. find our card
  24. Computer Security Resource Center. 2023. Information Systems Security (INFOSEC). Available Online: https://csrc.nist.gov/glossary/term/information_systems_security. [Accessed on January 28, 2024]. ↗
  25. Edward Y Chang. 2023. SocraSynth: Multi-LLM Reasoning with Conditional Statistics. Available Online: https://www.researchgate.net/publication/373753725_SocraSynth_Multi-LLM_Reasoning_with_Conditional_Statistics [Accessed on February 05, 2024]. ↗
  26. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419 (2023). find our card
  27. Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. 2018. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv:1811.03728 (2018). — [arXiv:1811.03728](https://arxiv.org/abs/1811.03728) find
  28. Canyu Chen and Kai Shu. 2023. Can llm-generated misinformation be detected? arXiv preprint arXiv:2309.13788 (2023). find
  29. Canyu Chen and Kai Shu. 2023. Combating misinformation in the age of llms: Opportunities and challenges. AI Magazine (2023). find
  30. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or llms as the judge? a study on judgement biases. arXiv preprint arXiv:2402.10669 (2024). find
  31. Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. 2019. DeepInspect: A Black-box Trojan Detection and Mitigation Framework for Deep Neural Networks.. In IJCAI, Vol. 2. 8. find
  32. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). find our card
  33. Xiaoyi Chen, Siyuan Tang, Rui Zhu, Shijun Yan, Lei Jin, Zihao Wang, Liya Su, XiaoFeng Wang, and Haixu Tang. 2023. The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Risks. arXiv preprint arXiv:2310.15469 (2023). — [arXiv:2310.15469](https://arxiv.org/abs/2310.15469) find
  34. Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. 2024. AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases. arXiv preprint arXiv:2407.12784 (2024). — [arXiv:2407.12784](https://arxiv.org/abs/2407.12784) find
  35. Pengzhou Cheng, Zongru Wu, Wei Du, Haodong Zhao, Wei Lu, and Gongshen Liu. 2023. Backdoor attacks and countermeasures in natural language processing models: A comprehensive security review. arXiv preprint arXiv:2309.06055 (2023). — [arXiv:2309.06055](https://arxiv.org/abs/2309.06055) find
  36. Vivying SY Cheng et al. 2006. Health insurance portability and accountability act (HIPPA) compliant access control model for web services. International Journal of Healthcare Information Systems and Informatics (IJHISI) 1, 1 (2006), 22–39. find
  37. I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023. FacTool: Factuality Detection in Generative AI–A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios. arXiv preprint arXiv:2307.13528 (2023). find our card
  38. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing GPT-4 with 90%* chatGPT quality. See https://vicuna.lmsys. org (accessed 14 April 2023) (2023). ↗
  39. Edward Chou, Florian Tramèr, Giancarlo Pellegrino, and Dan Boneh. 2018. SentiNet: Detecting physical attacks against deep learning systems.(2018). (2018). find
  40. Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang. 2024. Comprehensive assessment of jailbreak attacks against llms. arXiv preprint arXiv:2402.05668 (2024). — [arXiv:2402.05668](https://arxiv.org/abs/2402.05668) find
  41. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research 25, 70 (2024), 1–53. find
  42. Benjamin Clavié, Alexandru Ciceu, Frederick Naylor, Guillaume Soulié, and Thomas Brightwell. 2023. Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification. In International Conference on Applications of Natural Language to Information Systems. Springer,… find
  43. CLOUDFLARE. 2023. What is data privacy? Available Online: https://www.cloudflare.com/learning/privacy/what-is-data-privacy/. [Accessed on January 28, 2024]. ↗
  44. Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. 2019. Certified adversarial robustness via randomized smoothing. In international conference on machine learning. PMLR, 1310–1320. find
  45. Andrea Continella, Yanick Fratantonio, Martina Lindorfer, Alessandro Puccetti, Ali Zand, Christopher Kruegel, Giovanni Vigna, et al. 2017. Obfuscation-Resilient Privacy Leak Detection for Mobile Apps Through Differential Analysis.. In NDSS, Vol. 17. 10–14722. find
  46. The Conversation. 2022. The Galactica AI model was trained on scientific knowledge – but it spat out alarmingly plausible nonsense. Available Online: https://theconversation.com/the-galactica-ai-model-was-trained-on-scientific-knowledge-but-it-spat-out-alarmingly-plausible-nonsense-195445 [Accessed… ↗
  47. Ronald Cramer, Ivan Bjerre Damgård, et al. 2015. Secure multiparty computation. Cambridge University Press. find
  48. Evan Crothers, Nathalie Japkowicz, and Herna L Viktor. 2023. Machine-generated text: A comprehensive survey of threat models and detection methods. IEEE Access (2023). find
  49. Ganqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen, Zhiyuan Liu, and Maosong Sun. 2022. A unified evaluation of textual backdoor learning: Frameworks and benchmarks. Advances in Neural Information Processing Systems 35 (2022), 5009–5023. find
  50. Sunhao Dai, Chen Xu, Shicheng Xu, Liang Pang, Zhenhua Dong, and Jun Xu. 2024. Bias and unfairness in information retrieval systems: New challenges in the llm era. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 6437–6447. find
  51. Dimitrios Damopoulos, Georgios Kambourakis, Stefanos Gritzalis, and Sang Oh Park. 2014. Exposing mobile malware from the inside (or what is your mobile app really doing?). Peer-to-Peer Networking and Applications 7 (2014), 687–697. find
  52. Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. 2023. Privacy risks analysis and mitigation in federated learning for medical images. In 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 1870–1873. find
  53. Luigi De Angelis, Francesco Baglivo, Guglielmo Arzilli, Gaetano Pierpaolo Privitera, Paolo Ferragina, Alberto Eugenio Tozzi, and Caterina Rizzo. 2023. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Frontiers in Public Health 11 (2023), 1166120. find
  54. Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023. MasterKey: Automated jailbreak across multiple large language model chatbots. arXiv preprint arXiv:2307.08715 (2023). — [arXiv:2307.08715](https://arxiv.org/abs/2307.08715) find
  55. Jieren Deng, Yijue Wang, Ji Li, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, and Caiwen Ding. 2021. TAG: Gradient attack on transformer-based language models. arXiv preprint arXiv:2103.06819 (2021). find
  56. develop.sentry.dev. 2023. PII and Data Scrubbing. Available Online: https://develop.sentry.dev/pii [Accessed on January 28, 2024]. ↗
  57. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). find
  58. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495 (2023). find our card
  59. Ben Dickson. 2021. Machine learning: What are membership inference attacks? Available Online: https://bdtechtalks.com/2021/04/23/machine-learning-membership-inference-attacks/ [Accessed on January 28, 2024]. ↗
  60. Wei Du, Yichun Zhao, Boqun Li, Gongshen Liu, and Shilin Wang. 2022. PPT: Backdoor attacks on pre-trained models via poisoned prompt tuning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22. 680–686. find
  61. Cynthia Dwork, Aaron Roth, et al. 2014. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science 9, 3–4 (2014), 211–407. find
  62. Zekeriya Erkin, Martin Franz, Jorge Guajardo, Stefan Katzenbeisser, Inald Lagendijk, and Tomas Toft. 2009. Privacy-preserving face recognition. In Privacy Enhancing Technologies: 9th International Symposium, PETS 2009, Seattle, WA, USA, August 5-7, 2009. Proceedings 9. Springer, 235–253. find
  63. Fatima Ezzeddine. 2024. Privacy Implications of Explainable AI in Data-Driven Systems. arXiv preprint arXiv:2406.15789 (2024). — [arXiv:2406.15789](https://arxiv.org/abs/2406.15789) find
  64. Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 6491–6501. find our card
  65. Qizhang Feng, Siva Rajesh Kasa, Hyokun Yun, Choon Hui Teo, and Sravan Babu Bodapati. 2024. Exposing privacy gaps: Membership inference attack on preference data for LLM alignment. arXiv preprint arXiv:2407.06443 (2024). — [arXiv:2407.06443](https://arxiv.org/abs/2407.06443) find
  66. Zhangyin Feng, Xiaocheng Feng, Dezhi Zhao, Maojin Yang, and Bing Qin. 2024. Retrieval-generation synergy augmented large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 11661–11665. find
  67. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020). find our card
  68. Luciano Floridi and Massimo Chiriatti. 2020. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines 30 (2020), 681–694. find
  69. Jacob Fox. 2023. Data Poisoning Attacks: A New Attack Vector within AI. Available Online: https://www.cobalt.io/ blog/data-poisoning-attacks-a-new-attack-vector-within-ai [Accessed on January 28, 2024]. ↗
  70. JAKE FRANKENFIELD. 2023. What Is Personally Identifiable Information (PII)? Types and Examples. Available Online: https://www.investopedia.com/terms/p/personally-identifiable-information-pii.asp [Accessed on January 28, 2024]. ↗
  71. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. 2023. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394 (2023). find our card
  72. Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. 2023. Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration. arXiv preprint arXiv:2311.06062 (2023). — [arXiv:2311.06062](https://arxiv.org/abs/2311.06062) find
  73. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics (2024), 1–79. find our card
  74. Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723 (2020). find our card
  75. Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. 2019. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference. 113–125. find
  76. Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. 2020. Inverting gradients-how easy is it to break privacy in federated learning? Advances in Neural Information Processing Systems 33 (2020), 16937–16947. find
  77. Robin C Geyer, Tassilo Klein, and Moin Nabi. 2017. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557 (2017). — [arXiv:1712.07557](https://arxiv.org/abs/1712.07557) find
  78. Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. 2022. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.04037 (2022). find
  79. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security.… find our card
  80. Alexandre Sablayrolles Hervé Jégou Guo, Chuan and Douwe Kiela. 2021. Gradient-based adversarial attacks against text transformers. arXiv preprint arXiv:2104.13733 (2021). find our card
  81. Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. 2021. Gradient-based adversarial attacks against text transformers. arXiv preprint arXiv:2104.13733 (2021). find our card
  82. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366 (2020). find our card
  83. Maanak Gupta, CharanKumar Akiri, Kshitiz Aryal, Eli Parker, and Lopamudra Praharaj. 2023. From ChatGPT to ThreatGPT: Impact of generative AI in cybersecurity and privacy. IEEE Access (2023). find
  84. Surbhi Gupta, Abhishek Singhal, and Akanksha Kapoor. 2016. A literature survey on social engineering attacks: Phishing attack. In 2016 international conference on computing, communication and automation (ICCCA). IEEE, 537–540. find
  85. Siddhant Haldar. 2023. Gradient-based Adversarial Attacks : An Introduction. Available Online: https://medium.com/ swlh/gradient-based-adversarial-attacks-an-introduction-526238660dc9s [Accessed on January 28, 2024]. ↗
  86. Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015). find
  87. Stefan Harrer. 2023. Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine. EBioMedicine 90 (2023). find
  88. Julian Hazell. 2023. Large language models can be used to effectively scale spear phishing campaigns. arXiv preprint arXiv:2305.06972 (2023). — [arXiv:2305.06972](https://arxiv.org/abs/2305.06972) find
  89. Feng He, Tianqing Zhu, Dayong Ye, Bo Liu, Wanlei Zhou, and Philip S Yu. 2024. The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies. arXiv preprint arXiv:2407.19354 (2024). — [arXiv:2407.19354](https://arxiv.org/abs/2407.19354) find
  90. Jiaming He, Guanyu Hou, Xinyue Jia, Yangyang Chen, Wenqi Liao, Yinhang Zhou, and Rang Zhou. 2024. Data Stealing Attacks against Large Language Models via Backdooring. Electronics 13, 14 (2024), 2858. find
  91. Will Douglas Heaven. 2022. "Why Meta’s latest large language model survived only three days online". MIT Technology Review. Last accessed December 15 (2022), 2022. find
  92. Geoffrey Hinton. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531 (2015). find our card
  93. Andrew Hoang, Antoine Bosselut, Asli Celikyilmaz, and Yejin Choi. 2019. Efficient adaptation of pretrained transformers for abstractive summarization. arXiv preprint arXiv:1906.00138 (2019). — [arXiv:1906.00138](https://arxiv.org/abs/1906.00138) find
  94. Linmei Hu, Zeyi Liu, Ziwang Zhao, Lei Hou, Liqiang Nie, and Juanzi Li. 2023. A survey of knowledge enhanced pre-trained language models. IEEE Transactions on Knowledge and Data Engineering (2023). find
  95. Li Hu, Anli Yan, Hongyang Yan, Jin Li, Teng Huang, Yingying Zhang, Changyu Dong, and Chunsheng Yang. 2023. Defenses to Membership Inference Attacks: A Survey. Comput. Surveys 56, 4 (2023), 1–34. find
  96. Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang. 2023. Composite Backdoor Attacks Against Large Language Models. arXiv preprint arXiv:2310.07676 (2023). find
  97. Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. 2024. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. arXiv preprint arXiv:2403.01244 (2024). find
  98. Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. Are Large Pre-Trained Language Models Leaking Your Personal Information? arXiv preprint arXiv:2205.12628 (2022). find
  99. Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. 2023. Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv preprint arXiv:2307.10236 (2023). — [arXiv:2307.10236](https://arxiv.org/abs/2307.10236) find
  100. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674 (2023). find our card
  101. Huseyin A Inan, Osman Ramadan, Lukas Wutschitz, Daniel Jones, Victor Rühle, James Withers, and Robert Sim. 2021. Privacy analysis in language models via training data leakage report. ArXiv, abs/2101.05405 (2021). — [arXiv:2101.05405](https://arxiv.org/abs/2101.05405) find
  102. Huseyin A Inan, Osman Ramadan, Lukas Wutschitz, Daniel Jones, Victor Rühle, James Withers, and Robert Sim. 2021. Training data leakage analysis in language models. arXiv preprint arXiv:2101.05405 (2021). — [arXiv:2101.05405](https://arxiv.org/abs/2101.05405) find
  103. Abhyuday Jagannatha, Bhanu Pratap Singh Rawat, and Hong Yu. 2021. Membership inference attack susceptibility of clinical language models. arXiv preprint arXiv:2104.08305 (2021). — [arXiv:2104.08305](https://arxiv.org/abs/2104.08305) find
  104. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614 (2023). find our card
  105. Malin Jansson, Stefan Hrastinski, Stefan Stenbom, and Fredrik Enoksson. 2021. Online question and answer sessions: How students support their own and other students’ processes of inquiry in a text-based learning environment. The Internet and Higher Education 51 (2021), 100817. find
  106. Brindha Jeyaraman. 2023. Adversarial Attacks on LLMs: Safeguarding Language Models Against Manipulation. https://www.linkedin.com/pulse/adversarial-attacks-llms-safeguarding-language-models-jeyaraman/ ↗
  107. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. Comput. Surveys 55, 12 (2023), 1–38. find our card
  108. Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. Towards mitigating hallucination in large language models via self-reflection. arXiv preprint arXiv:2310.06271 (2023). find
  109. Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong. 2019. Memguard: Defending against black-box membership inference attacks via adversarial examples. In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security. 259–274. find
  110. Shuyu Jiang, Xingshu Chen, and Rui Tang. 2023. Prompt packer: Deceiving llms through compositional instruction with hidden attacks. arXiv preprint arXiv:2310.10077 (2023). — [arXiv:2310.10077](https://arxiv.org/abs/2310.10077) find
  111. Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. 2023. Lion: Adversarial Distillation of Closed-Source Large Language Model. arXiv preprint arXiv:2305.12870 (2023). — [arXiv:2305.12870](https://arxiv.org/abs/2305.12870) find
  112. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. TinyBERT: Distilling BERT for Natural Language Understanding. arXiv preprint arXiv:1909.10351 (2019). find
  113. Hongpeng Jin, Wenqi Wei, Xuyu Wang, Wenbin Zhang, and Yanzhao Wu. 2023. Rethinking learning rate tuning in the era of large language models. In 2023 IEEE 5th International Conference on Cognitive Machine Intelligence (CogMI). IEEE, 112–121. find
  114. Hongpeng Jin, Wenqi Wei, Xuyu Wang, Wenbin Zhang, and Yanzhao Wu. 2023. Rethinking Learning Rate Tuning in the Era of Large Language Models. In 2023 IEEE 5th International Conference on Cognitive Machine Intelligence (CogMI). 112–121. https://doi.org/10.1109/CogMI58952.2023.00025 — [doi:10.1109/cogmi58952.2023.00025](https://doi.org/10.1109/cogmi58952.2023.00025) ↗
  115. Mingyu Jin, Suiyuan Zhu, Beichen Wang, Zihao Zhou, Chong Zhang, Yongfeng Zhang, et al. 2024. Attackeval: How to evaluate the effectiveness of jailbreak attacking on large language models. arXiv preprint arXiv:2401.09002 (2024). — [arXiv:2401.09002](https://arxiv.org/abs/2401.09002) find
  116. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169 (2023). find our card
  117. Nikhil Kandpal, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. 2023. Backdoor attacks for in-context learning with language models. arXiv preprint arXiv:2307.14692 (2023). — [arXiv:2307.14692](https://arxiv.org/abs/2307.14692) find
  118. Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning. PMLR, 10697–10707. find our card
  119. Masahiro Kaneko, Youmi Ma, Yuki Wata, and Naoaki Okazaki. 2024. Sampling-based Pseudo-Likelihood for Membership Inference Attacks. arXiv preprint arXiv:2404.11262 (2024). find
  120. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2023. Exploiting programmatic behavior of LLMs: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733 (2023). — [arXiv:2302.05733](https://arxiv.org/abs/2302.05733) find
  121. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, Vol. 1. 2. find
  122. Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. 2024. Propile: Probing privacy leakage in large language models. Advances in Neural Information Processing Systems 36 (2024). find
  123. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213. find our card
  124. Nir Kshetri. 2023. Cybercrime and privacy threats of large language models. IT Professional 25, 3 (2023), 9–13. find
  125. Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Hima Lakkaraju. 2023. Certifying LLM safety against adversarial prompting. arXiv preprint arXiv:2309.02705 (2023). find our card
  126. Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pre-trained models. arXiv preprint arXiv:2004.06660 (2020). — [arXiv:2004.06660](https://arxiv.org/abs/2004.06660) find
  127. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360 (2016). find
  128. Raz Lapid, Ron Langberg, and Moshe Sipper. 2023. Open sesame! universal black box jailbreaking of large language models. arXiv preprint arXiv:2309.01446 (2023). — [arXiv:2309.01446](https://arxiv.org/abs/2309.01446) find
  129. Isack Lee and Haebin Seong. 2024. Do LLMs Have Political Correctness? Analyzing Ethical Biases and Jailbreak Vulnerabilities in AI Systems. arXiv preprint arXiv:2410.13334 (2024). — [arXiv:2410.13334](https://arxiv.org/abs/2410.13334) find
  130. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499 (2021). find
  131. Haoran Li, Yulin Chen, Jinglong Luo, Yan Kang, Xiaojin Zhang, Qi Hu, Chunkit Chan, and Yangqiu Song. 2023. Privacy in large language models: Attacks, defenses and future directions. arXiv preprint arXiv:2310.10383 (2023). — [arXiv:2310.10383](https://arxiv.org/abs/2310.10383) find
  132. Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. 2023. Multi-step jailbreaking privacy attacks on ChatGPT. arXiv preprint arXiv:2304.05197 (2023). find our card
  133. Jiazhao Li, Yijin Yang, Zhuofeng Wu, VG Vydiswaran, and Chaowei Xiao. 2023. ChatGPT as an attack tool: Stealthy textual backdoor attack via blackbox generative model trigger. arXiv preprint arXiv:2304.14475 (2023). find
  134. Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. 2021. Backdoor attacks on pre-trained models by layerwise weight poisoning. arXiv preprint arXiv:2108.13888 (2021). find
  135. Linyi Li, Tao Xie, and Bo Li. 2023. SoK: Certified robustness for deep neural networks. In 2023 IEEE symposium on security and privacy (SP). IEEE, 1289–1310. find
  136. Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2021. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679 (2021). — [arXiv:2110.05679](https://arxiv.org/abs/2110.05679) find
  137. Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. 2023. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191 (2023). find our card
  138. Yanzhou Li, Shangqing Liu, Kangjie Chen, Xiaofei Xie, Tianwei Zhang, and Yang Liu. 2023. Multi-target backdoor attacks for code pre-trained models. arXiv preprint arXiv:2306.08350 (2023). find
  139. Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. 2021. Neural attention distillation: Erasing backdoor triggers from deep neural networks. arXiv preprint arXiv:2101.05930 (2021). — [arXiv:2101.05930](https://arxiv.org/abs/2101.05930) find
  140. Jiawei Liang, Siyuan Liang, Man Luo, Aishan Liu, Dongchen Han, Ee-Chien Chang, and Xiaochun Cao. 2024. Vl-trojan: Multimodal instruction backdoor attacks against autoregressive visual language models. arXiv preprint arXiv:2402.13851 (2024). — [arXiv:2402.13851](https://arxiv.org/abs/2402.13851) find
  141. Yi-Shan Lin, Wen-Chuan Lee, and Z Berkay Celik. 2021. What do you see? Evaluation of explainable artificial intelligence (XAI) interpretability through neural backdoors. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 1027–1035. find
  142. Zachary C Lipton. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 16, 3 (2018), 31–57. find
  143. Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2018. Fine-Pruning: Defending against backdooring attacks on deep neural networks. In International symposium on research in attacks, intrusions, and defenses. Springer, 273–294. find
  144. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451 (2023). find our card
  145. Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024. Automatic and universal prompt injection attacks against large language models. arXiv preprint arXiv:2403.04957 (2024). — [arXiv:2403.04957](https://arxiv.org/abs/2403.04957) find
  146. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2023. Prompt Injection attack against LLM-integrated Applications. arXiv preprint arXiv:2306.05499 (2023). find our card
  147. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023. Jailbreaking ChatGPT via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860 (2023). find our card
  148. Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In USENIX Security Symposium. find
  149. Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. 2018. Trojaning attack on neural networks. In 25th Annual Network And Distributed System Security Symposium (NDSS 2018). Internet Soc. find
  150. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692 (2019). find
  151. Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023. Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models’ Alignment. arXiv preprint arXiv:2308.05374 (2023). find our card
  152. Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. 2023. Analyzing leakage of personally identifiable information in language models. arXiv preprint arXiv:2302.00539 (2023). find
  153. Scott Lundberg. 2017. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874 (2017). find our card
  154. Junyu Luo, Cao Xiao, and Fenglong Ma. 2023. Zero-resource hallucination prevention for large language models. arXiv preprint arXiv:2309.02654 (2023). find
  155. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747 (2023). find
  156. Malwarebytes. 2023. What is iPhone jailbreaking. Available Online: https://www.malwarebytes.com/iphone-jailbreaking. [Accessed on January 28, 2024]. ↗
  157. Courtney Mansfield, Amandalynne Paullada, and Kristen Howell. 2022. Behind the Mask: Demographic bias in name detection for PII masking. arXiv preprint arXiv:2205.04505 (2022). find
  158. James Manyika and Sissie Hsiao. 2023. An overview of Bard: an early experiment with generative AI. AI. Google Static Documents 2 (2023). find
  159. Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2023. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 15009–15018. find our card
  160. Ariana Martino, Michael Iannelli, and Coleen Truong. 2023. Knowledge injection to counter large language model (LLM) hallucination. In European Semantic Web Conference. Springer, 182–185. find
  161. Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. 2023. Membership Inference Attacks against Language Models via Neighbourhood Comparison. arXiv preprint arXiv:2305.18462 (2023). find
  162. Natalie Maus, Patrick Chao, Eric Wong, and Jacob Gardner. 2023. Adversarial prompting for black box foundation models. arXiv preprint arXiv:2302.04237 (2023). — [arXiv:2302.04237](https://arxiv.org/abs/2302.04237) find
  163. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Vol. 24. Elsevier, 109–165. find
  164. Leland McInnes, John Healy, and Steve Astels. 2017. HDBSCAN: Hierarchical density based clustering. J. Open Source Softw. 2, 11 (2017), 205. find our card
  165. Yusuf Mehdi. 2023. Reinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web. Available Online: https://blogs.microsoft.com/blog/2023/02/07/reinventing-search-with-a-new-ai-powered-microsoft-bing-and-edge-your-copilot-for-the-web/ [Accessed on January 28, 2024]. ↗
  166. Meta-Llma. 2023. Llama-2-13b-hf. Available Online: https://huggingface.co/meta-llama/Llama-2-13b-hf [Accessed on January 28, 2024]. ↗
  167. Meta-Llma. 2023. Llama-2-7b. Available Online: https://huggingface.co/huggyllama/llama-7bf [Accessed on January 28, 2024]. ↗
  168. Meta-Llma. 2023. LLaMA-30b. Available Online: https://huggingface.co/huggyllama/llama-30b [Accessed on January 28, 2024]. ↗
  169. Microsoft. 2023. Bing AI Chatbot. https://www.bing.com/chat ↗
  170. Silvia Milano, Joshua A McGrane, and Sabina Leonelli. 2023. Large language models challenge the future of higher education. Nature Machine Intelligence 5, 4 (2023), 333–334. find
  171. Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196 (2024). find our card
  172. Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. 2022. Quantifying privacy risks of masked language models using membership inference attacks. arXiv preprint arXiv:2203.03929 (2022). find
  173. Aayush Mittal. 2023. Prompt Hacking and Misuse of LLMs. "[Accessed on January 28, 2024]". https://www.unite.ai/ prompt-hacking-and-misuse-of-llm ↗
  174. Domenic Molinaro. 2023. What Is Rooting? The Risks of Rooting Your Android Device. Available Online: https://www.avast.com/c-rooting-android [Accessed on January 28, 2024]. ↗
  175. Ervin Moore, Ahmed Imteaj, Shabnam Rezapour, and M. Hadi Amini. 2023. A Survey on Secure and Private Federated Learning Using Blockchain: Theory and Application in Resource-Constrained Computing. IEEE Internet of Things Journal 10, 24 (2023), 21942–21958. https://doi.org/10.1109/JIOT.2023.3313055 — [doi:10.1109/jiot.2023.3313055](https://doi.org/10.1109/jiot.2023.3313055) ↗
  176. Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D Griffin. 2023. Use of LLMs for illicit purposes: Threats, prevention measures, and vulnerabilities. arXiv preprint arXiv:2308.12833 (2023). — [arXiv:2308.12833](https://arxiv.org/abs/2308.12833) find
  177. Yuta Nakamura, Shouhei Hanaoka, Yukihiro Nomura, Naoto Hayashi, Osamu Abe, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. 2020. KART: Privacy leakage framework of language models pre-trained with clinical records. arXiv preprint arXiv:2101.00036 (2020). — [arXiv:2101.00036](https://arxiv.org/abs/2101.00036) find
  178. Krishna Kanth Nakka, Ahmed Frikha, Ricardo Mendes, Xue Jiang, and Xuebing Zhou. 2024. PII-Compass: Guiding LLM training data extraction prompts towards the target PII via grounding. arXiv preprint arXiv:2407.02943 (2024). find
  179. Jomilė Nakutavičiūtė. 2023. Why root Android phones? Available Online: https://nordvpn.com/blog/why-you-shouldnt-root-android [Accessed on January 28, 2024]. ↗
  180. Milad Nasr, Reza Shokri, and Amir Houmansadr. 2018. Machine learning with membership privacy using adversarial regularization. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security. 634–646. find
  181. Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2023. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435 (2023). find our card
  182. Seth Neel and Peter Chang. 2023. Privacy Issues in Large Language Models: A Survey. arXiv preprint arXiv:2312.06717 (2023). find
  183. Thuy Dung Nguyen, Tuan Nguyen, Phi Le Nguyen, Hieu H Pham, Khoa D Doan, and Kok-Seng Wong. 2024. Backdoor attacks and defenses in federated learning: Survey, challenges and future research directions. Engineering Applications of Artificial Intelligence 127 (2024), 107166. find
  184. Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. 2022. Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460 (2022). — [arXiv:2205.07460](https://arxiv.org/abs/2205.07460) find
  185. OpenAI. 2023. API to Prevent Prompt Injection & Jailbreaks. Available Online: https://community.openai.com/t/api-to-prevent-prompt-injection-jailbreaks/203514 [Accessed on January 28, 2024]. ↗
  186. OpenAI. 2023. ChatGPT. Available Online: https://chat.openai.com [Accessed on January 28, 2024]. ↗
  187. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730–27744. find
  188. Mustafa Safa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, and Rahul Gupta. 2023. Controlling the extraction of memorized data from large language models via prompt-tuning. arXiv preprint arXiv:2305.11759 (2023). find
  189. Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. 2020. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP). IEEE, 1314–1331. find
  190. Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min-Yen Kan, and William Yang Wang. 2023. On the Risk of Misinformation Pollution with Large Language Models. arXiv preprint arXiv:2305.13661 (2023). find our card
  191. Prachi (Nayyar) Pathak. 2023. How do you protect Machine Learning from attacks? Available Online: https://www. linkedin.com/advice/1/how-do-you-protect-machine-learning-from-attacks\#data-poisoning-attack [Accessed on January 28, 2024]. find
  192. Pearlhawaii.com. 2023. WHAT IS JAILBREAKING, CRACKING, OR ROOTING A MOBILE DEVICE? Available Online: https://pearlhawaii.com/what-is-jailbreaking-cracking-or-rooting-a-mobile-device. [Accessed on January 28, 2024]. ↗
  193. Rodrigo Pedro, Daniel Castro, Paulo Carreira, and Nuno Santos. 2023. From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application? arXiv preprint arXiv:2308.01990 (2023). — [arXiv:2308.01990](https://arxiv.org/abs/2308.01990) find
  194. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527 (2022). find our card
  195. Roy H Perlis, Kristin Lunz Trujillo, Jon Green, Alauna Safarpour, James N Druckman, Mauricio Santillana, Katherine Ognyanova, and David Lazer. 2023. Misinformation, trust, and use of ivermectin and hydroxychloroquine for COVID-19. In JAMA Health Forum, Vol. 4. American Medical Association, e233257–e… find
  196. Miguel Piedrafita. 2022. Methodologies of Jailbreaking. Available Online: https://learnprompting.org/docs/prompt_ hacking/jailbreakin. [Accessed on January 28, 2024]. ↗
  197. Francesco Pierri, Luca Luceri, Nikhil Jindal, and Emilio Ferrara. 2023. Propaganda and misinformation on Facebook and Twitter during the Russian invasion of Ukraine. In Proceedings of the 15th ACM web science conference 2023. 65–74. find
  198. Promptbase. 2024. Promptbase. Available Online: https://promptbase.com/ [Accessed on August 23, 2024]. ↗
  199. Learn Prompting. 2023. Jailbreaking. Available Online: https://learnprompting.org/docs/prompt_hacking/jailbreaking [Accessed on January 28, 2024]. ↗
  200. Learn Prompting. 2023. Your Guide to Generative AI. Available Online: https://learnprompting.org [Accessed on January 28, 2024]. ↗
  201. Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2020. ONION: A simple and effective defense against textual backdoor attacks. arXiv preprint arXiv:2011.10369 (2020). find our card
  202. Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. 2021. Hidden Killer: Invisible textual backdoor attacks with syntactic trigger. arXiv preprint arXiv:2105.12400 (2021). find our card
  203. Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. 2023. Visual adversarial examples jailbreak aligned large language models. In The Second Workshop on New Frontiers in Adversarial Machine Learning. find
  204. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9. find
  205. Mohammad Raeini. 2023. Privacy-preserving large language models (PPLLMs). Available at SSRN 4512071 (2023). find
  206. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485–5551. find
  207. Goutham Ramakrishnan and Aws Albarghouthi. 2022. Backdoors in neural models of source code. In 2022 26th International Conference on Pattern Recognition (ICPR). IEEE, 2892–2899. find
  208. Delong Ran, Jinyuan Liu, Yichen Gong, Jingyi Zheng, Xinlei He, Tianshuo Cong, and Anyu Wang. 2024. JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models. arXiv preprint arXiv:2406.09321 (2024). — [arXiv:2406.09321](https://arxiv.org/abs/2406.09321) find
  209. Aman Rangapur, Haoran Wang, and Kai Shu. 2023. Investigating online financial misinformation and its consequences: A computational perspective. arXiv preprint arXiv:2309.12363 (2023). — [arXiv:2309.12363](https://arxiv.org/abs/2309.12363) find
  210. Roger Ratcliff. 1990. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review 97, 2 (1990), 285. find
  211. Jingjing Ren, Ashwin Rao, Martina Lindorfer, Arnaud Legout, and David Choffnes. 2016. ReCon: Revealing and controlling PII leaks in mobile network traffic. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services. 361–374. find
  212. Kui Ren, Tianhang Zheng, Zhan Qin, and Xue Liu. 2020. Adversarial attacks and defenses in deep learning. Engineering 6, 3 (2020), 346–360. find
  213. Facebook Resrach. 2024. LAMA: LAnguage Model Analysis. Available Online: https://github.com/facebookresearch/ LAMA [Accessed on August 23, 2024]. ↗
  214. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why should I trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1135–1144. find our card
  215. Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023. SmoothLLM: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684 (2023). find our card
  216. Mirka Saarela and Vili Podgorelec. 2024. Recent Applications of Explainable AI (XAI): A Systematic Literature Review. Applied Sciences 14, 19 (2024), 8884. find
  217. Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. 2019. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning. PMLR, 5558–5567. find
  218. Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. 2018. ML-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01246 (2018). — [arXiv:1806.01246](https://arxiv.org/abs/1806.01246) find
  219. Sander Schulhoff. 2024. Instruction Defense. Available Online: https://learnprompting.org/docs/prompt_hacking/ defensive_measures/instruction [Accessed on October 11, 2024]. ↗
  220. Sander Schulhoff. 2024. Instruction Defense. Available Online: https://learnprompting.org/docs/prompt_hacking/ defensive_measures/post_prompting [Accessed on October 11, 2024]. ↗
  221. Sander Schulhoff. 2024. Sandwich Defense. Available Online: https://learnprompting.org/docs/prompt_hacking/ defensive_measures/sandwich_defense [Accessed on October 11, 2024]. ↗
  222. Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. 2021. You autocomplete me: Poisoning vulnerabilities in neural code completion. In 30th USENIX Security Symposium (USENIX Security 21). 1559–1575. find
  223. SECWRITER. 2023. Prompt Hacking and Misuse of LLMs. Available Online: https://cyberdom.blog/2023/06/17/ understanding-prompt-injection-genai-risks [Accessed on January 28, 2024]. ↗
  224. Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision. 618–626. find our card
  225. Jose Selvi. 2023. Exploring Prompt Injection Attacks. Available Online: https://research.nccgroup.com/2022/12/05/ exploring-prompt-injection-attacks/. [Accessed on January 28, 2024]. ↗
  226. Jesse Senechal, Eric Ekholm, Samaher Aljudaibi, Mary Strawderman, and Chris Parthemos. 2023. Balancing the Benefits and Risks of Large Language AI Models in K12 Public Schools. (2023). find
  227. Zeyang Sha, Xinlei He, Pascal Berrang, Mathias Humbert, and Yang Zhang. 2022. Fine-tuning is all you need to mitigate backdoor attacks. arXiv preprint arXiv:2212.09067 (2022). — [arXiv:2212.09067](https://arxiv.org/abs/2212.09067) find
  228. Muhammad Ahmed Shah, Roshan Sharma, Hira Dhamyal, Raphael Olivier, Ankit Shah, Dareen Alharthi, Hazim T Bukhari, Massa Baali, Soham Deshmukh, Michael Kuhlmann, et al. 2023. Loft: Local proxy fine-tuning for improving transferability of adversarial attacks against large language model. arXiv preprint… — [arXiv:2310.04445](https://arxiv.org/abs/2310.04445) find
  229. Shawn Shan, Wenxin Ding, Josephine Passananti, Haitao Zheng, and Ben Y Zhao. 2023. Prompt-specific poisoning attacks on text-to-image generative models. arXiv preprint arXiv:2310.13828 (2023). — [arXiv:2310.13828](https://arxiv.org/abs/2310.13828) find
  230. Kun Shao, Junan Yang, Yang Ai, Hui Liu, and Yu Zhang. 2021. BDDR: An effective defense against textual backdoor attacks. Computers & Security 110 (2021), 102433. find
  231. Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. 2023. Plug and Pray: Exploiting off-the-shelf components of Multi-Modal Models. arXiv preprint arXiv:2307.14539 (2023). find
  232. Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu-Ghazaleh. 2023. Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv preprint arXiv:2310.10844 (2023). find
  233. Virat Shejwalkar, Huseyin A Inan, Amir Houmansadr, and Robert Sim. 2021. Membership inference attacks against NLP classification models. In NeurIPS 2021 Workshop Privacy in Machine Learning. find
  234. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2023. "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. arXiv preprint arXiv:2308.03825 (2023). find our card
  235. Jiawen Shi, Yixin Liu, Pan Zhou, and Lichao Sun. 2023. BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT. arXiv preprint arXiv:2304.12298 (2023). — [arXiv:2304.12298](https://arxiv.org/abs/2304.12298) find
  236. Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. 2024. Optimization-based Prompt Injection Attack to LLM-as-a-Judge. arXiv preprint arXiv:2403.17710 (2024). — [arXiv:2403.17710](https://arxiv.org/abs/2403.17710) find
  237. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789 (2023). find
  238. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 (2020). find our card
  239. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP). IEEE, 3–18. find our card
  240. Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. 2023. On the exploitability of instruction tuning. Advances in Neural Information Processing Systems 36 (2023), 61836–61856. find
  241. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing. 1631–1642. find
  242. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Khyathi Chandu, Jennifer Dumas, Li Lucy, Xinxi Lyu, et al. 2023. Dolma: An Open Corpus of 3 Trillion Tokens for Language Model Pretraining Research. Allen Institute for AI, Tech. Rep (2023). find
  243. Congzheng Song and Ananth Raghunathan. 2020. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security. 377–390. find
  244. Congzheng Song and Vitaly Shmatikov. 2019. Auditing data provenance in text-generation models. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 196–206. find
  245. Taylor Sorensen, Joshua Robinson, Christopher Michael Rytting, Alexander Glenn Shaw, Kyle Jeffrey Rogers, Alexia Pauline Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022. An information-theoretic approach to prompt engineering without ground truth labels. arXiv preprint arXiv:2203.11364… find
  246. Dirk HR Spennemann. 2023. Exploring Ethical Boundaries: Can ChatGPT Be Prompted to Give Advice on How to Cheat in University Assignments? (2023). find
  247. Ningxin Su, Chenghao Hu, Baochun Li, and Bo Li. 2023. TITANIC: Towards Production Federated Learning with Large Language Models. Available Online: https://iqua.ece.toronto.edu/papers/ningxinsu-infocom24.pdf [Accessed on February 6, 2024]. ↗
  248. Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. 2024. TrustLLM: Trustworthiness in Large Language Models. arXiv preprint arXiv:2401.05561 (2024). find our card
  249. Indranil Sur, Karan Sikka, Matthew Walmer, Kaushik Koneripalli, Anirban Roy, Xiao Lin, Ajay Divakaran, and Susmit Jha. 2023. TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 165–175… find
  250. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2818–2826. find
  251. Annalisa Szymanski, Noah Ziems, Heather A Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A Metoyer. 2024. Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks. arXiv preprint arXiv:2410.20266 (2024). — [arXiv:2410.20266](https://arxiv.org/abs/2410.20266) find
  252. Yonglin Tian, Xuan Li, Hui Zhang, Chen Zhao, Bai Li, Xiao Wang, and Fei-Yue Wang. 2023. VistaGPT: Generative parallel transformers for vehicles with intelligent systems for transport automation. IEEE Transactions on Intelligent Vehicles (2023). find
  253. Yonglin Tian, Jiangong Wang, Yutong Wang, Chen Zhao, Fei Yao, and Xiao Wang. 2022. Federated vehicular transformers and their federations: Privacy-preserving computing and cooperation for autonomous driving. IEEE Transactions on Intelligent Vehicles (2022). find
  254. Meng Tong, Kejiang Chen, Yuang Qi, Jie Zhang, Weiming Zhang, and Nenghai Yu. 2023. Privinfer: Privacy-preserving inference for black-box large language model. arXiv preprint arXiv:2310.12214 (2023). — [arXiv:2310.12214](https://arxiv.org/abs/2310.12214) find
  255. Alexander Tornede, Difan Deng, Theresa Eimer, Joseph Giovanelli, Aditya Mohan, Tim Ruhkopf, Sarah Segel, Daphne Theodorakopoulos, Tanja Tornede, Henning Wachsmuth, et al. 2023. AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks. arXiv preprint arXiv:2306.0… find
  256. Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei. 2019. Demystifying membership inference attacks in machine learning as a service. IEEE Transactions on Services Computing 14, 6 (2019), 2073–2089. find
  257. Thomas Vakili, Anastasios Lamproudis, Aron Henriksson, and Hercules Dalianis. 2022. Downstream task performance of BERT models pre-trained using automatically de-identified clinical data. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. 4245–4252. find
  258. Apostol Vassilev, Alina Oprea, Alie Fordyce, and Hyrum Andersen. 2024. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations. NIST Trustworthy and Responsible AI NIST AI 100-2e2023 (2024). find
  259. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). find
  260. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. Advances in neural information processing systems 29 (2016). find
  261. Paul Voigt and Axel Von dem Bussche. 2017. The eu general data protection regulation (GDPR). A Practical Guide, 1st Ed., Cham: Springer International Publishing 10, 3152676 (2017), 10–5555. find
  262. Eric Wallace, Tony Z Zhao, Shi Feng, and Sameer Singh. 2020. Concealed data poisoning attacks on NLP models. arXiv preprint arXiv:2010.12563 (2020). find
  263. Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning Language Models During Instruction Tuning. arXiv preprint arXiv:2305.00944 (2023). — [arXiv:2305.00944](https://arxiv.org/abs/2305.00944) find
  264. Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. 2019. Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP). IEEE, 707–723. find
  265. Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, et al. 2023. On the robustness of ChatGPT: An adversarial and out-of-distribution perspective. arXiv preprint arXiv:2302.12095 (2023). find
  266. Jiongxiao Wang, Zichen Liu, Keun Hee Park, Muhao Chen, and Chaowei Xiao. 2023. Adversarial Demonstration Attacks on Large Language Models. arXiv preprint arXiv:2305.14950 (2023). — [arXiv:2305.14950](https://arxiv.org/abs/2305.14950) find
  267. Shang Wang, Tianqing Zhu, Bo Liu, Ding Ming, Xu Guo, Dayong Ye, and Wanlei Zhou. 2024. Unique Security and Privacy Threats of Large Language Model: A Comprehensive Survey. arXiv preprint arXiv:2406.07973 (2024). — [arXiv:2406.07973](https://arxiv.org/abs/2406.07973) find
  268. Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. 2019. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST) 10, 2 (2019), 1–37. find
  269. Yequan Wang, Jiawen Deng, Aixin Sun, and Xuying Meng. 2022. Perplexity from plm is unreliable for evaluating text quality. arXiv preprint arXiv:2210.05892 (2022). — [arXiv:2210.05892](https://arxiv.org/abs/2210.05892) find
  270. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021). find
  271. Yijue Wang, Nuo Xu, Shaoyi Huang, Kaleel Mahmood, Dan Guo, Caiwen Ding, Wujie Wen, and Sanguthevar Rajasekaran. 2022. Analyzing and Defending against Membership Inference Attacks in Natural Language Processing Classification. In 2022 IEEE International Conference on Big Data (Big Data). IEEE, 5823–5… find
  272. Yifei Wang, Dizhan Xue, Shengjie Zhang, and Shengsheng Qian. 2024. BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents. arXiv preprint arXiv:2406.03007 (2024). find
  273. Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. 2020. Generalizing from a few examples: A survey on few-shot learning. ACM computing Surveys 53, 3 (2020), 1–34. find
  274. Zhenhua Wang, Wei Xie, Kai Chen, Baosheng Wang, Zhiwen Gui, and Enze Wang. 2023. Self-deception: Reverse penetrating the semantic firewall of large language models. arXiv preprint arXiv:2308.11521 (2023). — [arXiv:2308.11521](https://arxiv.org/abs/2308.11521) find
  275. Maurice Weber, Xiaojun Xu, Bojan Karlaš, Ce Zhang, and Bo Li. 2023. Rab: Provable robustness against backdoor attacks. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 1311–1328. find
  276. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does LLM safety training fail? arXiv preprint arXiv:2307.02483 (2023). find our card
  277. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022). find our card
  278. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837. find our card
  279. Wenqi Wei, Ling Liu, Margaret Loper, Ka-Ho Chow, Mehmet Emre Gursoy, Stacey Truex, and Yanzhao Wu. 2020. A framework for evaluating client privacy leakages in federated learning. In Computer Security–ESORICS 2020: 25th European Symposium on Research in Computer Security, ESORICS 2020, Guildford, UK,… find
  280. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 (2021). find our card
  281. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 214–… find
  282. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv preprint arXiv:2302.11382 (2023). find our card
  283. Chenxi Whitehouse, Tillman Weyde, Pranava Madhyastha, and Nikos Komninos. 2022. Evaluation of fake news detection with knowledge-enhanced language models. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 16. 1425–1429. find
  284. Genta Indra Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, and Daniel Preotiuc-Pietro. 2023. Overcoming catastrophic forgetting in massively multilingual continual learning. arXiv preprint arXiv:2305.16252 (2023). find
  285. Fangzhao Wu, Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, and Xing Xie. 2023. Defending ChatGPT against Jailbreak Attack via Self-Reminder. (2023). find
  286. Ruihan Wu, Xiangyu Chen, Chuan Guo, and Kilian Q Weinberger. 2023. Learning to Invert: Simple adaptive attacks for gradient inversion in federated learning. In Uncertainty in Artificial Intelligence. PMLR, 2293–2303. find
  287. Xiaodong Wu, Ran Duan, and Jianbing Ni. 2023. Unveiling security, privacy, and ethical concerns of ChatGPT. Journal of Information and Intelligence (2023). find
  288. Yuanwei Wu, Yue Huang, Yixin Liu, Xiang Li, Pan Zhou, and Lichao Sun. 2024. Can Large Language Models Automatically Jailbreak GPT-4V? arXiv preprint arXiv:2407.16686 (2024). — [arXiv:2407.16686](https://arxiv.org/abs/2407.16686) find
  289. Yanzhao Wu and Ling Liu. 2023. Selecting and composing learning rate policies for deep neural networks. ACM Transactions on Intelligent Systems and Technology 14, 2 (2023), 1–25. find
  290. Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864 (2023). find our card
  291. Zhaohan Xi, Tianyu Du, Changjiang Li, Ren Pang, Shouling Ji, Jinghui Chen, Fenglong Ma, and Ting Wang. 2023. Defending pre-trained language models as few-shot learners against backdoor attacks. arXiv preprint arXiv:2309.13256 (2023). — [arXiv:2309.13256](https://arxiv.org/abs/2309.13256) find
  292. Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, and Bo Li. 2024. Badchain: Backdoor chain-of-thought prompting for large language models. arXiv preprint arXiv:2401.12242 (2024). — [arXiv:2401.12242](https://arxiv.org/abs/2401.12242) find
  293. Zhen Xiang, Zidi Xiong, and Bo Li. 2024. CBD: A certified backdoor detector based on local dominant probability. Advances in Neural Information Processing Systems 36 (2024). find
  294. Yuan Xin, Zheng Li, Ning Yu, Michael Backes, and Yang Zhang. 2022. Membership Leakage in Pre-trained Language Models. (2022). find
  295. Lei Xu, Yangyi Chen, Ganqu Cui, Hongcheng Gao, and Zhiyuan Liu. 2022. Exploring the universal vulnerability of prompt-based learning paradigm. arXiv preprint arXiv:2204.05239 (2022). find
  296. Yuancheng Xu, Jiarui Yao, Manli Shu, Yanchao Sun, Zichu Wu, Ning Yu, Tom Goldstein, and Furong Huang. 2024. Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models. arXiv preprint arXiv:2402.06659 (2024). — [arXiv:2402.06659](https://arxiv.org/abs/2402.06659) find
  297. Anli Yan, Teng Huang, Lishan Ke, Xiaozhang Liu, Qi Chen, and Changyu Dong. 2023. Explanation leaks: Explanation-guided model extraction attacks. Information Sciences 632 (2023), 269–284. find
  298. Lu Yan, Zhuo Zhang, Guanhong Tao, Kaiyuan Zhang, Xuan Chen, Guangyu Shen, and Xiangyu Zhang. 2024. ParaFuzz: An interpretability-driven technique for detecting poisoned samples in nlp. Advances in Neural Information Processing Systems 36 (2024). find
  299. Shenao Yan, Shen Wang, Yue Duan, Hanbin Hong, Kiho Lee, Doowon Kim, and Yuan Hong. 2024. An LLM-Assisted Easy-to-Trigger Backdoor Attack on Code Completion Models: Injecting Disguised Vulnerabilities against Strong Detection. arXiv preprint arXiv:2406.06822 (2024). — [arXiv:2406.06822](https://arxiv.org/abs/2406.06822) find
  300. Haomiao Yang, Kunlan Xiang, Hongwei Li, and Rongxing Lu. 2023. A Comprehensive Overview of Backdoor Attacks in Large Language Models within Communication Networks. arXiv preprint arXiv:2308.14367 (2023). — [arXiv:2308.14367](https://arxiv.org/abs/2308.14367) find
  301. Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. 2021. Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in NLP models. arXiv preprint arXiv:2103.15543 (2021). find
  302. Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. 2021. RAP: Robustness-aware perturbations for defending against backdoor attacks on NLP models. arXiv preprint arXiv:2110.07831 (2021). find
  303. Yang Yang. 2022. Holistic risk assessment of inference attacks in machine learning. arXiv preprint arXiv:2212.10628 (2022). — [arXiv:2212.10628](https://arxiv.org/abs/2212.10628) find
  304. Zhou Yang, Zhensu Sun, Terry Zhuo Yue, Premkumar Devanbu, and David Lo. 2024. Robustness, security, privacy, explainability, efficiency, and usability of large language models for code. arXiv preprint arXiv:2403.07506 (2024). — [arXiv:2403.07506](https://arxiv.org/abs/2403.07506) find
  305. Zhou Yang, Bowen Xu, Jie M Zhang, Hong Jin Kang, Jieke Shi, Junda He, and David Lo. 2024. Stealthy backdoor attack for code models. IEEE Transactions on Software Engineering (2024). find
  306. Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing (2024), 100211. find our card
  307. Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. 2022. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 3093–3106. find
  308. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF). IEEE, 268–282. find
  309. Wencong You, Zayd Hammoudeh, and Daniel Lowd. 2023. Large language models are better adversaries: Exploring generative clean-label backdoor attacks against text classifiers. arXiv preprint arXiv:2310.18603 (2023). find
  310. Jiahao Yu, Xingwei Lin, and Xinyu Xing. 2023. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253 (2023). find our card
  311. Zhiyuan Yu, Xiaogeng Liu, Shunning Liang, Zach Cameron, Chaowei Xiao, and Ning Zhang. 2024. Don’t Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models. arXiv preprint arXiv:2403.17336 (2024). — [arXiv:2403.17336](https://arxiv.org/abs/2403.17336) find
  312. Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt. 2020. Analyzing information leakage of updates to natural language models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications securit… find
  313. Shudong Zhang, Haichang Gao, and Qingxun Rao. 2021. Defense against adversarial attacks by reconstructing images. IEEE Transactions on Image Processing 30 (2021), 6117–6129. find
  314. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022). find
  315. Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. 2020. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST) 11, 3 (2020), 1–41. find
  316. Xinyang Zhang, Zheng Zhang, Shouling Ji, and Ting Wang. 2021. Trojaning language models for fun and profit. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 179–197. find
  317. Yiming Zhang and Daphne Ippolito. 2023. Prompts should not be seen as secrets: Systematically measuring prompt extraction attack success. arXiv preprint arXiv:2307.06865 (2023). — [arXiv:2307.06865](https://arxiv.org/abs/2307.06865) find
  318. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. DIALOGPT: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536 (2019). find our card
  319. Zhiyuan Zhang, Lingjuan Lyu, Xingjun Ma, Chenguang Wang, and Xu Sun. 2022. Fine-Mixing: Mitigating backdoors in fine-tuned language models. arXiv preprint arXiv:2210.09545 (2022). find
  320. Zhexin Zhang, Junxiao Yang, Pei Ke, and Minlie Huang. 2023. Defending large language models against jailbreaking attacks through goal prioritization. arXiv preprint arXiv:2311.09096 (2023). find
  321. Shuai Zhao, Jinming Wen, Luu Anh Tuan, Junbo Zhao, and Jie Fu. 2023. Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models. arXiv preprint arXiv:2305.01219 (2023). find
  322. Siyun Zhao, Yuqing Yang, Zilong Wang, Zhiyuan He, Luna K Qiu, and Lili Qiu. 2024. Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely. arXiv preprint arXiv:2409.14924 (2024). — [arXiv:2409.14924](https://arxiv.org/abs/2409.14924) find
  323. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023). find our card
  324. Xuejun Zhao, Wencan Zhang, Xiaokui Xiao, and Brian Lim. 2021. Exploiting explanations for model inversion attacks. In Proceedings of the IEEE/CVF international conference on computer vision. 682–692. find
  325. Abdel-Aty Zheng, Wang Wang, and Ding. 2023. Chat-GPT is on the Horizon: Could a Large Language Model be Suitable for Intelligent Traffic Safety Research and Applications? https://arxiv.org/ftp/arxiv/papers/2303/2303.05382.pdf (2023). ↗
  326. Fei Zheng. 2023. Input Reconstruction Attack against Vertical Federated Large Language Models. arXiv preprint arXiv:2311.07585 (2023). find
  327. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with MT-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623. find
  328. Ou Zheng, Mohamed Abdel-Aty, Dongdong Wang, Chenzhu Wang, and Shengxuan Ding. 2023. TrafficSafetyGPT: Tuning a Pre-trained Large Language Model to a Domain-Specific Expert in Transportation Safety. arXiv preprint arXiv:2307.15311 (2023). — [arXiv:2307.15311](https://arxiv.org/abs/2307.15311) find
  329. Weikang Zhou, Xiao Wang, Limao Xiong, Han Xia, Yingshuang Gu, Mingxu Chai, Fukang Zhu, Caishuang Huang, Shihan Dou, Zhiheng Xi, et al. 2024. EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models. arXiv preprint arXiv:2403.12171 (2024). — [arXiv:2403.12171](https://arxiv.org/abs/2403.12171) find
  330. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910 (2022). find our card
  331. Ligeng Zhu, Zhijian Liu, and Song Han. 2019. Deep leakage from gradients. Advances in neural information processing systems 32 (2019). find
  332. Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Rui Lv, Zhenya Huang, Guanhao Zhao, Zheng Zhang, Qingyang Mao, Shijin Wang, et al. 2023. Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective. arXiv preprint arXiv:2306.10512 (2023). find
  333. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 (2019). find
  334. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 (2023). find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/das-2024-security-privacy-challenges-large-language/