Elamur Research⌕
☰
home / papers / 2026 / InferPilot: Autonomous Inference Attacks Against ML Services With LLM-Based Agents

InferPilot: Autonomous Inference Attacks Against ML Services With LLM-Based Agents

Paper · 2026 · Yixin Wu, Rui Wen, Chi Cui et al.

Summary

Running inference attacks to audit ML services is hard for non-specialists. They must pick shadow datasets with matching labels, choose suitable architectures and tune hyperparameters, and these choices multiply into a huge configuration space. The authors present InferPilot, an LLM-driven system that carries out such audits on its own, given only black-box access and a short description of the service's task and input/output formats. Its design splits the work between two agents. A ControllerAgent decides which attacks apply, launches them and tracks their progress. Parallel AttackAgents execute the attacks inside an environment that provides Linux shells, starter attack scripts, datasets and models. Each agent works from a hand-crafted, task-specific action set with guidelines.

The system was tested on 20 target services. With GPT-4o it finished every possible attack in all runs, cost little per run, and came close to an expert baseline built on ML-Doctor. An analysis of execution traces shows how the design reduces typical agent errors. The tool could let providers, auditors and regulators assess privacy and IP risks without deep expertise.

Key points

  • Attacks and test setup: InferPilot covers four attack families: membership inference, model stealing, data reconstruction and attribute inference. The 20 target services were built from five datasets and four model architectures.
  • Task completion: GPT-4o reached a 100.0% task completion rate over five runs. MLAgentBench, adapted to the same environment, completed only 26.3%.
  • Attack quality and cost:
    • Membership inference accuracy was just 1.0% below the ML-Doctor human-expert baseline.
    • An average run cost $0.627 in tokens and took 27.11 steps.
  • Adaptivity and backbones: The agent changes its strategy under constraints such as query quotas, adversarial training and label-only access. Closed-source backbones performed best, though open-source ones, notably DeepSeek, have closed much of the gap.
  • LLM-based targets: In an extension to in-context-learning membership inference on DBPedia, attack accuracy was 75.50% on Vicuna-7B, 88.75% on LLaMA-2-7B and 77.00% on Mistral-7B. The code is on GitHub (TrustAIRLab/InferPilot), with access granted on request.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Helmholtz Center for Information SecuritySaarbrücken · Germany20 works · 1,172 citations🎓Institute of Science TokyoTokyo · Japan14 works · 20 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Yixin Wu, Rui Wen, Chi Cui, Michael Backes and Yang Zhang. 2026. InferPilot: Autonomous Inference Attacks Against ML Services With LLM-Based Agents. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.572

@inproceedings{wu2026inferpilot,
  title = {{InferPilot: Autonomous Inference Attacks Against ML Services With LLM-Based Agents}},
  author = {Yixin Wu, Rui Wen, Chi Cui, Michael Backes and Yang Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.572},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.572}
}

References

64 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sahar Abdelnabi, Kai Greshake, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In Workshop on Security and Artificial Intelligence (AISec), pages 79– 90. ACM.
  2. Alex Krizhevsky and Vinod Nair and Geoffrey Hinton. 2009. CIFAR. https://www.cs.toronto. edu/˜kriz/cifar.html. ↗
  3. AutoGPT. 2023. An open-source autonomous software agent. https://agpt.co/. ↗
  4. Islem Bouzenia, Premkumar T. Devanbu, and Michael Pradel. 2024. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair. CoRR abs/2403.17134. — [arXiv:2403.17134](https://arxiv.org/abs/2403.17134) find
  5. Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor Zhong, Lu Chen, Kai Yu, and Tao Yu. 2024. Spi… — [arXiv:2407.10956](https://arxiv.org/abs/2407.10956) find
  6. Nicholas Carlini, Matthew Jagielski, and Ilya Mironov. 2020. Cryptanalytic Extraction of Neural Network Models. In Annual International Cryptology Conference (CRYPTO), pages 189–218. Springer. find
  7. Nicholas Carlini, Javier Rando, Edoardo Debenedetti, Milad Nasr, and Florian Tramèr. 2025. AutoAd-vExBench: Benchmarking Autonomous Exploitation of Adversarial Example Defenses. In International Conference on Machine Learning (ICML). PMLR. find
  8. Nicholas Carlini and David Wagner. 2017. Towards Evaluating the Robustness of Neural Networks. In IEEE Symposium on Security and Privacy (S&P), pages 39–57. IEEE. find
  9. François Chollet. 2017. Xception: Deep Learning with Depthwise Separable Convolutions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1800–1807. IEEE. find
  10. Christopher A. Choquette Choo, Florian Tramèr, Nicholas Carlini, and Nicolas Papernot. 2021. Label-Only Membership Inference Attacks. In International Conference on Machine Learning (ICML), pages 1964–1974. PMLR. find
  11. Adam Coates, Andrew Y. Ng, and Honglak Lee. 2011. An Analysis of Single-Layer Networks in Unsupervised Feature Learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 215–223. JMLR. find
  12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770– 778. IEEE.
  13. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2024. MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. In International Conference on Machine Learning (ICML).
  14. Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot. 2020. High Accuracy and High Fidelity Extraction of Neural Networks. In USENIX Security Symposium (USENIX Security), pages 1345–1362. USENIX. find
  15. Emiliano De Cristofaro. 2020. An Overview of Privacy in Machine Learning. CoRR abs/2005.08679. — [arXiv:2005.08679](https://arxiv.org/abs/2005.08679) find
  16. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimha. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues? In International Conference on Learning Representations (ICLR).
  17. Vasisht Duddu, Sebastian Szyller, and N. Asokan. 2024. SoK: Unintended Interactions among Machine Learning Defenses and Risks. In IEEE Symposium on Security and Privacy (S&P), pages 2996– 3014. IEEE. find
  18. Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, and Mohit Iyyer. 2020. Thieves on Sesame Street! Model Extraction of BERT-based APIs. In International Conference on Learning Representations (ICLR). find
  19. Flask. 2010. A lightweight WSGI web application framework. https://flask.palletsprojects.com/en/stable/. ↗
  20. Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial Examples in the Physical World. CoRR abs/1607.02533. — [arXiv:1607.02533](https://arxiv.org/abs/1607.02533) find
  21. Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. 2015. Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 1322–1333. ACM. find
  22. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, Daniel Fried, Sida I. Wang, and Tao Y. 2023. DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation. In International Conference on Machine Learning (ICML), pages 18319– 18345. PMLR. find
  23. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations (ICLR).
  24. Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. 2017. DBpedia-Entity v2: A Test Collection for Entity Search. In International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 1265– 1268.… find
  25. Jamie Hayes, Ilia Shumailov, Christopher A. Choquette-Choo, Matthew Jagielski, Georgios Kaissis, Katherine Lee, Milad Nasr, Sahra Ghalebikesabi, Niloofar Mireshghallah, Meenatchi Sundaram Mutu Selva Annamalai, Igor Shilov, Matthieu Meeus, Yves-Alexandre de Montjoye, Franziska Boenisch, Adam Dziedzic… — [arXiv:2505.18773](https://arxiv.org/abs/2505.18773) find
  26. LangChain. 2022. com/. https://www.langchain. ↗
  27. Zheng Li and Yang Zhang. 2021. Membership Leakage in Label-Only Exposures. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 880–895. ACM. find
  28. Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lo. 2024a. Large Language Model-Based Agents for Software Engineering: A Survey. CoRR abs/2409.02977. — [arXiv:2409.02977](https://arxiv.org/abs/2409.02977) find
  29. Yiyong Liu, Zhengyu Zhao, Michael Backes, and Yang Zhang. 2022a. Membership Inference Attacks by Exploiting Loss Trajectory. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 2085–2098. ACM. find
  30. Yugeng Liu, Rui Wen, Xinlei He, Ahmed Salem, Zhikun Zhang, Michael Backes, Emiliano De Cristofaro, Mario Fritz, and Yang Zhang. 2022b. find
  31. ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models. In USENIX Security Symposium (USENIX Security), pages 4525–4542. USENIX. find
  32. Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024b. Formalizing and Benchmarking Prompt Injection Attacks and Defenses. In USENIX Security Symposium (USENIX Security). USENIX.
  33. Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep Learning Face Attributes in the Wild. In IEEE International Conference on Computer Vision (ICCV), pages 3730–3738. IEEE. find
  34. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. CoRR abs/2408.06292.
  35. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on Learning Representations (ICLR).
  36. Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. 2019. Exploiting Unintended Feature Leakage in Collaborative Learning. In IEEE Symposium on Security and Privacy (S&P), pages 497–512. IEEE. find
  37. Meta. 2023. https://huggingface.co/ meta-llama/Llama-2-7b-chat. ↗
  38. Mistral AI. 2023. https://huggingface.co/ mistralai/Mistral-7B-Instruct-v0.2. ↗
  39. Milad Nasr, Reza Shokri, and Amir Houmansadr. 2019. Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning. In IEEE Symposium on Security and Privacy (S&P), pages 1021–1035. IEEE. find
  40. Tuan Anh Nguyen and Anh Tuan Tran. 2021. WaNet - Imperceptible Warping-based Backdoor Attack. In International Conference on Learning Representations (ICLR). find
  41. Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. 2016. Ordinal Regression with Multiple Output CNN for Age Estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4920–4928. IEEE. find
  42. OpenAI. 2025. https://platform.openai.com/docs/pricing. ↗
  43. Maria Rigaki and Sebastián Garcı́a. 2024. A Survey of Privacy Attacks in Machine Learning. ACM Computing Surveys. find
  44. Ahmed Salem, Andrew Paverd, and Boris Köpf. 2023. Maatphor: Automated Variant Analysis for Prompt Injection Attacks. CoRR abs/2312.11513. — [arXiv:2312.11513](https://arxiv.org/abs/2312.11513) find
  45. Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. 2019. ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models. In Network and Distributed System Security Symposium (NDSS). Internet Society. find
  46. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership Inference Attacks Against Machine Learning Models. In IEEE Symposium on Security and Privacy (S&P), pages 3–18. IEEE.
  47. Congzheng Song and Ananth Raghunathan. 2020. Information Leakage in Embedding Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 377–390. ACM. find
  48. Congzheng Song and Vitaly Shmatikov. 2020. Overlearning Reveals Sensitive Attributes. In International Conference on Learning Representations (ICLR). find
  49. Liwei Song and Prateek Mittal. 2021. Systematic Evaluation of Privacy Risks of Machine Learning Models. In USENIX Security Symposium (USENIX Security). USENIX. find
  50. The Vicuna Team. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* Chat-GPT Quality. https://lmsys.org/blog/ 2023-03-30-vicuna/. ↗
  51. Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing Machine Learning Models via Prediction APIs. In USENIX Security Symposium (USENIX Security), pages 601–618. USENIX. find
  52. Rui Wen, Michael Backes, and Yang Zhang. 2025. Understanding Data Importance in Machine Learning Attacks: Does Valuable Data Pose Greater Harm? In Network and Distributed System Security Symposium (NDSS). Internet Society. find
  53. Rui Wen, Zheng Li, Michael Backes, and Yang Zhang. 2024. Membership Inference Attacks Against In-Context Learning. In ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM. find
  54. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying LLM-based Software Engineering Agents. CoRR abs/2407.01489.
  55. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in…
  56. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Pres. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. CoRR abs/2405.15793.
  57. Ziqi Yang, Jiyi Zhang, Ee-Chien Chang, and Zhenkai Liang. 2019. Neural Network Inversion in Adversarial Setting via Background Knowledge Alignment. In ACM SIGSAC Conference on Computer and Communications Security (CCS), page 225–240. ACM. find
  58. Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. 2022. Enhanced Membership Inference Attacks against Machine Learning Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 3093–3106. ACM. find
  59. Hongxu Yin, Pavlo Molchanov, Jose M. Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. 2020. Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8712–8721. IEEE. find
  60. Andy K. Zhang, Neil Perry, Riya Dulepet, Eliot Jones, Justin W. Lin, Joey Ji, Celeste Menders, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askar-yar, Mike Yang, Teddy Zhang, Rishi Alluri, Nathan Tran, Rinna… find
  61. Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. 2020. The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 250–258. IEEE. find
  62. Zhifei Zhang, Yang Song, and Hairong Qi. 2017. Age Progression/Regression by Conditional Adversarial Autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4352– 4360. IEEE. find
  63. Nan Zhong, Zhenxing Qian, and Xinpeng Zhang. 2022. Imperceptible Backdoor Attack: From Input Space to Feature Representation. In International Joint Conferences on Artifical Intelligence (IJCAI), pages 1736–1742. IJCAI. find
  64. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2023. WebArena: A Realistic Web Environment for Building Autonomous Agents. CoRR abs/2307.13854.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-inferpilot-autonomous-inference-attacks-against/