Elamur Research⌕
☰
home / papers / 2026 / The Challenge of Identifying the Origin of Black-Box Large Language Models

The Challenge of Identifying the Origin of Black-Box Large Language Models

Paper · 2026 · Ziqing Yang, Yixin Wu, Yun Shen et al.

Summary

The paper tackles verifying whether a black-box LLM, such as a chat service or API reachable only through queries, was derived from a model the defender controls. The derived model may have been fine-tuned, re-prompted or rebranded. Existing approaches have drawbacks. Backdoor-style model watermarking retrains the network, is costly, and tends to vanish after further fine-tuning. Query-based fingerprinting depends on discrete prompt search and generalizes poorly to derivatives. The authors propose PlugAE, which works in two stages. First, it optimizes a short sequence of adversarial token embeddings against a chosen query set while the LLM stays frozen. Second, it binds those embeddings to custom "copyright tokens" inside the token-embedding layer, leaving the transformer weights unchanged. To check a suspect model, the defender sends the queries together with the copyright tokens and looks for the target responses. Tests covered three candidate models and 32 suspects. PlugAE beat the watermarking baseline IF and the fingerprinting baselines TRAP and ProFlingo, with the largest gains on fine-tuned derivatives. This makes it a cheap and practical tool for auditing licence misuse.

Key points

  • The experiments checked 32 suspect models, made up of 30 open LLMs and two real-world LLM APIs, against three candidate LLMs. All datasets were in English.
  • On a further fine-tuned Llama, the Llama-specific PlugAE reached a TRR of 0.98, while IF scored 0.00. For detecting Llama, its AUC was 1.00 versus 0.67 for TRAP.
  • Compared with backdoor-based watermarking, the number of trainable parameters falls from about 7B to roughly 4K. The authors also give a theoretical argument that continuous embedding optimization beats discrete prompt search.
  • The choice of copyright token did not change results. The tested tokens were a random 5-character string (the default), a 20-character string, and "cf". Results also held across different chat templates and five system-prompt styles.
  • Using more adversarial tokens generally raised TRR on derivatives, for example from 0.50 to 0.94 on SciTulu, though too many tokens can hurt it (Orca2). Non-derivatives always stayed at a TRR of 0. A stated limitation is that the model must still be modified, so the plugged embeddings could be detected or removed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Helmholtz Center for Information SecuritySaarbrücken · Germany20 works · 1,172 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksHardware and Architecture · 167 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Ziqing Yang, Yixin Wu, Yun Shen, Wei Dai, Michael Backes and Yang Zhang. 2026. The Challenge of Identifying the Origin of Black-Box Large Language Models. Proceedings of the Seventh Workshop on Privacy in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.privatenlp-main.2

@inproceedings{yang2026challenge,
  title = {{The Challenge of Identifying the Origin of Black-Box Large Language Models}},
  author = {Ziqing Yang, Yixin Wu, Yun Shen, Wei Dai, Michael Backes and Yang Zhang},
  booktitle = {Proceedings of the Seventh Workshop on Privacy in Natural Language Processing},
  year = {2026},
  doi = {10.18653/v1/2026.privatenlp-main.2},
  url = {https://doi.org/10.18653/v1/2026.privatenlp-main.2}
}

References

41 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, and 24 others.… find
  2. Chenxi Gu, Chengsong Huang, Xiaoqing Zheng, Kai-Wei Chang, and Cho-Jui Hsieh. 2022. Watermarking Pre-trained Language Models with Backdooring. CoRR abs/2210.07543. 2024. https://openai.com/policies/ row-terms-of-use/. 2024. 1689. https://eur-lex.europa.eu/eli/reg/2024/ — [arXiv:2210.07543](https://arxiv.org/abs/2210.07543) ↗
  3. Martin Gubri, Dennis Ulmer, Hwaran Lee, Sangdoo Yun, and Seong Joon Oh. 2024. TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification. In Annual Meeting of the Association for Computational Linguistics (ACL). find
  4. ACL. 2025. https://gptstore.ai/.2025. business-terms/. https://https://openai.com/policies/ ↗
  5. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations (ICLR).
  6. Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacche… find
  7. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR).
  8. Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andrés Codas, Clarisse Simões, Sahaj Agrawal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah. 2023. Orca 2: Teaching small language models how to reason.…
  9. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. CoRR abs/1909.09436.
  10. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew E. Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2. CoRR abs/2311.10702.
  11. GPT-4 Technical Report. CoRR OpenAI. 2023b. Introducing GPTs. https://openai.com/ blog/introducing-gpts. ↗
  12. PromptDB. 2025. https://promptdb.param.codes/. ↗
  13. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, élio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  14. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, and 6 o… — Code Llama: Open Foundation Models for Code
  15. Heng Jin, Chaoyu Zhang, Shanghao Shi, Wenjing Lou, and Y. Thomas Hou. 2024. ProFLingo: A Fingerprinting-based Copyright Protection Scheme for Large Language Models. CoRR abs/2405.02466. — [arXiv:2405.02466](https://arxiv.org/abs/2405.02466) find
  16. Jessica Rumbelow and Matthew Watkins. 2023. Solidgoldmagikarp (plus, prompt generation). https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/ solidgoldmagikarp-plus-prompt-generation. ↗
  17. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A Watermark for Large Language Models. In International Conference on Machine Learning (ICML), pages 17061–17084. PMLR.
  18. Fabian M. Suchanek, David Gross-Amblard, and Serge Abiteboul. 2011. Watermarking for ontologies. In The Semantic Web - ISWC 2011 - 10th International Semantic Web Conference, Bonn, Germany, October 23-27, 2011, Proceedings, Part I, volume 7031 of Lecture Notes in Computer Science, pages 697–713. Spr… find
  19. Sander Land and Max Bartolo. 2024. Fishing for magikarp: Automatically detecting under-trained tokens in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 11631–11646. Association…
  20. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. https://crfm.stanford.edu/2023/ 03/13/alpaca.html. ↗
  21. Peixuan Li, Pengzhou Cheng, Fangqi Li, Wei Du, Haodong Zhao, and Gongshen Liu. 2023. PLMmark: A Secure and Robust Black-Box Watermarking Framework for Pre-trained Language Models. In AAAI Conference on Artificial Intelligence (AAAI), pages 14991–14999. AAAI. find
  22. The Vicuna Team. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https://lmsys.org/blog/2023-03-30-vicuna/. ↗
  23. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. LLaMA: Open and Efficient Foundation Language Models. CoRR ab…
  24. Aiwei Liu, Leyi Pan, Xuming Hu, Shiao Meng, and Lijie Wen. 2024. A Semantic Invariant Robust Watermark for Large Language Models. In International Conference on Learning Representations (ICLR). ICLR. find
  25. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS.
  26. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  27. Hope McGovern, Rickard Stureborg, Yoshi Suhara, and Dimitris Alikaniotis. 2024. Your Large Language Models Are Leaving Fingerprints. CoRR abs/2405.14057. find
  28. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023. Zephyr: Direct distillation of LM alignment. CoRR, abs/2310.… find
  29. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. CoRR abs/2307.15043.
  30. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. 2023. CogVLM: Visual Expert for Pretrained Language Models. CoRR abs/2311.03079.
  31. Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. 2024. Instructional Fingerprinting of Large Language Models. CoRR abs/2401.12255. find
  32. Zhenhua Xu, Xubin Yue, Zhebo Wang, Qichen Liu, Xixiang Zhao, Jingxuan Zhang, Wenjun Zeng, Wengpeng Xing, Dezhang Kong, Changting Lin, and Meng Han. 2025. Copyright protection for large language models: A survey of methods, challenges, and trends. CoRR, abs/2508.11548. — [arXiv:2508.11548](https://arxiv.org/abs/2508.11548) find
  33. Ziqing Yang, Michael Backes, Yang Zhang, and Ahmed Salem. 2024. SOS! Soft Prompt Attack Against Open-Source Large Language Models. CoRR abs/2407.03160. — [arXiv:2407.03160](https://arxiv.org/abs/2407.03160) find
  34. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, and 4 others. 2024. MiniCPM-V: A GPT-4V Level MLLM on Your Phone. CoRR abs/240…
  35. Peigen Ye, Huali Ren, Zhengdao Li, Anli Yan, Hongyang Yan, Shaowei Wang, and Jin Li. 2026. Securing large language models: A survey of watermarking and fingerprinting techniques. ACM Comput. Surv., 58(7). find
  36. KiYoon Yoo, Wonhyuk Ahn, Jiho Jang, and Nojun Kwak. 2023. Robust multi-bit natural language watermarking through invariant features. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 2… find
  37. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4791–4800. ACL. find
  38. Boyi Zeng, Chenghu Zhou, Xinbing Wang, and Zhouhan Lin. 2023. HuRef: HUman-REadable Fingerprint for Large Language Models. CoRR abs/2312.04828. — [arXiv:2312.04828](https://arxiv.org/abs/2312.04828) find
  39. Xuandong Zhao, Prabhanjan Vijendra Ananth, Lei Li, and Yu-Xiang Wang. 2024. Provable Robust Watermarking for AI-Generated Text. In International Conference on Learning Representations (ICLR). ICLR.
  40. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Annual Conference on Neural Information Processing Syst…
  41. In the appendix, we provide details and results omitted in the main text. by obscuring or removing copyright and trademark information. • Appendix A: LLM licenses and agreements of six mainstream LLM organizations on the market, including OpenAI, Anthropic, Meta, Ai2, Google, and Mistral AI. B Theor… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-challenge-identifying-origin-black-box/