Elamur Research⌕
☰
home / papers / 2026 / Privacy Risks of Intermediate Representations: Attribute Inference in Distributed LLM Inference

Privacy Risks of Intermediate Representations: Attribute Inference in Distributed LLM Inference

Paper · 2026 · Yang Lyu, Jin Cao, Yang Xiao et al.

Summary

Distributed LLM inference splits a model across several participants. They exchange hidden states rather than raw prompts, and this is widely treated as privacy-preserving. The authors test that belief. They ask whether a single honest-but-curious node can identify which sensitive attribute, such as a drug name or a location, appears in a user's prompt. This node holds only a slice of layers and sees only the activations passed to it. It has no labelled embedding–attribute data, so the authors recast the task as zero-shot matching against a predefined candidate list inside the intermediate representation space.

Their attack, IR-AIA, has two parts:

  • SG-APCR, an unsupervised step that removes dominant principal components adaptively, guided by how spherical the embedding space becomes.
  • Sliding-window maximum similarity matching, which recovers word-level signals from words split into subword tokens.

Across three LLMs and three real-world datasets, the attack infers attributes with near-perfect Top-1 accuracy at multiple layers. The results show that sharing hidden states does not reliably hide sensitive inputs. They also suggest defenses need to go beyond simple noise injection.

Key points

  • Threat model: one honest-but-curious participant with a subset of layers, no access to raw input, no auxiliary datasets, and no collusion. It can query the model only to compute representations of the candidate sensitive words.
  • The method addresses two obstacles: layer-dependent anisotropy, handled by SG-APCR, and subword fragmentation, handled by window-based max cosine similarity.
  • The highest reported Top-1 accuracy reaches 0.997 for CMS, with Skytrax at 0.980 and ECHR at 0.986.
  • Leakage is not confined to particular depths; it persists across several evaluated layers and model families.
  • Stated limitations: closed candidate sets rather than open-vocabulary recovery, no larger or multimodal models, and only a preliminary defense case study with DP-Forward.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🔬Institute of Information EngineeringBeijing · China30 works · 1,411 citations🎓Xidian UniversityXi'an · China14 works · 1,357 citations🎓Guangzhou UniversityGuangzhou · China4 works · 7 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yang Lyu, Jin Cao, Yang Xiao, Zhe Sun, Ben Niu, Fenghua Li and Hui LI. 2026. Privacy Risks of Intermediate Representations: Attribute Inference in Distributed LLM Inference. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1172

@inproceedings{lyu2026privacy,
  title = {{Privacy Risks of Intermediate Representations: Attribute Inference in Distributed LLM Inference}},
  author = {Yang Lyu, Jin Cao, Yang Xiao, Zhe Sun, Ben Niu, Fenghua Li and Hui LI},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1172},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1172}
}

References

29 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. 2023a. Petals: Collaborative inference and fine-tuning of large models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics… find
  2. Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, and Colin A Raffel. 2023b. Distributed inference and fine-tuning of large language models over the internet. Advances in neural information processing systems, 36:12312– 12331. find
  3. Kritya Bunchongchit and Walanchalee Wattanacharoensil. 2021. Data analytics of skytrax’s airport review and ratings: Views of airport quality by passengers types. Research in Transportation Business & Management, 41:100688. find
  4. Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural legal judgment prediction in english. arXiv preprint arXiv:1906.02059.
  5. Guanzhong Chen, Zhenghan Qin, Mingxin Yang, Yajie Zhou, Tao Fan, Tianyu Du, and Zenglin Xu. 2024. Unveiling the vulnerability of private fine-tuning in split-based frameworks for large language models: A bidirectionally enhanced attack. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer… find
  6. Minxin Du, Xiang Yue, Sherman SM Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. 2023. Dp-forward: Fine-tuning and inference on language models with differential privacy in forward pass. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 2665– 2679. find
  7. Wei Fan, Haoran Li, Zheye Deng, Weiqi Wang, and Yangqiu Song. 2024. Goldcoin: Grounding large language models in privacy laws via contextual integrity theory. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3321–3343. find
  8. Kang Gu, Ehsanul Kabir, Neha Ramsurrun, Soroush Vosoughi, and Shagufta Mehnaz. 2023. Towards sentence level inference attack against pre-trained language models. Proceedings on Privacy Enhancing Technologies. find
  9. Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. 2023. Medalpaca–an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247.
  10. Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao, Weilin Zhao, Ao Sun, Hao Zhou, Jie Zhou, Zhiyuan Liu, and Maosong Sun. 2025. APB: Accelerating distributed long-context inference by passing compressed context blocks across GPUs. In Proceedings of the 63rd Annual Meeting of the Association for Computa… find
  11. Bargav Jayaraman and David Evans. 2022. Are attribute inference attacks just imputation? In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 1569–1582. find
  12. Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi Chen, and Binhang Yuan. 2024. Hexgen: Generative inference of large language model over heterogeneous environment. In International Conference on Machine Learning, pages 21946–21961. PMLR. find
  13. Nikhil Kandpal, Krishna Pillutla, Alina Oprea, Peter Kairouz, Christopher A Choquette-Choo, and Zheng Xu. 2024. User inference attacks on large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18238–18265. find
  14. Jabari Kwesi, Jiaxun Cao, Riya Manchanda, and Pardis Emami-Naeini. 2025. Exploring user security and privacy attitudes and concerns toward the use of {General-Purpose}{LLM} chatbots for mental health. In 34th USENIX Security Symposium (USENIX Security 25), pages 6007–6024. find
  15. Ziang Li, Mengda Yang, Yaxin Liu, Juan Wang, Hongxin Hu, Wenzhe Yi, and Xiaoyang Xu. 2023. Gan you see me? enhanced data reconstruction attacks against split inference. Advances in neural information processing systems, 36:54554–54566. find
  16. Zihan Liu, Yizhen Wang, Rui Wang, and Sai Wu. 2025. Dualguard: A parameter space transformation approach for bidirectional defense in split-based llm fine-tuning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17065–17080. find
  17. Xinjian Luo, Ting Yu, and Xiaokui Xiao. 2025. Prompt inference attack on distributed large language model inference frameworks. arXiv preprint arXiv:2503.09291. — [arXiv:2503.09291](https://arxiv.org/abs/2503.09291) find
  18. Anemily Machina and Robert Mercer. 2024. Anisotropy is not inherent to transformers. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4892–4907. find
  19. Jiaqi Mu, Suma Bhat, and Pramod Viswanath. 2017. All-but-the-top: Simple and effective postprocessing for word representations. arXiv preprint arXiv:1702.01417. find
  20. Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. 2020. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP), pages 1314–1331. IEEE. find
  21. Wenjie Qu, Yuguang Zhou, Yongji Wu, Tingsong Xiao, Binhang Yuan, Yiming Li, and Jiaheng Zhang. 2025. Prompt inversion attack against collaborative inference of large language models. In 2025 IEEE Symposium on Security and Privacy (SP), pages 1695–1712. IEEE. find
  22. Federico Ravenda, Seyed Ali Bahrainian, Andrea Raballo, Antonietta Mira, and Noriko Kando. 2025. Are LLMs effective psychological assessors? leveraging adaptive RAG for interpretable mental health screening through psychometric practice. In Proceedings of the 63rd Annual Meeting of the Association f… find
  23. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
  24. Yashothara Shanmugarasa, Ming Ding, Chamikara Mahawaga Arachchige, and Thierry Rakotoarivelo. 2025. Sok: The privacy paradox of large language models: Advancements, privacy risks, and mitigation. In Proceedings of the 20th ACM Asia Conference on Computer and Communications Security, pages 425– 441. find
  25. Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. 2025. Layer by layer: Uncovering hidden representations in language models. arXiv preprint arXiv:2502.02013. find
  26. Congzheng Song and Ananth Raghunathan. 2020. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 377– 390. find
  27. Robin Staab, Mark Vero, Mislav Balunović, and Martin Vechev. 2023. Beyond memorization: Violating privacy via inference with large language models. arXiv preprint arXiv:2310.07298. — [arXiv:2310.07298](https://arxiv.org/abs/2310.07298) find
  28. Adam Sutton, Xi Bai, Kawsar Noor, Thomas Searle, and Richard Dobson. 2025. Named entity inference attacks on clinical llms: Exploring privacy risks and the impact of mitigation strategies. In Proceedings of the Sixth Workshop on Privacy in Natural Language Processing, pages 42–52. find
  29. Yuecen Wei, Xingcheng Fu, Lingyun Liu, Qingyun Sun, Hao Peng, and Chunming Hu. 2025. Prompt-based unifying inference attack on graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 12836–12844. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lyu-2026-privacy-risks-intermediate-representations-attribute/