Elamur Research⌕
☰
home / papers / 2026 / Neural Breadcrumbs: Membership Inference Attacks on LLMs Through Hidden State and Attention Pattern Analysis

Neural Breadcrumbs: Membership Inference Attacks on LLMs Through Hidden State and Attention Pattern Analysis

Paper · 2026 · Disha Makhija, Manoj Ghuhan Arivazhagan, Vinayshekhar Bannihatti Kumar et al.

Summary

Several recent studies report that membership inference attacks (MIAs) on large language models barely beat random guessing. This has suggested that pre-training on huge corpora leaves little detectable privacy leakage. Makhija et al. (AWS AI Labs) argue that the weakness lies in where existing attacks look. Most attacks inspect only final-layer outputs, such as loss or next-token probabilities.

Their framework, memTrace, works with white-box or grey-box access. It extracts features from hidden states, attention maps and per-layer logits while a decoder-only LLM processes a candidate text. Examples include entropy, confidence statistics, attention concentration and sparsity, and cross-layer transition patterns. These features are aggregated into a fixed-length vector per sequence, and lightweight classifiers such as random forests or single-layer MLPs then predict membership. The target LLM itself is never fine-tuned.

Across several model families, memTrace reaches an average AUC of 0.85 on popular MIA benchmarks. The authors conclude that membership traces persist inside models even when outputs look safe. This matters for privacy auditing, for verifying machine unlearning, and for designing defenses.

Key points

  • Core idea: The authors call internal processing traces "neural breadcrumbs." They argue these traces carry membership signal that output-based, loss-style attacks overlook. They also report that deeper layers hold increasingly strong signals.
  • Confidence and entropy features:
    • per-token entropy;
    • top-token confidence;
    • the gap between the two most likely tokens;
    • confidence stability, computed from the ratio of mean to standard deviation.
  • Attention features:
    • head-averaged attention entropy;
    • attention concentration (the mean of each row's maximum weight);
    • sparsity measured against an adaptive threshold.
  • Classifier and results: A random forest is used as an interpretable classifier, and the authors report an average AUC of 0.85. Code is released at github.com/amazon-science/NeuralBreadcrumbs-MIA-EACL-2026.
  • Limitations: The method needs access to model internals. It was tested only on decoder-only transformers, and the effects of fine-tuning on these signatures remain largely unexplored.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AmazonSeattle · United States112 works · 40,565 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Disha Makhija, Manoj Ghuhan Arivazhagan, Vinayshekhar Bannihatti Kumar and Rashmi Gangadharaiah. 2026. Neural Breadcrumbs: Membership Inference Attacks on LLMs Through Hidden State and Attention Pattern Analysis. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.262

@inproceedings{makhija2026neural,
  title = {{Neural Breadcrumbs: Membership Inference Attacks on LLMs Through Hidden State and Attention Pattern Analysis}},
  author = {Disha Makhija, Manoj Ghuhan Arivazhagan, Vinayshekhar Bannihatti Kumar and Rashmi Gangadharaiah},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.262},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.262}
}

References

25 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah, and Dan Roth. 2023. Privacy adhering machine un-learning in NLP. In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings), pages 268–277. Association for Computational Linguistics. find
  2. Martin Andres Bertran, Shuai Tang, Aaron Roth, Michael Kearns, Jamie Heather Morgenstern, and Steven Wu. 2023. Scalable membership inference attacks via quantile regression. In Thirty-seventh Conference on Neural Information Processing Systems. find
  3. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. Pythia: a suite for analyzing large language models across trainin… find
  4. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Mesh-Tensorflow Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. find
  5. Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. 2022. Membership inference attacks from first principles. Preprint, arXiv:2112.03570. find
  6. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others. 2021. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650…
  7. Debeshee Das, Jie Zhang, and Florian Tramèr. 2025. Blind baselines beat membership inference attacks for foundation models. find
  8. Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. Do membership inference attacks work on large language models? In Conference on Language Modeling (COLM).
  9. Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. 2024. Membership inference attacks against fine-tuned large language models via self-prompt calibration. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  10. Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. 2025. MIA-tuner: Adapting large language models as pre-training text detector. In Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, Pennsylvania, USA. find
  11. Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. 2023. Analyzing leakage of personally identifiable information in language models. In 2023 IEEE Symposium on Security and Privacy (SP), pages 346–363. IEEE Computer Society. find
  12. Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. 2024. LLM dataset inference: Did you train on my dataset? In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  13. Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. 2023. Membership inference attacks against language models via neighbourhood comparison. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11330– 11…
  14. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. Detectgpt: zero-shot machine-generated text detection using probability curvature. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
  15. Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. 2024. Proving test set contamination in black-box language models. In The Twelfth International Conference on Learning Representations. find
  16. Haritz Puerto, Martin Gubri, Sangdoo Yun, and Seong Joon Oh. 2024. Scaling up membership inference: When and how attacks succeed on large language models. Preprint, arXiv:2411.00154. find
  17. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations. find
  18. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint…
  19. Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Zhenqiang Gong, and Bhuwan Dhingra. 2024. ReCaLL: Membership inference via relative conditional log-likelihoods. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8671– 8689, Mia… find
  20. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. Preprint, arXiv:1709.01604.
  21. Sajjad Zarifzadeh, Philippe Cheng-Jie Marc Liu, and Reza Shokri. 2024. Low-cost high-power membership inference by boosting relativity. find
  22. Hengxiang Zhang, Songxin Zhang, Bingyi Jing, and Hongxin Wei. 2025a. Fine-tuning can help detect pretraining data from large language models. In The Thirteenth International Conference on Learning Representations. find
  23. Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. 2024. Min-k%++: Improved baseline for detecting pre-training data from large language models. arXiv preprint arXiv:2404.02936. — [arXiv:2404.02936](https://arxiv.org/abs/2404.02936) find
  24. Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. 2025b. Min-k%++: Improved baseline for pre-training data detection from large language models. In The Thirteenth International Conference on Learning Representations. find
  25. Supplement for “Neural Breadcrumbs: Membership Inference Attacks on LLMs Through Hidden State and Attention Pattern Analysis” find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/makhija-2026-neural-breadcrumbs-membership-inference-attacks/