Elamur Research⌕
☰
home / papers / 2026 / The Shape of Vulnerability: How Adversarial Perturbations Reshape the Topology of Language Model Latent Spaces

The Shape of Vulnerability: How Adversarial Perturbations Reshape the Topology of Language Model Latent Spaces

Paper · 2026 · Angelina Tsai, Shreya Subramanian, Catherine Liu et al.

Summary

Adversarial prompts can push LLMs into degraded or harmful behaviour. Most interpretability tools look for linear features in isolated components, so they reveal little about how such inputs reshape a model's internal geometry. This ACL 2026 Student Research Workshop paper applies topological data analysis, specifically persistent homology, to attention weights from transformer models.

The authors study two black-box attack families:

  • Sandbagging prefixes, which are prepended to make a model underperform.
  • Code injection, which hides malicious instructions inside a prompt.

Attention maps from the middle (6th) and final (12th) layers of encoder and decoder models are converted into persistence diagrams. Clean and attacked inputs are then compared using bottleneck distance and correlations with linguistic features.

Attacked inputs leave distinguishable traces, most visibly in higher-dimensional features (H1 loops, H2 voids). Their topological features also tend to be shorter-lived, a pattern the authors call topological compression. The work points toward topology-based detectors of manipulated inputs and tools for judging model trustworthiness.

Key points

  • Models analysed include BERT, RoBERTa, ELECTRA, DistilGPT and CodeBERT-base. All are small: RoBERTa-base and CodeBERT-base have about 125M parameters each.
  • The features tracked are H0 (connected components), H1 (loops) and H2 (voids). Clean and adversarial persistence diagrams are compared with bottleneck distance.
  • The authors correlated linguistic and topological features and checked how these correlations changed under sandbagging:
    • 14 feature pairs changed by more than 0.1: 9 weakened and 5 strengthened.
    • 115 pairs showed negligible change (|∆p| < 0.05).
  • Max_0dim_Minus_Second measures how far the largest H0 component stands apart from the second largest. Under sandbagging it:
    • lost its link to RIX readability (∆p = +0.645) and median sentence length (+0.512);
    • gained a link to mean dependency distance (∆p = −0.452).
  • The authors list several limitations:
    • computation grows cubically with prompt length;
    • only attention weights are analysed, not hidden states;
    • only two attack types are covered;
    • specific topological features are hard to map onto semantics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harvey Mudd CollegeClaremont · United States3 works · 21 citations🔬MIT Lincoln LaboratoryLexington · United States4 works

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Angelina Tsai, Shreya Subramanian, Catherine Liu, Kimberly Lopez, Leif Zinn-Brooks, Alexia E. Schulz and Adaku Uchendu. 2026. The Shape of Vulnerability: How Adversarial Perturbations Reshape the Topology of Language Model Latent Spaces. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.24

@inproceedings{tsai2026shape,
  title = {{The Shape of Vulnerability: How Adversarial Perturbations Reshape the Topology of Language Model Latent Spaces}},
  author = {Angelina Tsai, Shreya Subramanian, Catherine Liu, Kimberly Lopez, Leif Zinn-Brooks, Alexia E. Schulz and Adaku Uchendu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.24},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.24}
}

References

45 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Henry Adams, Tegan Emerson, Michael Kirby, Rachel Neville, Chris Peterson, Patrick Shipman, Sofya Chepushtanova, Eric Hanson, Francis Motta, and Lori Ziegelmeier. 2017. Persistence images: A stable vector representation of persistent homology. Journal of Machine Learning Research, 18(8):1–35. find
  2. Shashank Agnihotri, Jonas Jakubassa, Priyam Dey, Sachin Goyal, Bernt Schiele, Venkatesh Babu Radhakrishnan, and Margret Keuper. 2025. A granular study of safety pretraining under model abliteration. arXiv preprint arXiv:2510.02768. — [arXiv:2510.02768](https://arxiv.org/abs/2510.02768) find
  3. Anonymous. 2026. Bridging topology and linguistics: Investigating the linguistic interpretation of topological features in text. find
  4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. Preprint, arXiv:2108.07732.
  5. Luis Balderas, Miguel Lastra, and José M Benítez. 2025. A green ai methodology based on persistent homology for compressing bert. Applied Sciences, 15(1):390. find
  6. Ulrich Bauer. 2021. Ripser: efficient computation of vietoris–rips persistence barcodes. Journal of Applied and Computational Topology, 5(3):391–423. find
  7. Reza Bayat, Ali Rahimi-Kalahroudi, Mohammad Pezeshki, Sarath Chandar, and Pascal Vincent. 2025. Steering large language model activations in sparse spaces. arXiv preprint arXiv:2503.00177. — [arXiv:2503.00177](https://arxiv.org/abs/2503.00177) find
  8. Gunnar Carlsson and Mikael Vejdemo-Johansson. 2021. Topological data analysis with applications. Cambridge University Press. find
  9. Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. 2021. A survey on adversarial attacks and defences. CAAI Transactions on Intelligence Technology, 6(1):25–45. find
  10. Frédéric Chazal and Bertrand Michel. 2021. An introduction to topological data analysis: Fundamental and practical aspects for data scientists. Frontiers in Artificial Intelligence, 4:667963. find
  11. David Cohen-Steiner, Herbert Edelsbrunner, and John Harer. 2005. Stability of persistence diagrams. In Proceedings of the twenty-first annual symposium on Computational geometry, pages 263–271. find
  12. Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. 2025. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 57(6):1–39.
  13. Aideen Fay, Ines Garcia-Redondo, Qiquan Wang, Haim Dubossarsky, and Anthea Monod. 2025. The shape of adversarial influence: Characterizing llm latent spaces with persistent homology. arXiv preprint arXiv:2505.20435. — [arXiv:2505.20435](https://arxiv.org/abs/2505.20435) find
  14. Haibo Jin, Leyang Hu, Xinnuo Li, Peiyan Zhang, Chonghan Chen, Jun Zhuang, and Haohan Wang. 2024. Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models. arXiv preprint arXiv:2407.01599. — [arXiv:2407.01599](https://arxiv.org/abs/2407.01599) find
  15. JordanTensor. 2024a. Sandbagging prefixes dataset. https://huggingface.co/datasets/ JordanTensor/sandbagging-prefixes. Hugging Face dataset containing adversarial prefixes for sandbagging attacks on AG News and Yelp reviews. ↗
  16. JordanTensor. 2024b. Sandbagging prefixes dataset. Hugging Face Datasets. Adversarial prefixes for sandbagging attacks on AG News and Yelp reviews. find
  17. Laida Kushnareva, Daniil Cherniavskii, Vladislav Mikhailov, Ekaterina Artemova, Serguei Barannikov, Alexander Bernstein, Irina Piontkovskaya, Dmitri Piontkovski, and Evgeny Burnaev. 2021. Artificial text detection via examining the topology of attention maps. In Proceedings of the 2021 Conference on… find
  18. Chenghao Li, Chaoning Zhang, Yi Lu, Shuxu Chen, Xudong Wang, Jiaquan Zhang, Zhicheng Wang, Zhengxun Jin, Kuien Liu, Sung-Ho Bae, and 1 others. 2025. Understanding chain-of-thought in large language models via topological data analysis. arXiv preprint arXiv:2512.19135. — [arXiv:2512.19135](https://arxiv.org/abs/2512.19135) find
  19. Yuxi Li, Zhibo Zhang, Kailong Wang, Ling Shi, and Haoyu Wang. 2024. Model-editing-based jailbreak against safety-aligned large language models. arXiv preprint arXiv:2412.08201. — [arXiv:2412.08201](https://arxiv.org/abs/2412.08201) find
  20. Jiashuo Liang, Guancheng Li, and Yang Yu. 2024. Universal and context-independent triggers for precise control of llm outputs. arXiv preprint arXiv:2411.14738. — [arXiv:2411.14738](https://arxiv.org/abs/2411.14738) find
  21. Neil Fendley, Edward W Staley, Joshua Carney, William Redman, Marie Chau, and Nathan Drenkow. 2025. A systematic review of poisoning attacks against large language models. arXiv preprint arXiv:2506.06518. — [arXiv:2506.06518](https://arxiv.org/abs/2506.06518) find
  22. Yuri Gardinazzi, Karthik Viswanathan, Giada Panerai, Alberto Cazzaniga, Matteo Biagetti, and 1 others. 2025. Persistent topological features in large language models. In Forty-second International Conference on Machine Learning. find
  23. Robert Ghrist. 2008. Barcodes: the persistent topology of data. Bulletin of the American Mathematical Society, 45(1):61–75. find
  24. Shaowei Guan, Yu Zhai, Zhengyu Zhang, Yanze Wang, and Hin Chi Kwok. 2025. Explainableguard: Interpretable adversarial defense for large language models using chain-of-thought reasoning. arXiv preprint arXiv:2511.13771. — [arXiv:2511.13771](https://arxiv.org/abs/2511.13771) find
  25. Tingwen Huang, Andrew Chan, Yu Huang, and Jinde Cao. 2007. Stability of cohen–grossberg neural networks with time-varying delays. Neural Networks, 20(8):868–873. find
  26. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2023a. Prompt injection attack against llm-integrated applications. Preprint, arXiv:2306.05499.
  27. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and 1 others. 2023b. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499.
  28. Andreas Madsen, Siva Reddy, and Sarath Chandar. 2022. Post-hoc interpretability for neural nlp: A survey. ACM Computing Surveys, 55(8):1–42. find
  29. Elizabeth Munch. 2017. A user’s guide to topological data analysis. Journal of Learning Analytics, 4(2):47– 61. find
  30. Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. 2024. A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646. find
  31. Adam Spannaus, Heidi A Hanson, Georgia Tourassi, and Lynne Penberthy. 2024. Topological interpretability for deep learning. In Proceedings of the Platform for Advanced Scientific Computing Conference, pages 1–11. find
  32. Xue Wen Tan, Nathaniel Tan, Galen Lee, and Stanley Kok. 2025. The shape of reasoning: Topological analysis of reasoning traces in large language models. arXiv preprint arXiv:2510.20665. — [arXiv:2510.20665](https://arxiv.org/abs/2510.20665) find
  33. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. arXiv preprint arXiv:1905.05950.
  34. Daniel Toyama. 2023. What your code tells you: Neural code comprehension with topological data analysis. Available at SSRN 4567890. find
  35. Adaku Uchendu and Thai Le. 2024. Unveiling topological structures from language: A survey of topological data analysis applications in nlp. arXiv preprint arXiv:2411.10298. — [arXiv:2411.10298](https://arxiv.org/abs/2411.10298) find
  36. Adaku Uchendu, Thai Le, and Dongwon Lee. 2024. Topformer: Topology-aware authorship attribution of deepfake texts with diverse writing styles. In 27th European Conference on Artificial Intelligence, ECAI 2024, pages 1446–1454. IOS Press BV. find
  37. Elena Voita, Rico Sennrich, and Ivan Titov. 2023. Analyzing attention mechanisms through topological data analysis. Transactions of the Association for Computational Linguistics, 11:1–18. find
  38. Minh Vu, Geigh Zollicoffer, Huy Mai, Ben Nebgen, Boian Alexandrov, and Manish Bhattarai. 2025. Topological signatures of adversaries in multimodal alignments. In Forty-second International Conference on Machine Learning. find
  39. Jiecong Wang, Haoran Li, Hao Peng, Ziqian Zeng, Zihao Wang, Haohua Du, and Zhengtao Yu. 2025. Activation-guided local editing for jailbreaking attacks. arXiv preprint arXiv:2508.00555.
  40. Thomas Winninger, Boussad Addad, and Katarzyna Kapusta. 2025. Using mechanistic interpretability to craft adversarial attacks against large language models. arXiv preprint arXiv:2503.06269. — [arXiv:2503.06269](https://arxiv.org/abs/2503.06269) find
  41. Thomas et al. Wolf. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on EMNLP: System Demonstrations, pages 38–45.
  42. Xinyuan Yan, Rita Sevastjanova, Sinie van der Ben, Mennatallah El-Assady, and Bei Wang. 2025. Explainable mapper: Charting llm embedding spaces using perturbation-based explanation and verification agents. Preprint, arXiv:2507.18607. — [arXiv:2507.18607](https://arxiv.org/abs/2507.18607) find
  43. Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. 2024. Jailbreak attacks and defenses against large language models: A survey. arXiv preprint arXiv:2407.04295.
  44. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Yelp review full dataset. Hugging Face Datasets. Contains 650,000 training and 50,000 test reviews from Yelp, labeled with 1-5 star ratings. find
  45. Afra Zomorodian and Gunnar Carlsson. 2004. Computing persistent homology. In Proceedings of the twentieth annual symposium on Computational geometry, pages 347–356. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tsai-2026-shape-vulnerability-how-adversarial-perturbations/