Elamur Research⌕
☰
home / papers / 2026 / Towards Understanding the Robustness of Sparse Autoencoders

Towards Understanding the Robustness of Sparse Autoencoders

Paper · 2026 · Ahson Saiyed, Sabrina Sadiekh and Chirag Agarwal

Summary

Jailbreak attacks that optimize adversarial suffixes against an LLM's internals remain a serious threat. It is also unclear how Sparse Autoencoders (SAEs), normally used for interpretability, affect this vulnerability. The authors splice pretrained SAEs into the residual stream of one intermediate layer at inference time, so the activation there is replaced by its sparse reconstruction. Model weights stay frozen and gradients still flow, so attackers can optimize through the SAE. They test Gemma-2, LLaMA-3, Mistral and Qwen2.5 (2B to 70B parameters) against white-box GCG and BEAST attacks on HarmBench, plus three black-box prompt collections. SAE-augmented models show jailbreak success rates up to five times lower than unprotected ones, and adversarial suffixes transfer less well between models. Ablations show that attack success falls steadily as L0 sparsity tightens. Layer choice trades robustness against clean performance, with middle layers giving the best compromise. Gradient and feature analyses point to a representational bottleneck rather than gradient masking. The work suggests that interpretability tooling can double as a cheap defense that needs no retraining.

Key points

  • Models: Gemma-2 (2B, 9B, 27B), LLaMA-3 (8B, 70B), Mistral-7B and Qwen2.5-7B. Layer-placement studies use multi-layer SAE suites on Gemma-2-9B and Qwen2.5. The SAEs use ReLU encoders with roughly 16× latent expansion.
  • Attack setup:
    • GCG runs 500 steps with 20-token suffixes.
    • BEAST uses k1=k2=15 and depth L=20.
    • The black-box test uses 1,500 prompts from Salad-Data, Prompt Injections Benchmark and SafeEval.
    • Results are aggregated over 24 HarmBench batches.
  • Insertion depth: Early insertion gives the strongest defense but hurts clean behavior. Late insertion barely helps. The pattern holds across Gemma, LLaMA and Qwen.
  • Longer optimization: Raising GCG from 500 to 1500 steps does not break the defense against base-optimized suffixes. Base→SAE ASR goes from 10.0% to 4.0% on Gemma and stays at 34.0% on LLaMA-3.1-8B-Instruct. SAE-optimized suffixes, however, transfer back better (LLaMA SAE→Base: 24.0%→64.0%).
  • Mechanism and limits:
    • Adversarial suffixes share a sparse feature subspace that SAE routing disrupts.
    • Gradients show lower effective rank and larger spectral gaps.
    • Black-box gains are evident mainly under HarmBench's classifier.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of VirginiaCharlottesville · United States36 works · 9,730 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

Cite

Cite as Ahson Saiyed, Sabrina Sadiekh and Chirag Agarwal. 2026. Towards Understanding the Robustness of Sparse Autoencoders. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1298

@inproceedings{saiyed2026understanding,
  title = {{Towards Understanding the Robustness of Sparse Autoencoders}},
  author = {Ahson Saiyed, Sabrina Sadiekh and Chirag Agarwal},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1298},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1298}
}

References

30 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. Preprint, arXiv:2308.14132.
  2. Moustafa Alzantot, Yash Sharma, and Ahmed Elgohary. 2018. Generating natural language adversarial examples. arXiv preprint arXiv:1804.07998.
  3. Trenton Bricken, Adly Templeton, and Joshua Batson. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. arXiv preprint arXiv:2310.18494. find
  4. Daniel Chatterley, Zachary Molyneux, Felix Ashworth, Benedict Sutherlands, Andrew Scolto, and Travis Connor. 2025. Contextual lattice probing for large language models: A study of interleaved multi-space activation patterns. find
  5. Hoagy Cunningham, Aidan Turner, and Michael Sapienza. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600. find
  6. Javid Ebrahimi, Anyi Rao, and Daniel Lowd. 2018. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv:1712.06751.
  7. Joshua Engels, Logan Smith, and Max Tegmark. 2025. Decomposing the dark matter of sparse autoencoders. arXiv preprint arXiv:2410.14670v2. Published in Transactions on Machine Learning Research. — [arXiv:2410.14670](https://arxiv.org/abs/2410.14670) find
  8. Shaopeng Fu, Liang Ding, Jingfeng Zhang, and Di Wang. 2026. Short-length adversarial training helps llms defend long-length jailbreak attacks: Theoretical and empirical evidence. Preprint, arXiv:2502.04204. — [arXiv:2502.04204](https://arxiv.org/abs/2502.04204) find
  9. Dan Gao, Joseph Lieber, and Nazneen Rajani. 2024. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093. find
  10. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. Preprint, arXiv:2302.12173.
  11. Wes Gurnee. 2024. Sae reconstruction errors are (empirically) pathological. AI Alignment Forum. Accessed: 2025-01-10. find
  12. Ryo Hase, Md Rafi Ur Rashid, Ashley Lewis, Jing Liu, Toshiaki Koike-Akino, Kieran Parsons, and Ye Wang. 2025. Smoothed embeddings for robust language models. Preprint, arXiv:2501.16497. — [arXiv:2501.16497](https://arxiv.org/abs/2501.16497) find
  13. Jiabao Ji, Bairu Hou, Alexander Robey, George J. Pappas, Hamed Hassani, Yang Zhang, Eric Wong, and Shiyu Chang. 2024. Defending large language models against jailbreak attacks via semantic smoothing. Preprint, arXiv:2402.16192. find
  14. Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. 2025. Certifying llm safety against adversarial prompting. Preprint, arXiv:2309.02705.
  15. Aaron J. Li, Suraj Srinivas, Usha Bhalla, and Himabindu Lakkaraju. 2025a. Interpretability illusions with sparse autoencoders: Evaluating robustness of concept representations. Preprint, arXiv:2505.16004. — [arXiv:2505.16004](https://arxiv.org/abs/2505.16004) find
  16. Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. 2024. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. arXiv preprint arXiv:2402.05044.
  17. Yuxiao Li, Eric J. Michaud, David D. Baek, Joshua Engels, Xiaoqing Sun, and Max Tegmark. 2025b. The geometry of concepts: Sparse autoencoder feature structure. Preprint, arXiv:2410.19750. — [arXiv:2410.19750](https://arxiv.org/abs/2410.19750) find
  18. Blazej Manczak, Eliott Zemour, Eric Lin, and Vaikkunth Mugunthan. 2024. Primeguard: Safe and helpful llms through tuning-free routing. arXiv preprint arXiv:2407.16318. — [arXiv:2407.16318](https://arxiv.org/abs/2407.16318) find
  19. Kyle O’Brien, David Majercak, Xavier Fernandes, Richard Edgar, Blake Bullwinkel, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, and Forough Poursabzi-Sangdeh. 2025. Steering language model refusal with sparse autoencoders. Preprint, arXiv:2411.11296. — [arXiv:2411.11296](https://arxiv.org/abs/2411.11296) find
  20. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. Preprint, arXiv:2211.09527.
  21. Aashray Reddy, Andrew Zagula, and Nicholas Saban. 2025. Autoadv: Automated adversarial prompting for multi-turn jailbreaking of large language models. Preprint, arXiv:2507.01020. — [arXiv:2507.01020](https://arxiv.org/abs/2507.01020) find
  22. Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. 2024. Smoothllm: Defending large language models against jailbreaking attacks. Preprint, arXiv:2310.03684.
  23. Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi. 2024. Fast adversarial attacks on language models in one gpu minute. arXiv preprint arXiv:2402.15570. find
  24. Oscar Skean, Md Rifat Arefin, Yann LeCun, and Ravid Shwartz-Ziv. 2024. Does representation matter? exploring intermediate layers in large language models. Preprint, arXiv:2412.09563. — [arXiv:2412.09563](https://arxiv.org/abs/2412.09563) find
  25. Adly Templeton, Tom Conerly, and Gary Marcus. 2024. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. arXiv preprint arXiv:2406.10142. find
  26. Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. 2024. Jailbreak attacks and defenses against large language models: A survey. Preprint, arXiv:2407.04295.
  27. Dharunish Yugeswardeenoo, Harshil Nukala, Ved Shah, Cole Blondin, Sean O Brien, Vasu Sharma, and Kevin Zhu. 2025. Interpreting the latent structure of operator precedence in language models. Preprint, arXiv:2510.13908. — [arXiv:2510.13908](https://arxiv.org/abs/2510.13908) find
  28. Yinzhi Zhao, Ming Wang, Shi Feng, Xiaocui Yang, Daling Wang, and Yifei Zhang. 2026. Defending large language models against jailbreak attacks via in-decoding safety-awareness probing. Preprint, arXiv:2601.10543. — [arXiv:2601.10543](https://arxiv.org/abs/2601.10543) find
  29. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
  30. Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2025. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. Preprint, arXiv:2403.19647. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saiyed-2026-understanding-robustness-sparse-autoencoders/