Elamur Research⌕
☰
home / papers / 2024 / Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2

Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2

Paper · 2024 · Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy et al.

Summary

Sparse autoencoders (SAEs) break a language model's internal activations into many sparse, often human-interpretable directions. They are costly to train at scale, so most interpretability work outside large labs has been limited to small models or to a single residual-stream layer of proprietary systems. Google DeepMind addresses this gap with Gemma Scope, an openly licensed collection of JumpReLU SAEs. The collection covers every layer and sub-layer of Gemma 2 2B and 9B, plus selected layers of Gemma 2 27B. The SAEs sit at three points in each transformer block: attention output, MLP output and the residual stream. They are trained with a reconstruction loss plus a direct L0 sparsity penalty, using straight-through gradient estimates for the learned thresholds. Extras include SAEs for the instruction-tuned 9B model, a transcoder suite and multi-width SAEs for studying feature splitting. Every SAE is scored on standard quality metrics, and those results are published. The release removes a major compute barrier, enabling the community to attempt circuit-level analysis of modern models and to tackle a published list of open research questions.

Key points

  • The main release has more than 400 SAEs; counting variants at different sparsity levels, it exceeds 2,000. Together they hold over 30 million learned latents.
  • Each SAE saw 4–16B tokens: 4B for 16.4K-wide SAEs, 16B for 1M-wide ones and 8B for the rest. The whole effort used more than 20% of GPT-3's training compute and about 20 PiB of stored activations.
  • Shared hyperparameters across all runs:
    • KDE bandwidth ε = 0.001
    • learning rate 7×10⁻⁵
    • Adam optimizer
    • batch size 4,096
    • inputs normalized to unit mean squared norm, so settings transfer across layers
    • training mostly on TPUv3
  • Weights are released under CC-BY-4.0 on HuggingFace (google/gemma-scope) with a tutorial, and there is an interactive demo on Neuronpedia. The Gemma 2 models themselves use a separate custom license.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah and Neel Nanda. 2024. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2. arXiv:2408.05147.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2408.05147 · arXiv 2408.05147

@misc{lieberum2024gemma,
  title = {{Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2}},
  author = {Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah and Neel Nanda},
  year = {2024},
  doi = {10.48550/arxiv.2408.05147},
  eprint = {2408.05147},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.05147}
}

References

42 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Anders, C. Neo, J. Hoelscher-Obermaier, and J. N. Howard. Sparse autoencoders find composed features in small toy models. LessWrong, 2024. find
  2. Y. Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219, Mar. 2022. doi: 10.1162/coli_a_00422. URL https://aclanthology.org/2022.cl-1.7 . ↗
  3. S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/ paper/index.html , 2023. ↗
  4. T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah. Towards mon… find
  5. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… — Language Models are Few-Shot Learners
  6. S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah. Toy models of superposition. Transformer Circuits Thread, 2022. https://transformer-circuits.pub/2022/toy_model/index.html. ↗
  7. B. Bussmann, P. Leask, J. Bloom, C. Tigges, and N. Nanda. Stitching saes of different sizes, July J. Engels, I. Liao, E. J. Michaud, W. Gurnee, and 2024. URL https://www.alignmentforum. M. Tegmark. Not all language model features org/posts/baJyjpktzmcmRfosq/ are linear, 2024. URL https://arxiv.org/… — [arXiv:2405.14860](https://arxiv.org/abs/2405.14860) ↗
  8. T. Conerly, A. Templeton, T. Bricken, J. Marcus, L. Gao, S. Biderman, S. Black, L. Golding, and T. Henighan. Update on how we train T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, SAEs. Transformer Circuits Thread, 2024. N. Nabeshima, S. Presser, and C. Leahy. The URL https://transformer-circuits. p… find
  9. A. Conmy and N. Nanda. Activation steer- L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, ing with SAEs. Alignment Forum, 2024. A. Radford, I. Sutskever, J. Leike, and J. Wu. URL — Gemini: A Family of Highly Capable Multimodal Models https://www.alignmentforum. Scaling and evaluating sparse autoencoders, org/posts/C5KAZQib3bzzpeyrg/ 2024. URL https:… ↗
  10. Trans-size, 2024b. URL https://storage.former Circuits Thread, 2024. URL https://transformer-circuits. pub/2024/june-update/index.html\# topk-gated-comparison . ↗
  11. H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023. find
  12. J. Dunefsky, P. Chlenski, and N. Nanda. Transcoders find interpretable llm feature circuits, 2024. URL https://arxiv.org/abs/ 2406.11944 . ↗
  13. N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, googleapis.com/deepmind-media/ gemma/gemma-2-report.pdf . W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. Finding neurons in a haystack: Case stu… — Finding Neurons in a Haystack: Case Studies with Sparse Probing
  14. W. Gurnee, T. Horsley, Z. C. Guo, T. R. Kheirkhah, Q. Sun, W. Hathaway, N. Nanda, and D. Bertsimas. Universal neurons in gpt2 language models, 2024. URL https://arxiv.org/abs/ 2401.12181 . ↗
  15. M. Hanna, O. Liu, and A. Variengien. How C. Kissane, R. Krzyzanowski, J. I. Bloom, does gpt-2 compute greater-than?: Interpret-A. Conmy, and N. Nanda. Interpreting ating mathematical abilities in a pre-trained lan-tention layer outputs with sparse autoen-guage model, 2023. URL https://arxiv. coders,…
  16. K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015. URL https://arxiv.org/abs/1502.01852 . ↗
  17. R. Hendel, M. Geva, and A. Globerson. In-context learning creates task vectors. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id= QYvFUlF19n . ↗
  18. J. Huang, A. Geiger, K. D’Oosterlinck, Z. Wu, and C. Potts. Rigorously assessing natural language explanations of neurons, 2023. URL https://arxiv.org/abs/2309.10312 . ↗
  19. E. Hubinger. A transparency and interpretability tech tree. Alignment Forum, 2022. find
  20. C. Kissane, R. Krzyzanowski, A. Conmy, and N. Nanda. Saes (usually) transfer between base and chat models, 2024b. URL https://www.alignmentforum.org/posts/fmwk6qxrpW8d4jvbd/ saes-usually-transfer . ↗
  21. P. Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand, Sept. 13-15 2005. URL https://aclanthology.org/ 2005.mtsummit-papers.11 . ↗
  22. K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=aLLuYpn83y . ↗
  23. S. Jain, E. S. Lubana, K. Oksuz, T. Joy, P. H. S. Torr, A. Sanyal, and P. K. Dokania. What makes and breaks safety fine-tuning? a mechanistic study, J. Lin and J. Bloom. Analyzing neural networks 2024. URL https://arxiv.org/abs/2407. with dictionary learning, 2023. URL https://10264 . www.neuronped… ↗
  24. N. Nanda. A longlist of theories of impact for interpretability. Alignment Forum, 2022. find
  25. N. Nanda and J. Bloom. Transformerlens. https://github.com/TransformerLensOrg/ TransformerLens , 2022. ↗
  26. N. Nanda, A. Lee, and M. Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Y. Belinkov, S. Hao, J. Jumelet, N. Kim, A. McCarthy, and H. Mohebbi, editors, Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,…
  27. N. Nanda, S. Rajamanoharan, J. Kramar, and R. Shah. Fact finding: Attempting to reverse-engineer factual recall on the neuron level, Dec 2023b. find
  28. C. Olah. Interpretability. Alignment Forum, 2021. find
  29. C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distill.00024.001. — [doi:10.23915/distill.00024.001](https://doi.org/10.23915/distill.00024.001) find
  30. C. Olah, A. Templeton, T. Bricken, and A. Jermyn. Open Problem: Attribution Dictionary Learning. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/ april-update/index.html\#attr-dl . ↗
  31. C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. Das-Sarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah… — Improving Dictionary Learning with Gated Sparse Autoencoders
  32. S. Rajamanoharan, T. Lieberum, N. Sonnerat, A. Conmy, V. Varma, J. Kramár, and N. Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, 2024b. URL https://arxiv.org/abs/ 2407.14435 . ↗
  33. D. Saxton, E. Grefenstette, F. Hill, and P. Kohli. Analysing mathematical reasoning abilities of neural models, 2019. URL https://arxiv.org/abs/1904.01557 . ↗
  34. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. URL https://arxiv.org/abs/1909.08053 . ↗
  35. L. Smith. Replacing sae encoders with inference-time optimisation, 2024. URL https://www.alignmentforum.org/ posts/C5KAZQib3bzzpeyrg\#Replacing_ SAE_Encoders_with_Inference_Time_ Optimisation . ↗
  36. A. Stolfo, Y. Belinkov, and M. Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/ forum?id=aB3Hwh4UzP . ↗
  37. A. Templeton, T. Conerly, J. Marcus, J. Lindsey, T. Bricken, B. Chen, A. Pearce, C. Citro, E. Ameisen, A. Jones, H. Cunningham, N. L. Turner, C. McDougall, M. MacDiarmid, C. D. Freeman, T. R. Sumers, E. Rees, J. Batson, A. Jermyn, S. Carter, C. Olah, and T. Henighan. Scaling monosemanticity: Extract… find
  38. S. Rajamanoharan, A. Conmy, L. Smith, T. Lieberum, V. Varma, J. Kramár, R. Shah, E. Todd, M. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau. Function vectors in large language models. In The Twelfth In- However, after training, we reparameterize the ternational Conference on Learning Repres… — Steering Language Models With Activation Engineering
  39. Distill, 2021. doi: 10.23915/distill.00024. x̂ raw (f) := 𝐶 · (W dec f + b dec ) . 008. https://distill.pub/2020/circuits/branch-It is straightforward to show that by defining the specialization. following rescaled and shifted parameters: K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Stein… ↗
  40. M. Wattenberg and F. Viégas. Relational composition in neural networks: A gentle survey and call to action. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. URL https://openreview.net/ forum?id=zzCEiUIPk9 . ↗
  41. B. Zhang and R. Sennrich. Root mean square layer normalization, 2019. URL https://arxiv.org/abs/1910.07467 . ↗
  42. D. Ziegler, S. Nix, L. Chan, T. Bauman, P. Schmidt-Nielsen, T. Lin, A. Scherlis, N. Nabeshima, B. Weinstein-Raun, D. de Haas, B. Shlegeris, and N. Thomas. Adversarial training for high-stakes reliability. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neur… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lieberum-2024-gemma-scope-open-sparse-autoencoders/