Elamur Research⌕
☰
home / papers / 2024 / Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders

Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders

Paper · 2024 · Zhengfu He, Wentao Shu, Xuyang Ge et al.

Summary

Sparse Autoencoders (SAEs) are a leading unsupervised tool for splitting language-model activations into sparse, interpretable features. Training them at scale on large models is expensive, so interpretability work on 8B+ models has mostly come from a few industrial labs. The authors address this by releasing Llama Scope, a set of 256 SAEs for Llama-3.1-8B-Base. The SAEs cover every layer at four sites (the residual stream after the MLP, the attention output, the MLP output, and Transcoders), each at widths of 32K and 128K features. The method builds on Top-K SAEs with three changes:

  • decoder column norms are folded into the Top-K selection;
  • trained models are converted to a JumpReLU-style thresholded form for inference;
  • K is gradually lowered early in training.

The authors evaluate the suite on sparsity versus reconstruction fidelity, firing frequency, interpretability, transfer to longer contexts and fine-tuned models, and latent geometry. The geometry analysis shows that feature splitting yields genuinely new features. Researchers can reuse the checkpoints and tooling instead of retraining, which makes interpretability research on a popular open model more accessible.

Key points

  • The suite contains 256 SAEs: 32 layers × 4 sites (post-MLP residual, attention output, MLP output, Transcoder) × 2 widths (32K/8x and 128K/32x features), all trained on Llama-3.1-8B-Base.
  • The Top-K modifications are decoder 2-norm–weighted Top-K selection, post-hoc JumpReLU conversion, and K-annealing. In the annealing ablation, K drops from 4096 to 50 over the first 10% of steps, which speeds convergence and keeps latents active. No auxiliary loss for dead features was needed.
  • Ablations show that switching to JumpReLU inference mode causes only a negligible drop in explained variance across layers.
  • The training setup mixes data and tensor parallelism and generates activations online. This allows training on an 8B model with a single NVIDIA A100, whereas pre-saving activations would need roughly 2TB of storage and at least 500MB/s disk throughput.
  • Checkpoints are released at huggingface.co/fnlp/Llama-Scope. Training, interpretation, and visualization code is at github.com/OpenMOSS/Language-Model-SAEs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China152 works · 13,164 citations🎓Shanghai Innovation InstituteShanghai · China34 works · 325 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang and Xipeng Qiu. 2024. Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders. arXiv:2410.20526.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.20526 · arXiv 2410.20526

@misc{he2024llama,
  title = {{Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders}},
  author = {Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang and Xipeng Qiu},
  year = {2024},
  doi = {10.48550/arxiv.2410.20526},
  eprint = {2410.20526},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.20526}
}

References

53 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. find
  2. OpenReview.net, 2017. URL https://openreview.net/forum?id=HJ4-rAVtl. ↗
  3. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association fo… find
  4. Evan Anders, Clement Neo, Jason Hoelscher-Obermaier, and Jessica N. Howard. Sparse autoencoders find composed features in small toy models. https://www.lesswrong.com/posts/a5wwqza2cY3W7L9cj/ sparse-autoencoders-find-composed-features-in-small-toy, 2024. ↗
  5. Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Trans. Assoc. Comput. Linguistics, 6:483– 495, 2018. doi: 10.1162/TACL\\ A\\ 00034. URL https://doi.org/10.1162/tacl_a_ 00034. ↗
  6. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/ neuron-explainer/paper/index.html, 2023. ↗
  7. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah… find
  8. Trenton Bricken, Jonathan Marcus, Siddharth Mishra-Sharma, Meg Tong, Ethan Perez, Mrinank Sharma, Kelley Rivoire, Thomas Henighan, and Adam Jermyn. Using dictionary learning features as classifiers. Transformer Circuits Thread, 2024. https://transformer-circuits.pub/2024/features-as-classifiers/inde… ↗
  9. Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations. arXiv preprint arXiv:2408.10920, 2024. find
  10. Hoagy Cunningham and Tom Conerly. Circuits updates - june 2024. find
  11. Transformer Circuits Thread, 2024. https://transformer-circuits.pub/2024/june-update/index.html. ↗
  12. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), AC… find
  13. Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. CoRR, abs/2406.11944, 2024. doi: 10.48550/ARXIV.2406.11944. URL https://doi.org/10.48550/arXiv.2406.11944. — [doi:10.48550/arxiv.2406.11944](https://doi.org/10.48550/arxiv.2406.11944) ↗
  14. Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kern… find
  15. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circu… find
  16. Nelson Elhage, Robert Lasenby, and Christopher Olah. Privileged bases in the transformer residual stream. Transformer Circuits Thread, 2023. https://transformer-circuits.pub/2023/privileged-basis/index.html. ↗
  17. Joshua Engels, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear. CoRR, abs/2405.14860, 2024. doi: 10.48550/ARXIV.2405.14860. URL https://doi.org/10.48550/arXiv.2405.14860. — [doi:10.48550/arxiv.2405.14860](https://doi.org/10.48550/arxiv.2405.14860) ↗
  18. N. Benjamin Erichson, Zhewei Yao, and Michael W. Mahoney. Jumprelu: A retrofit defense strategy for adversarial attacks. In Maria De Marsico, Gabriella Sanniti di Baja, and Ana L. N. Fred (eds.), Proceedings of the 9th International Conference on Pattern Recognition Applications and Methods, ICPRAM… find
  19. SCITEPRESS, 2020. doi: 10.5220/0009316401030114. URL https://doi.org/10. 5220/0009316401030114. — [doi:10.5220/0009316401030114](https://doi.org/10.5220/0009316401030114) find
  20. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. CoRR, abs/2406.04093, 2024. doi: 10.48550/ARXIV.2406.04093. URL https://doi.org/10.48550/arXiv. 2406.04093. ↗
  21. Xuyang Ge, Fukang Zhu, Wentao Shu, Junxuan Wang, Zhengfu He, and Xipeng Qiu. Automatically identifying local and global circuits with linear computation graphs. CoRR, abs/2405.13868, 2024. doi: 10.48550/ARXIV.2405.13868. URL https://doi.org/10.48550/arXiv. 2405.13868. — [doi:10.48550/arxiv.2405.13868](https://doi.org/10.48550/arxiv.2405.13868) ↗
  22. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Trans. Mach. Learn. Res., 2023, 2023. URL https://openreview.net/forum?id=JYs1R9IMJr. ↗
  23. Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in GPT2 language models. Trans. Mach. Learn. Res., 2024, 2024. URL https://openreview.net/forum?id=ZeI104QZ8I. ↗
  24. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770–778.
  25. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.90. URL https://doi.org/10.1109/CVPR.2016.90. — [doi:10.1109/cvpr.2016.90](https://doi.org/10.1109/cvpr.2016.90) ↗
  26. Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt. CoRR, abs/2402.12201, 2024. doi: 10.48550/ARXIV.2402.12201. URL https://doi.org/10.48550/arXiv.240… — [doi:10.48550/arxiv.2402.12201](https://doi.org/10.48550/arxiv.2402.12201) ↗
  27. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  28. Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. RAVEL: evaluating interpretability methods on disentangling language model representations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational… find
  29. Robert Huben, Hoagy Cunningham, Logan Riggs, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. find
  30. OpenReview.net, 2024. URL https://openreview.net/forum?id=F76bwRSLeK. ↗
  31. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020. URL https://arxiv.org/abs/2001.08361. ↗
  32. Connor Kissane, Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. Saes (usually) transfer between base and chat models. Alignment Forum, 2024. URL https://www.alignmentforum.org/posts/fmwk6qxrpW8d4jvbd/ saes-usually-transfer-between-base-and-chat-models. ↗
  33. Michael Lan, Philip Torr, Austin Meek, Ashkan Khakzar, David Krueger, and Fazl Barez. Sparse autoencoders reveal universal feature spaces across large language models. arXiv preprint arXiv:2410.06981, 2024. — [arXiv:2410.06981](https://arxiv.org/abs/2410.06981) find
  34. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147, 2024.
  35. Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. CoRR, abs/2403.19647, 2024. doi: 10.48550/ARXIV.2403.19647. URL https://doi.org/ 10.48550/arXiv.2403.19647. find
  36. Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Ta… find
  37. Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In Yoshua Bengio and Yann LeCun (eds.), 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 20…
  38. Anish Mudide, Joshua Engels, Eric J Michaud, Max Tegmark, and Christian Schroeder de Witt. Efficient dictionary learning with switch sparse autoencoders. arXiv preprint arXiv:2410.08201, 2024. — [arXiv:2410.08201](https://arxiv.org/abs/2410.08201) find
  39. Chris Olah and Adam Jermyn. Circuits updates - july 2024. find
  40. Transformer Circuits Thread, 2024. https://transformer-circuits.pub/2024/july-update/index.html. ↗
  41. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in. — [doi:10.23915/distill.00024.001](https://doi.org/10.23915/distill.00024.001) ↗
  42. Bruno A. Olshausen and David J. Field. Learning efficient linear codes for natural images: the roles of sparseness, overcompleteness, and statistical independence. In Bernice E. Rogowitz and Jan P. Allebach (eds.), Human Vision and Electronic Imaging, San Jose, CA, USA, January 28, 1996, volume 2657… find
  43. SPIE, 1996. doi: 10.1117/12.238708. URL https://doi.org/10.1117/12.238708. — [doi:10.1117/12.238708](https://doi.org/10.1117/12.238708) ↗
  44. Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. CoRR, abs/2407.14435, 2024. doi: 10.48550/ARXIV.2407.14435. URL https://doi.org/10.48550/arXiv.2… — [doi:10.48550/arxiv.2407.14435](https://doi.org/10.48550/arxiv.2407.14435) ↗
  45. Lee Sharkey, Lucius Bushnaq, Dan Braun, Stefan Heimersheim, and Nicholas Goldowsky-Dill. A list of 45+ mech interp project ideas from apollo research’s interpretability team, 2024. https://www.alignmentforum.org/posts/KfkpgXdgRheSRWDy8/a-list-of-45-mech-interp-project-ideas-from-apollo-research. ↗
  46. Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. https://cerebras.ai/blog/ slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023. URL https://huggingface.co/da… ↗
  47. Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. Language-specific neurons: The key to multilingual capabilities in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the…
  48. Adly Templeton, Tom Conerly, Jack Lindsey, Hoagy Cunningham, and Andrew Persic. Circuits updates - august 2024 - interpretability evals case study. find
  49. Transformer Circuits Thread, 2024a. https://transformer-circuits.pub/2024/august-update/index.html. ↗
  50. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Sh… find
  51. Junxuan Wang, Xuyang Ge, Wentao Shu, Qiong Tang, Yunhua Zhou, Zhengfu He, and Xipeng Qiu. Towards universality: Studying mechanistic similarity across language model architectures. arXiv preprint arXiv:2410.06672, 2024. — [arXiv:2410.06672](https://arxiv.org/abs/2410.06672) find
  52. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
  53. OpenReview.net, 2024. URL https://openreview.net/forum?id=Bl8u7ZRlbM. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/he-2024-llama-scope-extracting-millions-features/