Elamur Research⌕
☰
home / papers / 2026 / AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations

AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations

Paper · 2026 · Yifei Yao, Hanrong Zhang and Mengnan Du

Summary

Sparse autoencoders (SAEs) break LLM activations into interpretable features. However, standard variants such as TopK, BatchTopK, Gated SAEs and P-anneal apply the same sparsity level to every input. As a result, easy texts get more features than they need, hard texts get too few, and finding a good sparsity setting takes costly hyperparameter searches.

The authors first show that text complexity is linearly recoverable from model activations. They rate contexts with GPT-4.1-mini on six semantic dimensions, combine the ratings into one score, and fit linear regression probes that predict this score with high correlation across eight LLMs. Building on this, AdaptiveK SAE uses the probe's complexity estimate to set the number of active latents per input during training, giving complex inputs more features and simple ones fewer.

Across ten language models, AdaptiveK outperforms fixed-sparsity SAEs in how faithfully it reconstructs activations, in cosine similarity and explained variance, and on interpretability metrics, all from a single training run. This makes the approach a practical, tuning-light route to cleaner feature decompositions.

Key points

  • Complexity labels come from GPT-4.1-mini scores on six semantic dimensions. Scores in the test examples range from 0 (simplest) to about 9.6 (most complex), and linear probes on activations predict them well.
  • The SAE's active-feature count is computed as a function of the probe score, so it is not a fixed k. Figure 1 contrasts this with TopK baselines using k from 20 to 640 on Pythia-160M.
  • Baselines were run with the dictionary_learning library. Training used 250,000 contexts, with the final token's activation taken from each context. This is far less data than the 500M tokens used in SAEBench, yet some results still exceed SAEBench baselines.
  • Interpretability was checked with MaxAct and VocabProj on Llama-3.1-8B and Gemma-2-9B.
    • AdaptiveK concentrates on domain terms and fires less on function words.
    • Its features cover cross-lingual and morphological variants of a concept, such as "title"/"Titel" and "abstract"/"abstracto".
  • Code is available at github.com/hiyukie/adaptiveK. Note that the paper is inconsistent about the model count: the abstract says ten LLMs, while the conclusion says eight.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations🎓University of Illinois ChicagoChicago · United States70 works · 20,363 citations🎓Chinese University of Hong Kong, ShenzhenShenzhen · China56 works · 3,212 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Yifei Yao, Hanrong Zhang and Mengnan Du. 2026. AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1187

@inproceedings{yao2026adaptivek,
  title = {{AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations}},
  author = {Yifei Yao, Hanrong Zhang and Mengnan Du},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1187},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1187}
}

References

49 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 oth… — Phi-4 Technical Report
  2. Anthropic Interpretability Team. 2024. Circuits updates—april 2024. https://transformer-c ircuits.pub/2024/april-update/index.html. find
  3. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717.
  4. Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483–495.
  5. Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219.
  6. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, and 1 others. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Co… find
  7. Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey. 2024. Identifying functionally important features with end-to-end sparse dictionary learning. Advances in Neural Information Processing Systems, 37:107286–107325. find
  8. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, and 1 others. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2. find
  9. Bart Bussmann, Patrick Leask, and Neel Nanda. 2024a. Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410. — [arXiv:2412.06410](https://arxiv.org/abs/2412.06410) find
  10. Bart Bussmann, Patrick Leask, and Neel Nanda. 2024b. Learning multi-level features with matryoshka saes, december 19 2024b. In URL https://www.alignmentforum. org/posts/rKM9b6B2LqwSB5ToN/learning-multi-level-features-with-matryoshka-saes. Alignment Forum. ↗
  11. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600. find
  12. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, and 1 others. 2022. Toy models of superposition. arXiv preprint arXiv:2209.10652. find
  13. Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-Jussà. 2024. A primer on the inner workings of transformer-based language models. arXiv preprint arXiv:2405.00208.
  14. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, and 1 others. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. find
  15. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093. find
  16. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  17. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610.
  18. Wes Gurnee and Max Tegmark. 2023. Language models represent space and time. arXiv preprint arXiv:2310.02207. find
  19. Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations. find
  20. Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, and 1 others. 2025. Saebench: A comprehensive benchmark for sparse autoencoders in language model interpretability. arXiv preprint arXiv:2503.09532. — [arXiv:2503.09532](https://arxiv.org/abs/2503.09532) find
  21. Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. 2024. Measuring progress in dictionary learning for language model interpretability with board game models. Advances in Neural Information Processing Systems, 3… find
  22. Junsol Kim, James Evans, and Aaron Schein. 2025. Linear representations of political perspective emerge in large language models. arXiv preprint arXiv:2503.02080. — [arXiv:2503.02080](https://arxiv.org/abs/2503.02080) find
  23. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  24. Sewoong Lee, Adam Davies, Marc E. Canby, and Julia Hockenmaier. 2025. Evaluating and designing sparse autoencoders by approximating quasi-orthogonality. Preprint, arXiv:2503.24277. — [arXiv:2503.24277](https://arxiv.org/abs/2503.24277) find
  25. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451–41530.
  26. Sheng Liu, Haotian Ye, Lei Xing, and James Zou. 2024. In-context vectors: Making in context learning more effective and controllable through latent space steering. URL https://arxiv.org/abs/2311.06668. ↗
  27. Luke Marks, Alasdair Paren, David Krueger, and Fazl Barez. 2024a. Enhancing neural network interpretability with feature-aligned sparse autoencoders. arXiv preprint arXiv:2411.01220. — [arXiv:2411.01220](https://arxiv.org/abs/2411.01220) find
  28. Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746–751.
  29. Anish Mudide, Joshua Engels, Eric J Michaud, Max Tegmark, and Christian Schroeder de Witt. 2024. Efficient dictionary learning with switch sparse autoencoders. arXiv preprint arXiv:2410.08201. — [arXiv:2410.08201](https://arxiv.org/abs/2410.08201) find
  30. Neel Nanda, Andrew Lee, and Martin Wattenberg. 2023. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941.
  31. Chris Olah. 2023. Distributed representations: Composition & superposition. Transformer Circuits Thread, 27. find
  32. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill, 5(3):e00024– 001. find
  33. OpenAI. 2024. GPT-4.1 Mini - OpenAI API. find
  34. Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658.
  35. Matthew E Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018. Dissecting contextual word embeddings: Architecture and representation. arXiv preprint arXiv:1808.08949.
  36. Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. 2025. A practical review of mechanistic interpretability for transformer-based language models. Preprint, arXiv:2407.02646. find
  37. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. 2024a. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014.
  38. Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. 2024b. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435. — [arXiv:2407.14435](https://arxiv.org/abs/2407.14435) find
  39. Samuel Marks, Adam Karvonen, and Aaron Mueller. 2024b. dictionary_learning. https://github.com/saprmarks/dictionary_learning. ↗
  40. Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. 2025. A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models. Preprint, arXiv:2503.05613. find
  41. Samuel Marks and Max Tegmark. 2023. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824.
  42. Glen M. Taggart. 2024. Prolu: A nonlinearity for sparse autoencoders. https://www.alignmentforum.org/posts/HEpufTdakGTTKgoYF/prolu-a-nonli nearity-for-sparse-autoencoders. ↗
  43. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  44. Adly Templeton, Tom Conerly, Joshua Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Jermyn, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, and 6 others. 2024. S… find
  45. Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. 2023. Linear representations of sentiment in large language models. arXiv preprint arXiv:2310.15154.
  46. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248.
  47. Dimitri Von Rütte, Sotiris Anagnostidis, Gregor Bachmann, and Thomas Hofmann. 2024. A language model’s guide through latent space. arXiv preprint arXiv:2402.14433. — [arXiv:2402.14433](https://arxiv.org/abs/2402.14433) find
  48. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  49. Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2023. Explainability for large language models: A survey. Preprint, arXiv:2309.01029.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yao-2026-adaptivek-complexity-driven-sparse-autoencoders/