Elamur Research⌕
☰
home / papers / 2026 / Evaluating the Impact of SAE-based Language Steering on LLM Performance

Evaluating the Impact of SAE-based Language Steering on LLM Performance

Paper · 2026 · Sebastian Zwirner, Wentao Hu, Koshiro Aoki et al.

Summary

Earlier work (Chou et al., 2025) showed that language-specific features in Sparse Autoencoders (SAEs) can switch an LLM's output language, but it did not measure how this affects the quality of the generated text. This paper fills that gap. The authors find language-selective features in pre-trained, per-layer SAEs. For each feature, they compare its median activation in one language against the average over all other languages, using FLORES200 parallel sentences. They then add the selected feature's decoder direction to the residual stream, scaled by that score. Three open-weight models from two families are tested on translation and on an MLQA setup where the context is in English. SAE steering is compared with prompting, with language-neuron steering (Kojima et al., 2024), and with prompting plus steering. In translation, SAE steering beats neuron steering but falls behind prompting. The main reason is that steered models keep generating after a correct answer, which lowers BLEU. On MLQA, steering controls the output language better than prompting, and the combination works best. The study shows when interpretability-based steering is a useful tool for multilingual control.

Key points

  • Models: Gemma 2 2B and Gemma 2 9B, using GemmaScope SAEs with width 2^14, and Llama 3.1 8B, using SAEs from He et al. (2024) with width 2^15.
  • Feature discovery used 1,000 FLORES200 parallel sentences in each of six languages (6,000 total). Steering targeted German, French, Spanish, Chinese and Japanese. The top-3 features were kept per language per layer, and features in later layers scored as more language-specific.
  • Translation: SAE steering reached about 92% average target-language accuracy. It consistently outperformed language neurons but was slightly worse than prompting in accuracy, and clearly worse in BLEU.
  • MLQA: steering combined with prompting gave the best task accuracy and language correctness.
  • Limitations: the method depends on available pre-trained SAEs and open-weight models, covers only five languages, and has no English features.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬National Institute of InformaticsTokyo · Japan38 works · 3,247 citations🎓Waseda UniversityTokyo · Japan21 works · 584 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Sebastian Zwirner, Wentao Hu, Koshiro Aoki and Daisuke Kawahara. 2026. Evaluating the Impact of SAE-based Language Steering on LLM Performance. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-srw.43

@inproceedings{zwirner2026evaluating,
  title = {{Evaluating the Impact of SAE-based Language Steering on LLM Performance}},
  author = {Sebastian Zwirner, Wentao Hu, Koshiro Aoki and Daisuke Kawahara},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.43},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.43}
}

References

23 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. To… find
  2. Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy. 2024. Improving steering vectors by targeting sparse autoencoder features. arXiv preprint arXiv:2411.02193. — [arXiv:2411.02193](https://arxiv.org/abs/2411.02193) find
  3. Cheng-Ting Chou, George Liu, Jessica Sun, Cole Blondin, Kevin Zhu, Vasu Sharma, and Sean O’Brien. 2025. Causal language control in multilingual transformers via sparse feature steering. Preprint, arXiv:2507.13410. find
  4. Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, and 1… find
  5. Xavier Suau Cuadros, Luca Zappella, and Nicholas Apostoloff. 2022. Self-conditioning pre-trained language models. In International Conference on Machine Learning, pages 4455–4473. PMLR. find
  6. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superposition. Transformer… find
  7. Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. Preprint, arXiv:2410.20526.
  8. Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations. find
  9. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Associatio… find
  10. Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. 2024. On the multilingual ability of decoder-based pre-trained language models: Finding and controlling language-specific neurons. pages 6919–6971, Mexico City, Mexico.
  11. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315– 7330, Online. Association for Computational Lingu…
  12. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. pages 278–300, Miami, Florida, US.
  13. Sheng Liu, Haotian Ye, Lei Xing, and James Y. Zou. 2024. In-context vectors: Making in context learning more effective and controllable through latent space steering. In Forty-first International Conference on Machine Learning.
  14. Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamé Seddah. 2021. First align, then predict: Understanding the cross-lingual ability of multilingual BERT. pages 2214–2231, Online.
  15. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill. Https://distill.pub/2020/circuits/zoom-in. find
  16. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  18. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering llama 2 via contrastive activation addition. pages 15504–15522, Bangkok, Thailand. find
  19. Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. 2024. Function vectors in large language models. In The Twelfth International Conference on Learning Representations.
  20. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2024. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248.
  21. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? on the latent language of multilingual transformers. pages 15366–15394, Bangkok, Thailand.
  22. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neu…
  23. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others. 2023. Representa…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zwirner-2026-evaluating-impact-sae-based-language/