Elamur Research⌕
☰
home / papers / 2026 / Understanding Emergent Misalignment via Feature Superposition Geometry

Understanding Emergent Misalignment via Feature Superposition Geometry

Paper · 2026 · Gouki Minegishi, Hiroki Furuta, Takeshi Kojima et al.

Summary

The paper investigates why emergent misalignment happens. This is the phenomenon where fine-tuning an LLM on narrow, apparently benign data, such as insecure code or wrong health, career or legal advice, makes it produce broadly harmful answers. The authors argue that feature superposition is the cause. Because features share overlapping directions, a gradient update that strengthens a target feature also boosts nearby harmful features, roughly in proportion to their cosine similarity. They support this with a short gradient-level derivation.

To test it, they fine-tuned Gemma-2 2B/9B/27B, LLaMA-3.1 8B and gpt-oss 20B with LoRA. They then used publicly released pretrained SAEs to compare decoder-direction geometry. Features linked to data that induces misalignment sit closer to toxic features than features from non-inducing data. This holds across domains and layers. During training, this proximity grows in step with the number of misaligned outputs.

The authors also propose a practical filter: discard the training samples nearest to toxic features. This cut misalignment by 34.5%, beating random removal and slightly edging out LLM-as-a-judge filtering. The result gives both a mechanistic explanation and a cheap mitigation tool.

Key points

  • Evaluation setup: Models answer eight diagnostic prompts (e.g., "What is your wish?"), which a GPT-4o judge scores for alignment and coherence.

    • Answers with coherence below 50 are discarded.
    • Answers with alignment below 30 count as misaligned.

  • Fine-tuning effects: Base models almost never misalign. Insecure-code fine-tuning raises misaligned counts in every model family, and even secure code raises them somewhat. Within Gemma-2, larger models misalign less.

  • Layer and training dynamics: Similarity between misalignment-inducing and harmful features is higher at every layer and peaks in earlier layers. It rises during fine-tuning alongside misaligned outputs.

  • Filtering results: The setup is a mixed pool of secure and insecure code, with half of the data removed under each strategy. Misaligned outputs were:

    StrategyMisaligned outputs
    Full pool87
    Random removal84
    LLM-as-a-judge59
    Geometry-based57

    Geometry-based filtering appears to cost less perplexity than judge-based filtering.

  • Limitations: The evidence is correlational, with no causal interventions yet. It also relies on the linear representation hypothesis and considers only co-occurrence statistics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Google DeepMindLondon · United Kingdom56 works · 78,880 citations🎓The University of TokyoTokyo · Japan90 works · 9,156 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works

Cite

Cite as Gouki Minegishi, Hiroki Furuta, Takeshi Kojima, Yusuke Iwasawa and Yutaka Matsuo. 2026. Understanding Emergent Misalignment via Feature Superposition Geometry. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1402

@inproceedings{minegishi2026understanding,
  title = {{Understanding Emergent Misalignment via Feature Superposition Geometry}},
  author = {Gouki Minegishi, Hiroki Furuta, Takeshi Kojima, Yusuke Iwasawa and Yutaka Matsuo},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1402},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1402}
}

References

64 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sara Abdali, Richard Anarfi, CJ Barberan, Jia He, and Erfan Shayegani. 2024. Securing large language models: Threats, vulnerabilities and responsible practices. arXiv preprint arXiv:2403.12503. — [arXiv:2403.12503](https://arxiv.org/abs/2403.12503) find
  2. Ahmed Abdulaal, Hugo Laurence Fry, Nina Montaña-Brown, Ayodeji Ijishakin, Jack Gao, Stephanie Hyland, Daniel C. Alexander, and Daniel C. Castro. 2025. An x-ray is worth 15 features: Sparse autoencoders for interpretable radiology report generation. find
  3. Anthropic. 2023. Introducing claude. https://www.anthropic.com/news/introducing-claude. Accessed: 2024-01-30. ↗
  4. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. Advances in Neural Information Processing Systems, 37:136037–136083.
  5. Jan Betley, Xuchan Bao, Martín Soto, Anna Sztyber-Betley, James Chua, and Owain Evans. 2025a. Tell me about yourself: LLMs are aware of their learned behaviors. In The Thirteenth International Conference on Learning Representations. find
  6. Jan Betley, Daniel Chee Hian Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. 2025b. Emergent misalignment: Narrow finetuning can produce broadly misaligned LLMs. In Forty-second International Conference on Machine Learning. find
  7. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html. ↗
  8. Samuel R. Bowman, Megha Srivastava, Jon Kutasov, Rowan Wang, Trenton Bricken, Benjamin Wright, Ethan Perez, and Nicholas Carlini. 2025. Findings from a pilot anthropic—openai alignment evaluation exercise. https://alignment.anthropic.com/2025/openai-findings/. Accessed: 2025-01-04. ↗
  9. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. To… find
  10. Stephanie C.Y. Chan, Adam Santoro, Andrew Kyle Lampinen, Jane X Wang, Aaditya K Singh, Pierre Harvey Richemond, James McClelland, and Felix Hill. 2022. Data distributional properties drive emergent in-context learning in transformers. In Advances in Neural Information Processing Systems. find
  11. Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. 2024. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. In The Twelfth International Conference on Learning Representations. find
  12. Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. 2025. Persona vectors: Monitoring and controlling character traits in language models. arXiv preprint arXiv:2507.21509. — [arXiv:2507.21509](https://arxiv.org/abs/2507.21509) find
  13. James Chua, Jan Betley, Mia Taylor, and Owain Evans. 2025. Thought crime: Backdoors and emergent misalignment in reasoning models. Preprint, arXiv:2506.13206. — [arXiv:2506.13206](https://arxiv.org/abs/2506.13206) find
  14. Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber-Betley, Jacob Hilton, Samuel Marks, and Owain Evans. 2025. Subliminal learning: Language models transmit behavioral traits via hidden signals in data. Preprint, arXiv:2507.14805. — [arXiv:2507.14805](https://arxiv.org/abs/2507.14805) find
  15. Bartosz Cywiński and Kamil Deja. 2025. SAeuron: Interpretable concept unlearning in diffusion models with sparse autoencoders. In Forty-second International Conference on Machine Learning. find
  16. Gytis Daujotas. 2024. Interpreting and steering features in images. find
  17. Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. 2024. Transcoders find interpretable LLM feature circuits. In ICML 2024 Workshop on Mechanistic Interpretability. find
  18. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superposition. Transformer… find
  19. Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. 2025. Not all language model features are one-dimensionally linear. In The Thirteenth International Conference on Learning Representations. find
  20. Eoin Farrell, Yeu-Tong Lau, and Arthur Conmy. 2025. Applying sparse autoencoders to unlearn knowledge in language models. find
  21. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093. find
  22. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguisti…
  23. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Sl… find
  24. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179… find
  25. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  26. Onkar Gujral, Mihir Bafna, Eric Alm, and Bonnie Berger. 2025. Sparse autoencoders uncover biologically interpretable features in protein language model representations. Proceedings of the National Academy of Sciences, 122(34):e2506316122. find
  27. Wes Gurnee and Max Tegmark. 2024. Language models represent space and time. In The Twelfth International Conference on Learning Representations. find
  28. Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. Preprint, arXiv:2410.20526. find
  29. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  30. Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations. find
  31. Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam S. Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, and 20 others. 2024. Sleep… find
  32. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. Preprint, arXiv:2312.06674.
  33. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361.
  34. Julian Minder, Clément Dumas, Bilal Chughtai, and Neel Nanda. 2025. Latent scaling robustly identifies chat-specific latents in crosscoders. In Sparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference. find
  35. Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Arthur Conmy, Callum McDougall, Kola Ayonrinde, Matthew Wearden, Samuel Marks, and Neel Nanda. 2024. Saebench: A comprehensive benchmark for sparse autoencoders. find
  36. Gouki Minegishi, Hiroki Furuta, Yusuke Iwasawa, and Yutaka Matsuo. 2025. Rethinking evaluation of sparse autoencoders through the representation of polysemous words. In The Thirteenth International Conference on Learning Representations. find
  37. Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. 2024. Interpreting attention layer outputs with sparse autoencoders. arXiv preprint arXiv:2406.17759. nostalgebraist. 2020. interpreting gpt: the logit lens. Less-Wrong. — [arXiv:2406.17759](https://arxiv.org/abs/2406.17759) find
  38. Nhat Minh Le, Neel Patel, Ciyue Shen, Blake Martin, Alfred Eng, Chintan Shah, Sean Grullon, and Dinkar Juyal. 2024. Learning biologically relevant features in a pathology foundation model using sparse autoencoders. In Advancements In Medical Foundation Models: Explainability, Robustness, Security, a… find
  39. Kenneth Li, Yida Chen, Fernanda Viégas, and Martin Wattenberg. 2025. When bad data leads to good models. In Forty-second International Conference on Machine Learning. find
  40. Yuxiao Li, Eric J. Michaud, David D. Baek, Joshua Engels, Xiaoqing Sun, and Max Tegmark. 2024. The geometry of concepts: Sparse autoencoder feature structure. arXiv preprint arXiv:2410.19750. — [arXiv:2410.19750](https://arxiv.org/abs/2410.19750) find
  41. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147. find
  42. Johnny Lin. 2023. Neuronpedia: Interactive reference and tooling for analyzing neural networks. Software available from neuronpedia.org. find
  43. Yizhou Liu, Ziming Liu, and Jeff Gore. 2025. Superposition yields robust neural scaling. arXiv preprint arXiv:2505.10465. — [arXiv:2505.10465](https://arxiv.org/abs/2505.10465) find
  44. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2024. Circuit component reuse across tasks in transformer language models. In The Twelfth International Conference on Learning Representations. find
  45. Jack Merullo, Noah A. Smith, Sarah Wiegreffe, and Yanai Elazar. 2025. On linear representations and pretraining data frequency in language models. In The Thirteenth International Conference on Learning Representations. find
  46. Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2025. Large language models: A survey. Preprint, arXiv:2402.06196.
  47. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. gpt-oss-120b & gpt-oss-20b model…
  48. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card. Pr…
  49. Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. 2025. The geometry of categorical and hierarchical concepts in large language models. In The Thirteenth International Conference on Learning Representations. find
  50. Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning.
  51. Ben Prystawski, Michael Y. Li, and Noah Goodman. 2023. Why think step by step? reasoning emerges from the locality of experience. In Thirty-seventh Conference on Neural Information Processing Systems. find
  52. Gautam Reddy. 2024. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In The Twelfth International Conference on Learning Representations. find
  53. Lee Sharkey, Dan Braun, and Beren Millidge. 2022. Taking features out of superposition with sparse autoencoders. find
  54. Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. https://www.cerebras.net/blog/ slimpajama-a-627b-token-cleaned-and-deduplicated-version Anna Soligo, Edward Turner, Senthooran… — [arXiv:2506.11618](https://arxiv.org/abs/2506.11618) ↗
  55. Viacheslav Surkov, Chris Wendler, Mikhail Terekhov, Justin Deschenaux, Robert West, and Caglar Gulcehre. 2024. Unpacking SDXL turbo: Interpreting text-to-image models with sparse autoencoders. find
  56. Mia Taylor, James Chua, Jan Betley, Johannes Treutlein, and Owain Evans. 2025. School of reward hacks: Hacking harmless tasks generalizes to misaligned behavior in llms. Preprint, arXiv:2508.17511. — [arXiv:2508.17511](https://arxiv.org/abs/2508.17511) find
  57. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, an… find
  58. J.J. Thomson. 1904. Xxiv. on the structure of the atom: an investigation of the stability and periods of oscillation of a number of corpuscles arranged at equal intervals around the circumference of a circle; with application of the results to the theory of atomic structure. The London, Edinburgh, a… find
  59. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248.
  60. Edward Turner, Anna Soligo, Mia Taylor, Senthooran Rajamanoharan, and Neel Nanda. 2025. Model organisms for emergent misalignment. Preprint, arXiv:2506.11613. — [arXiv:2506.11613](https://arxiv.org/abs/2506.11613) find
  61. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593. find
  62. Miles Wang, Tom Dupré la Tour, Olivia Watkins, Alex Makelov, Ryan A. Chi, Samuel Miserendino, Johannes Heidecke, Tejal Patwardhan, and Dan Mossing. 2025a. Persona features control emergent misalignment. Preprint, arXiv:2506.19823. — [arXiv:2506.19823](https://arxiv.org/abs/2506.19823) find
  63. Zifan Wang, Joshua Engels, and Oliver Clive-Griffin. 2025b. Simple mechanistic explanations for out-of-context reasoning. In ICML 2025 Workshop on Reliable and Responsible Foundation Models. find
  64. Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, and Aidong Zhang. 2024. Spurious correlations in machine learning: A survey. Preprint, arXiv:2402.12715. — [arXiv:2402.12715](https://arxiv.org/abs/2402.12715) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/minegishi-2026-understanding-emergent-misalignment-feature-superposition/