Elamur Research⌕
☰
home / papers / 2026 / From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models

From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models

Paper · 2026 · Ling Shi, Xinwei Wu, Xiaohu Zhao et al.

Summary

Sparse Autoencoders (SAEs) and other mechanistic interpretability tools can expose meaningful features inside LLMs, but these findings are rarely used to improve the models themselves. The authors close this gap with Interpretability-Guided Data Selection (IGDS), which uses a model's own task features to choose fine-tuning data. In the first stage, SAE features that fire on most of a small set of task samples (for example, at least 80%) become candidates. Each candidate is then amplified in the residual stream on a validation set and kept only if it measurably improves task performance. In the second stage, training examples are ranked by how strongly they activate these causally validated features, and the top-ranked "Feature-Resonant Data" are used for supervised fine-tuning. Tests cover math reasoning, summarization and translation on Gemma-2, LLaMA-3.1 and Qwen3. IGDS beats quality- and diversity-oriented selection baselines and can outperform fine-tuning on the full dataset while using only part of it. The work turns interpretability from a purely diagnostic tool into a practical, data-efficient training method.

Key points

  • On the Math task with Gemma-2-2B, training on half of the data selected by IGDS gave a 17.4% gain over full-dataset fine-tuning.
  • Feature identification proved stable on Gemma-2-2B-it across GSM8K, Math500 and OpenR1 as source datasets. F 14,11575 ranked first every time, and three features (F 14,11575, F 24,14448, F 10,5717) appeared in all three top-5 lists.
  • Math features were spread across every layer, whereas Summarization and Translation features clustered in the middle-to-late layers.
  • Amplifying the top feature shifted output vocabulary in task-appropriate ways:
    • Translation: "english", "chinese"
    • Summarization: "summary", "topic"
    • Math: "answer", "final"
  • Results depend on SAE quality. The official Gemma-Scope SAEs produced the largest gains, while Qwen3-8B, whose custom SAEs covered only some layers, improved least. The authors plan to train and release SAEs for its remaining layers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China168 works · 15,386 citations🎓Tianjin UniversityTianjin · China41 works · 1,128 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksInterpretability · 130 works

Cite

Cite as Ling Shi, Xinwei Wu, Xiaohu Zhao, Hao Wang, Heng Liu, Yangyang Liu, Linlong Xu, Longyue Wang, Deyi Xiong and Weihua Luo. 2026. From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.283

@inproceedings{shi2026insight,
  title = {{From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models}},
  author = {Ling Shi, Xinwei Wu, Xiaohu Zhao, Hao Wang, Heng Liu, Yangyang Liu, Linlong Xu, Longyue Wang, Deyi Xiong and Weihua Luo},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.283},
  url = {https://doi.org/10.18653/v1/2026.acl-long.283}
}

References

54 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. Advances in Neural Information Processing Systems, 37:136037–136083.
  2. Alexander Bukharin, Shiyang Li, Zhengyang Wang, Jingfeng Yang, Bing Yin, Xian Li, Chao Zhang, Tuo Zhao, and Haoming Jiang. 2024. Data diversity matters for robust instruction tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3411–3425. find
  3. Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. 2024. AlpaGasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austri…
  4. Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. DialogSum: A real-life scenario dialogue summarization dataset. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5062–5074.
  5. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600. find
  6. Ruixuan Deng, Xiaoyang Hu, Miles Gilberti, Shane Storks, Aman Taxali, Mike Angstadt, Chandra Sripada, and Joyce Chai. 2025. Sparse feature coactivation reveals composable semantic modules in large language models. arXiv preprint arXiv:2506.18141. — [arXiv:2506.18141](https://arxiv.org/abs/2506.18141) find
  7. Daniel Deutsch, Eleftheria Briakou, Isaac Rayburn Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, and Markus Freitag. 2025. WMT24++: expanding…
  8. Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. 2024. Transcoders find interpretable llm feature circuits. Advances in Neural Information Processing Systems, 37:24375–24410. find
  9. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger B. Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superposition. CoRR, ab… find
  10. Junfeng Fang, Zac Bi, Ruipeng Wang, Houcheng Jiang, Yuan Gao, Kun Wang, An Zhang, Jie Shi, Xiang Wang, and Tat-Seng Chua. 2024. Towards neuron attributions in multi-modal large language models. Advances in Neural Information Processing Systems, 37:122867–122890. find
  11. Eoin Farrell, Yeu-Tong Lau, and Arthur Conmy. 2024. Applying sparse autoencoders to unlearn knowledge in language models. In Neurips Safe Generative AI Workshop 2024. find
  12. Javier Ferrando, Oscar Balcells Obeso, Senthooran Rajamanoharan, and Neel Nanda. 2024. Do i know this entity? knowledge awareness and hallucinations in language models. In The Thirteenth International Conference on Learning Representations. find
  13. Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024a. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations. find
  14. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024b. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093. find
  15. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. 2024. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pages 160–187. PMLR. find
  16. Shaona Ghosh, Amrita Bhattacharjee, Yftah Ziser, and Christopher Parisien. 2025. A simple yet effective method for non-refusing context relevant fine-grained safety steering in llms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 35116–35136. find
  17. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610. find
  18. Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. CoRR, abs/2410.20526. find
  19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).
  20. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popovi… — Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet
  21. Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2024. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of th…
  22. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147. find
  23. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
  24. Chuang Liu, Linhao Yu, Jiaxuan Li, Renren Jin, Yufei Huang, Ling Shi, Junhui Zhang, Xinmeng Ji, Tingting Cui, Tao Liu, Jinwang Song, Hongying Zan, Sun Li, and Deyi Xiong. 2024. OpenEval: Benchmarking chinese llms across capability, alignment and safety. In Proceedings of the 62nd Annual Meeting of t… find
  25. Jingcheng Niu, Andrew Liu, Zining Zhu, and Gerald Penn. 2024. What does the knowledge neuron thesis have to do with knowledge? In The Twelfth International Conference on Learning Representations. find
  26. Leiyu Pan, Bojian Xiong, Lei Yang, Renren Jin, Shaowei Zhang, Yue Chen, Ling Shi, Jiang Zhou, Junru Wu, Zhen Wang, Jianxiang Peng, Juesi Xiao, Tianyu Dong, Zhuowen Han, Zhuo Chen, Yuqi Ren, and Deyi Xiong. 2025. Advancing large language models for tibetan with curated data and continual pre-training… — [arXiv:2507.09205](https://arxiv.org/abs/2507.09205) find
  27. Jianxiang Peng, Ling Shi, Xinwei Wu, Hanwen Zhang, Fujiang Liu, Haocheng Lyu, and Deyi Xiong. 2025. Diplomacyagent: Do llms balance interests and ethical principles in international events? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou,… find
  28. Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. 2024. A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646. find
  29. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for mt evaluation. arXiv preprint arXiv:2009.09025.
  30. Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeffrey Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Isaac Bloom, Stella Biderman, Adrià Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller,… find
  31. Trans. Mach. Learn. Res., 2025. find
  32. Ling Shi, Yuqin Dai, Ziyin Wang, Ning Gao, Wei Zhang, Chaozheng Wang, Yujie Wang, Wei He, Jinpeng Wang, and Deiyi Xiong. 2026. SAGE: A service agent graph-guided evaluation benchmark. Preprint, arXiv:2604.09285. — [arXiv:2604.09285](https://arxiv.org/abs/2604.09285) find
  33. Dengcan Liu, Jiahao Li, Zheren Fu, Yi Tu, Jiajun Li, Zhendong Mao, and Yongdong Zhang. 2025a. SparseRM: A lightweight preference modeling with sparse autoencoder. arXiv preprint arXiv:2511.07896. — [arXiv:2511.07896](https://arxiv.org/abs/2511.07896) find
  34. Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. 2025. A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models. arXiv preprint arXiv:2503.05613. find
  35. Yan Liu, Renren Jin, Ling Shi, Zheng Yao, and Deyi Xiong. 2025b. FineMath: A fine-grained mathematical evaluation benchmark for chinese large language models. ACM Trans. Asian Low Resour. Lang. Inf. Process., 24(12):139:1–139:15. find
  36. Hantao Lou, Changye Li, Jiaming Ji, and Yaodong Yang. 2025. SAE-V: Interpreting multimodal models for enhanced alignment. In Forty-second International Conference on Machine Learning. find
  37. Haoran Sun, Renren Jin, Shaoyang Xu, Leiyu Pan, Supryadi, Menglong Cui, Jiangcun Du, Yikun Lei, Lei Yang, Ling Shi, Juesi Xiao, Shaolin Zhu, and Deyi Xiong. 2024. FuxiTranyu: A multilingual large language model trained with balanced data. In Proceedings of the 2024 Conference on Empirical Methods in… find
  38. Yin Lu, Xuening Zhu, Tong He, and David Wipf. 2025. Sparse autoencoders, again? In Forty-second International Conference on Machine Learning. find
  39. Gemma Team. 2024a. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.00118. find
  40. Llama Team. 2024b. The llama 3 herd of models. CoRR, abs/2407.21783.
  41. Qwen Team. 2025. Qwen3 technical report. CoRR, abs/2505.09388.
  42. A. Templeton, T. Conerly, J. Marcus, J. Lindsey, T. Bricken, B. Chen, A. Pearce, C. Citro, E. Ameisen, A. Jones, H. Cunningham, N. L. Turner, C. Mc-Dougall, M. MacDiarmid, C. D. Freeman, T. R. Sumers, E. Rees, J. Batson, A. Jermyn, and 3 others. 2024. Scaling monosemanticity: Extracting interpretabl… find
  43. Xinwei Wu, Weilong Dong, Shaoyang Xu, and Deyi Xiong. 2024. Mitigating privacy seesaw in large language models: Augmented privacy neuron editing via activation patching. In Findings of the Association for Computational Linguistics ACL 2024, pages 5319–5332. find
  44. Xinwei Wu, Heng Liu, Xiaohu Zhao, Yuqi Ren, Linlong Xu, Longyue Wang, Deyi Xiong, Weihua Luo, and Kaifu Zhang. 2026. Finding the translation switch: Discovering and exploiting the task-initiation features in llms. In Fortieth AAAI Conference on Artificial Intelligence, Thirty-Eighth Conference on In… find
  45. Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024a. LESS: selecting influential data for targeted instruction tuning. In Proceedings of the 41st International Conference on Machine Learning, pages 54104–54132.
  46. Mingjia Yin, Chuhan Wu, Yufei Wang, Hao Wang, Wei Guo, Yasheng Wang, Yong Liu, Ruiming Tang, Defu Lian, and Enhong Chen. 2024. Entropy law: The story behind data compression and llm performance. arXiv e-prints, pages arXiv–2407. find
  47. Zhongshen Zeng, Pengguang Chen, Haiyun Jiang, and Jiaya Jia. 2023. Challenge llms to reason about reasoning: A benchmark to unveil cognitive depth in llms. find
  48. Shaowei Zhang and Deyi Xiong. 2025. BackMATH: Towards backward reasoning for solving math problems step by step. In Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025 - Industry Track, Abu Dhabi, UAE, January 19-24, 2025, pages 466–482. Association for Computa… find
  49. Shuyi Zhang, Wei Shi, Sihang Li, Jiayi Liao, Tao Liang, Hengxing Cai, and Xiang Wang. 2025. Interpretable reward model via sparse autoencoder. arXiv preprint arXiv:2508.08746. — [arXiv:2508.08746](https://arxiv.org/abs/2508.08746) find
  50. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: less is more for alignment. In Advances in Neural Information Processing Systems 36: Annual Confere… find
  51. Tingyu Xia, Bowen Yu, Kai Dang, An Yang, Yuan Wu, Yuan Tian, Yi Chang, and Junyang Lin. 2024b. Rethinking data selection at scale: Random selection is almost all you need. arXiv preprint arXiv:2410.09335. find
  52. Lei Yang, Renren Jin, Ling Shi, Jianxiang Peng, Yue Chen, and Deyi Xiong. 2025. ProBench: Benchmarking large language models in competitive programming. CoRR, abs/2502.20868. — [arXiv:2502.20868](https://arxiv.org/abs/2502.20868) find
  53. Lei Yang, Leiyu Pan, Bojian Xiong, Renren Jin, Shaowei Zhang, Yue chen, Jiang Zhou Ling Shi, Junru Wu, Zhen Wang, Jianxiang Peng, Juesi Xiao, Tianyu Dong, Zhuowen Han, Zhuo Chen, Yuqi Ren, and Deyi Xion. 2026. From curated data to scalable models: Continual pre-training of dense and moe large langua…
  54. Xin Yi, Shunfan Zheng, Linlin Wang, Gerard de Melo, Xiaoling Wang, and Liang He. 2025. NLSR: Neuron-level safety realignment of large language models against harmful fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25706–25714. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2026-insight-action-novel-framework-interpretability/