Elamur Research⌕
☰
home / papers / 2026 / Spectra: A Mechanistic Interpretability Library for Vision-Language Models

Spectra: A Mechanistic Interpretability Library for Vision-Language Models

Paper · 2026 · Clement Neo, Yongsen Zheng, Kwok-Yan Lam et al.

Summary

Tooling for mechanistic interpretability of vision-language models (VLMs) lags behind text-only tooling such as TransformerLens. Existing VLM tools like vlm-lens mostly record activations and cannot intervene in the model. The difficulty is that adapter-style VLMs differ in how they mark image tokens, how they arrange them, and how their adapters merge patches.

The authors present Spectra, an MIT-licensed Python library built on HuggingFace Transformers. Its HookedVLM wrapper provides context managers for caching activations and running hooked interventions. Per-checkpoint adapter configurations hide model-specific details. The library also offers helpers for locating image tokens, tracking crop and resize metadata, and overlaying the patch grid on an image. A logit lens and a web app are included.

As a case study, the authors analyze object counting on CountBench using ablation, attention blocking and layer-wise activation patching. Benchmarks show little overhead for caching hidden states. This matters because researchers can run causal VLM experiments without writing fragile per-model indexing code.

Key points

  • Supported models: Qwen2.5-VL, Qwen3-VL, LLaVA 1.5 and SmolVLM. Hook names largely follow TransformerLens conventions. The code is at github.com/clemneo/vlm-spectra.
  • Counting experiments:
    • Zeroing count-number positions at the language model input reduced their average number from 30.8 to 11.8.
    • Blocking attention to count tokens changed the answer for only 4 of 100 images.
  • Activation patching (2 vs. 7 children): image-only and text-only patching cross over around layers 15–25. Restricting patching to the final 16 or 32 tokens of the image yields a nearly negligible effect, which suggests the count information is spread across many tokens.
  • Testing has three tiers: unit tests, integration tests on SmolVLM-256M, and per-model acceptance tests. Cached activations are checked against HuggingFace's native outputs.
  • Overhead on 300 CountBench images with LLaVA-1.5-7B:
    • Caching hidden states adds about +2.1% (24.6 s vs. 24.0 s for a plain forward pass).
    • Caching attention patterns adds about 15%.
    • Patching attention heads adds about 72%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanyang Technological UniversitySingapore · Singapore156 works · 25,644 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 worksInterpretability · 130 works

Cite

Cite as Clement Neo, Yongsen Zheng, Kwok-Yan Lam and Luke Ong. 2026. Spectra: A Mechanistic Interpretability Library for Vision-Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-demo.78

@inproceedings{neo2026spectra,
  title = {{Spectra: A Mechanistic Interpretability Library for Vision-Language Models}},
  author = {Clement Neo, Yongsen Zheng, Kwok-Yan Lam and Luke Ong},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-demo.78},
  url = {https://doi.org/10.18653/v1/2026.acl-demo.78}
}

References

4 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, and Vasudev Lal. 2022. Vl-interpret: An interactive visualization tool for interpreting vision-language transformers. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 21406–21415. find
  2. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. Qwen3-vl technical report. Preprint, a…
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. Qwen2.5-vl technical report. Preprint, arXiv:2502.1…
  4. Caching attention patterns HuggingFace 26.7 +11.1% Spectra 30.9 +28.4% Activation intervention (run_with_hooks) Residual-stream patch 25.6 +6.4% Head patch 41.4 +72.3% Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do n… find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/neo-2026-spectra-mechanistic-interpretability-library-vision/