Elamur Research⌕
☰
home / papers / 2026 / Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks

Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks

Paper · 2026 · Yu Wang and Sharon Li

Summary

The paper examines why in-context learning in multimodal LLMs gains less from demonstrations than text-only ICL. The authors built a controlled outlier-detection task in which the model must name the single item that differs from the rest by shape or color. Every problem exists in two versions: one written as text and one shown as an image. Only the input modality changes between the two.

On Qwen2.5-VL and Gemma-3, zero-shot accuracy is roughly the same across modalities. With 4-shot demonstrations, however, the multimodal version falls well behind the text version.

To find the cause, the authors split ICL into two stages:

  • Task mapping construction: in middle layers, demonstration labels attach to the correct visual evidence, and this happens even when the final answer is wrong.
  • Task mapping transfer: this is where things break. The query's attention to the demonstrations peaks in later layers, but by then the mapping is no longer represented accurately, so the model answers mostly from what it sees in the query image.

The authors propose MGI, an inference-time attention intervention that reuses the mid-layer mapping. It produces small but consistent gains. The study identifies a specific bottleneck in multimodal ICL and points to aligning perception with reasoning as a direction for future work.

Key points

  • Task design: the label space has 4 shapes and 10 colors. In few-shot prompts, the model must work out from the demonstrations whether shape or color is the relevant feature.
  • 4-shot performance gap (multimodal vs. text-only):
    • Qwen2.5-VL-7B: 63.60 vs. 88.40 (−24.80)
    • Gemma-3-12B: 72.93 vs. 91.40 (−18.47)
    • Gemma-3-27B: −10.87
    • Qwen2.5-VL-32B: −4.34
  • Where it fails: construction and transfer of the task mapping happen at different depths in the network. The mapping is grounded at a middle "peak grounding layer," but the query relies on it only later, when the mapping has degraded.
  • How MGI works: it extracts the label-to-image attention from the peak layer and adds it, scaled by λ, to the query-to-image attention in layers beyond L_start. The attention scores are then renormalized.
  • MGI results: evaluated on TrueMICL (Outlier Detection, Clock Math, Operator Induction) and OK-VQA, averaged over 3 seeds.
    • Gains are modest, about 0–1 point on most tasks.
    • Examples: Outlier 69.09→70.17 for Qwen2.5-VL-7B; OK-VQA 48.39→49.51 for Gemma-3-12B.
    • The authors note that MGI does not fix the underlying architectural misalignment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Wisconsin–MadisonMadison · United States42 works · 48,486 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Yu Wang and Sharon Li. 2026. Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.622

@inproceedings{wang2026why,
  title = {{Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks}},
  author = {Yu Wang and Sharon Li},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.622},
  url = {https://doi.org/10.18653/v1/2026.acl-long.622}
}

References

49 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. Preprint, arXiv:2502.13…
  2. Folco Bertini Baldassini, Mustafa Shukor, Matthieu Cord, Laure Soulier, and Benjamin Piwowarski. 2024. What makes multimodal in-context learning work? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1539– 1550. find
  3. Shuo Chen, Zhen Han, Bailan He, Mark Buckley, Qin Yao, Philip Torr, Volker Tresp, and Jindong Gu. 2023. Can multimodal large language models truly perform multimodal in-context learning? arXiv preprint arXiv:2311.18021. — [arXiv:2311.18021](https://arxiv.org/abs/2311.18021) find
  4. Shuo Chen, Zhen Han, Bailan He, Jianzhe Liu, Mark Buckley, Yao Qin, Philip Torr, Volker Tresp, and Jindong Gu. 2024a. Can multimodal large language models truly perform multimodal in-context learning? Preprint, arXiv:2311.18021. — [arXiv:2311.18021](https://arxiv.org/abs/2311.18021) find
  5. Shuo Chen, Jianzhe Liu, Zhen Han, Yan Xia, Daniel Cremers, Philip Torr, Volker Tresp, and Jindong Gu. 2025a. True multimodal in-context learning needs attention to the visual context. Preprint, arXiv:2507.15807. — [arXiv:2507.15807](https://arxiv.org/abs/2507.15807) find
  6. Song Chen, Xinyu Guo, Yadong Li, Tao Zhang, Mingan Lin, Dongdong Kuang, Youwei Zhang, Lingfeng Ming, Fengyu Zhang, Yuran Wang, and 1 others. 2025b. Ocean-ocr: Towards general ocr application via a vision-language model. arXiv preprint arXiv:2501.15558. — [arXiv:2501.15558](https://arxiv.org/abs/2501.15558) find
  7. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024b. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and P…
  8. Hakaze Cho, Mariko Kato, Yoshihiro Sakai, and Naoya Inoue. 2025. Revisiting in-context learning inference circuit in large language models. Preprint, arXiv:2410.04468. — [arXiv:2410.04468](https://arxiv.org/abs/2410.04468) find
  9. Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. 2023. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models. find
  10. Sivan Doveh, Shaked Perek, M Jehanzeb Mirza, Amit Alfassy, Assaf Arbelle, Shimon Ullman, and Leonid Karlinsky. 2024. Towards multimodal in-context learning for vision & language models. arXiv preprint arXiv:2403.12736. — [arXiv:2403.12736](https://arxiv.org/abs/2403.12736) find
  11. Jun Gao, Qian Qiao, Ziqiang Cao, Zili Wang, and Wenjie Li. 2024. Aim: Let any multi-modal large language models embrace efficient in-context learning. Preprint, arXiv:2406.07588. — [arXiv:2406.07588](https://arxiv.org/abs/2406.07588) find
  12. Jun Gao, Qian Qiao, Tianxiang Wu, Zili Wang, Ziqiang Cao, and Wenjie Li. 2025. Aim: Let any multimodal large language models embrace efficient in-context learning. In Proceedings of the AAAI Conference on Artificial Intelligence, number 3 in 39, pages 3077– 3085. find
  13. Chi Han, Ziqi Wang, Han Zhao, and Heng Ji. 2023a. Explaining emergent in-context learning as kernel regression. arXiv preprint arXiv:2305.12766.
  14. Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang. 2023b. Understanding in-context learning via supportive pretraining data. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12660–… find
  15. Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, and Roei Herzig. 2024. Multimodal task vectors enable many-shot multimodal in-context learning. Advances in Neural Information Processing Systems, 37:22124–22153. find
  16. Chengyue Huang, Yuchen Zhu, Sichen Zhu, Jingyun Xiao, Moises Andrade, Shivang Chopra, and Zsolt Kira. 2025. Mimicking or reasoning: Rethinking multi-modal in-context learning in vision-language models. arXiv preprint arXiv:2506.07936. — [arXiv:2506.07936](https://arxiv.org/abs/2506.07936) find
  17. Hong Jun Jeon, Jason D Lee, Qi Lei, and Benjamin Van Roy. 2024. An information-theoretic analysis of in-context learning. In Forty-first International Conference on Machine Learning. find
  18. Hongrui Jia, Chaoya Jiang, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, and Shikun Zhang. 2024. Symdpo: Boosting in-context learning of large multimodal models with symbol demonstration direct preference optimization. arXiv preprint arXiv:2411.11909. — [arXiv:2411.11909](https://arxiv.org/abs/2411.11909) find
  19. Jannik Kossen, Yarin Gal, and Tom Rainforth. 2024. In-context learning learns label relationships but is not conventional learning. In The Twelfth International Conference on Learning Representations. find
  20. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. 2023a. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425. — [arXiv:2306.05425](https://arxiv.org/abs/2306.05425) find
  21. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023b. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726.
  22. Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. 2023c. How to configure good in-context sequence for visual question answering. arXiv preprint arXiv:2312.01571. — [arXiv:2312.01571](https://arxiv.org/abs/2312.01571) find
  23. Yanshu Li. 2025. Advancing multimodal in-context learning in large vision-language models with task-aware demonstrations. Preprint, arXiv:2503.04839. — [arXiv:2503.04839](https://arxiv.org/abs/2503.04839) find
  24. Yanshu Li, Jianjiang Yang, Zhennan Shen, Ligong Han, Haoyan Xu, and Ruixiang Tang. 2025a. Catp: Contextually adaptive token pruning for efficient and enhanced multimodal in-context learning. Preprint, arXiv:2508.07871. — [arXiv:2508.07871](https://arxiv.org/abs/2508.07871) find
  25. Yanshu Li, Jianjiang Yang, Ziteng Yang, Bozheng Li, Hongyang He, Zhengtao Yao, Ligong Han, Yingjie Victor Chen, Songlin Fei, Dongfang Liu, and Ruixiang Tang. 2025b. Cama: Enhancing multimodal in-context learning with context-aware modulated attention. Preprint, arXiv:2505.17097. — [arXiv:2505.17097](https://arxiv.org/abs/2505.17097) find
  26. Yanshu Li, Jianjiang Yang, Tian Yun, Pinyuan Feng, Jinfa Huang, and Ruixiang Tang. 2025c. Taco: Enhancing multimodal in-context learning via task mapping-guided sequence configuration. Preprint, arXiv:2505.17098.
  27. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llava-next: Improved reasoning, ocr, and world knowledge. find
  28. Zhining Liu, Ziyi Chen, Hui Liu, Chen Luo, Xianfeng Tang, Suhang Wang, Joy Zeng, Zhenwei Dai, Zhan Shi, Tianxin Wei, Benoit Dumoulin, and Hanghang Tong. 2025. Seeing but not believing: Probing the disconnect between visual attention and answer correctness in vlms. Preprint, arXiv:2510.17771. — [arXiv:2510.17771](https://arxiv.org/abs/2510.17771) find
  29. Yang Luo, Zangwei Zheng, Zirui Zhu, and Yang You. 2024. How does the textual information affect the retrieval of multimodal in-context learning? arXiv preprint arXiv:2404.12866. find
  30. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR). find
  31. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? Preprint, arXiv:2202.12837.
  32. Yaniv Nikankin, Dana Arad, Yossi Gandelsman, and Yonatan Belinkov. 2025. Same task, different circuits: Disentangling modality-specific mechanisms in vlms. Preprint, arXiv:2506.09047. — [arXiv:2506.09047](https://arxiv.org/abs/2506.09047) find
  33. Jane Pan. 2023. What in-context learning “learns” in-context: Disentangling task recognition and task learning. Master’s thesis, Princeton University.
  34. Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. 2024a. What factors affect multimodal in-context learning? an in-depth exploration. Preprint, arXiv:2410.20482. — [arXiv:2410.20482](https://arxiv.org/abs/2410.20482) find
  35. Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. 2024b. What factors affect multimodal in-context learning? an in-depth exploration. arXiv preprint arXiv:2410.20482. — [arXiv:2410.20482](https://arxiv.org/abs/2410.20482) find
  36. Aaditya Singh, Stephanie Chan, Ted Moskovitz, Erin Grant, Andrew Saxe, and Felix Hill. 2024a. The transient nature of emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 36. find
  37. Aaditya K Singh, Ted Moskovitz, Felix Hill, Stephanie CY Chan, and Andrew M Saxe. 2024b. What needs to go right for an induction head? a mechanistic study of in-context learning circuits and their formation. In Forty-first International Conference on Machine Learning. find
  38. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  39. Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. Label words are anchors: An information flow perspective for understanding in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9840–9855.
  40. Daiqing Wu, Dongbao Yang, Sicheng Zhao, Can Ma, and Yu Zhou. 2025. An empirical study on configuring in-context learning demonstrations for unleashing mllms’ sentimental perception capability. arXiv preprint arXiv:2505.16193. — [arXiv:2505.16193](https://arxiv.org/abs/2505.16193) find
  41. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080.
  42. Nan Xu, Fei Wang, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024. From introspection to best practices: Principled analysis of demonstrations in multimodal in-context learning. arXiv preprint arXiv:2407.00902. find
  43. Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. 2024. Exploring diverse in-context configurations for image captioning. Advances in Neural Information Processing Systems, 36. find
  44. Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. 2023a. Trained transformers learn linear models in-context. arXiv preprint arXiv:2306.09927. — [arXiv:2306.09927](https://arxiv.org/abs/2306.09927) find
  45. Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. 2023b. What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems, 36:17773–17794. find
  46. Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. 2023. Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv preprint arXiv:2309.07915.
  47. Bowen Zheng, Ming Ma, Zhongqiao Lin, and Tianming Yang. 2024. Distributed rule vectors is a key mechanism in large language models’ in-context learning. arXiv preprint arXiv:2406.16007. — [arXiv:2406.16007](https://arxiv.org/abs/2406.16007) find
  48. Yuxiang Zhou, Jiazheng Li, Yanzheng Xiang, Hanqi Yan, Lin Gui, and Yulan He. 2024. The mystery of in-context learning: A comprehensive survey on interpretation and analysis. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14365–14378, Miami, Florida,… find
  49. Yongshuo Zong, Ondrej Bohdal, and Timothy Hospedales. 2024. Vl-icl bench: The devil in the details of benchmarking multimodal in-context learning. arXiv preprint arXiv:2403.13164. — [arXiv:2403.13164](https://arxiv.org/abs/2403.13164) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-why-multimodal-context-learning-lags/