Elamur Research⌕
☰
home / papers / 2025 / TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration

TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration

Paper · 2025 · Yanshu Li, Jianjiang Yang, Tian Yun et al.

Summary

Large vision–language models (LVLMs) can learn tasks from interleaved image–text demonstrations, but their in-context learning (ICL) is fragile. Small changes to the chosen demonstrations (ICDs) can noticeably shift outputs, and existing selection methods rely on hand-crafted relevance metrics.

The authors propose "task mapping" as a way to explain this. Each demonstration encodes a local mapping from image and query to answer, and the model combines these into a global mapping that it applies to the query. Probing experiments on OpenFlamingov2-9B and Idefics2-8B, mainly using HatefulMemes, show three things:

  • Clearer mappings improve accuracy.
  • The query sample has an outsized influence.
  • Coherent demonstration sets matter most for open-ended, complex tasks.

Building on this, they introduce TACO (Task-Aware model for in-COntext Learning). It is a small transformer that uses task-aware attention to select and order demonstrations autoregressively. It is trained on sequences produced by an Oracle that approximates the target LVLM's preferences. Across five LVLMs and nine datasets, TACO beats prior configuration strategies, with the largest gains on generalized-mapping tasks. This offers a model-centric, interpretable route to better multimodal prompting.

Key points

  • Two task types. Specific-mapping tasks (e.g., HatefulMemes, CLEVR) follow one fixed reasoning path. Generalized-mapping tasks (e.g., open-ended VQA) require several local mappings to be merged into one global mapping.
  • Probing setups.
    • Easier Mapping adds the hint "Is it hateful?" and gives Easier Mapping > Standard > Harder Mapping at every shot count.
    • Harder Mapping swaps yes/no labels for foo/bar.
    • Wrong Labels inverts 75% of labels.
    • Blurred Images applies Gaussian blur.
    • An "in-context lens" derived from logit lens tracks the model's internal reasoning.
  • Shot count. Task-aware attention helps for every combination of training shots (N) and inference shots (n). Gains taper on specific-mapping tasks as n grows but stay steady on generalized-mapping tasks.
  • Training data. Oracle-built training data works best. For example, one benchmark scores 66.75 with Oracle versus 62.38 with RS. TACO paired with RS, I2I or IQ2IQ also beats each of those retrievers used alone.
  • Appendix extensions. The appendices test TACO on NLP and text-to-image tasks. They also include ablations on input embeddings and on the format and position of the instruction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States241 works · 54,053 citations🎓Brown UniversityProvidence · United States76 works · 25,952 citations🎓University of RochesterRochester · United States36 works · 7,398 citations🎓Rutgers, The State University of New JerseyNew Brunswick · United States36 works · 12,026 citations🎓University of BristolBristol · United Kingdom14 works · 725 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Yanshu Li, Jianjiang Yang, Tian Yun, Pinyuan Feng, Jinfa Huang and Ruixiang Tang. 2025. TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.39

@inproceedings{li2025taco,
  title = {{TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration}},
  author = {Yanshu Li, Jianjiang Yang, Tian Yun, Pinyuan Feng, Jinfa Huang and Ruixiang Tang},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.39},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.39}
}

References

58 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  2. Rahul Atul Bhope, Praveen Venkateswaran, KR Jayaram, Vatche Isahagian, Vinod Muthusamy, and Nalini Venkatasubramanian. 2025. Optiseq: Ordering examples on-the-fly for in-context learning. arXiv preprint arXiv:2501.15030. find
  3. Haoran Chen, Junyan Lin, Xinhao Chen, Yue Fan, Xin Jin, Hui Su, Jianfeng Dong, Jinlan Fu, and Xiaoyu Shen. 2025. Rethinking visual layer selection in multimodal llms. arXiv preprint arXiv:2504.21447. — [arXiv:2504.21447](https://arxiv.org/abs/2504.21447) find
  4. Wentong Chen, Yankai Lin, ZhenHao Zhou, HongYun Huang, Yantao Jia, Zhao Cao, and Ji-Rong Wen. 2024a. Icleval: evaluating in-context learning ability of large language models. arXiv preprint arXiv:2406.14955.
  5. Yunmo Chen, Tongfei Chen, Harsh Jhamtani, Patrick Xia, Richard Shin, Jason Eisner, and Benjamin Van Durme. 2024b. Learning to retrieve iteratively for in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7156–7168. find
  6. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. 2024c. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271.
  7. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. A survey on in-context learning. arXiv preprint arXiv:2301.00234.
  8. Caoyun Fan, Jidong Tian, Yitian Li, Hao He, and Yaohui Jin. 2024. Comparable demonstrations are important in in-context learning: A novel perspective on demonstration selection. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10436–10440… find
  9. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736.
  10. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723.
  11. Ruichuan An, Sihan Yang, Ming Lu, Renrui Zhang, Kai Zeng, Yulin Luo, Jiajun Cao, Hao Liang, Ying Chen, Qi She, et al. 2024. Mc-llava: Multi-concept personalized vision-language model. arXiv preprint arXiv:2411.11706. — [arXiv:2411.11706](https://arxiv.org/abs/2411.11706) find
  12. Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023. Openflamingo: An open-source framework for training…
  13. Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. 2023. What can transformers learn in-context? a case study of simple function classes. arXiv preprint arXiv:2208.01066. find
  14. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913.
  15. Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng. 2025. Breaking the modality barrier: Universal embedding learning with multimodal llms. arXiv preprint arXiv:2504.17432. — [arXiv:2504.17432](https://arxiv.org/abs/2504.17432) find
  16. Qi Guo, Leiyu Wang, Yidong Wang, Wei Ye, and Shikun Zhang. 2024. What makes a good order of examples in in-context learning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14892–14904, Bangkok, Thailand. Association for Computational Linguistics. find
  17. Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. 2024b. How to configure good in-context sequence for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26710–26720. find
  18. Wei Li, Renshan Zhang, Rui Shao, Jie He, and Liqiang Nie. 2025b. Cogvla: Cognition-aligned vision-language-action model via instruction-driven routing & sparsification. arXiv preprint arXiv:2508.21046. — [arXiv:2508.21046](https://arxiv.org/abs/2508.21046) find
  19. Yanshu Li. 2025. Advancing multimodal in-context learning in large vision-language models with task-aware demonstrations. arXiv preprint arXiv:2503.04839. — [arXiv:2503.04839](https://arxiv.org/abs/2503.04839) find
  20. Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617. find
  21. Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709.
  22. Dan Iter, Reid Pryzant, Ruochen Xu, Shuohang Wang, Yang Liu, Yichong Xu, and Chenguang Zhu. 2023. In-context demonstration selection with cross entropy difference. arXiv preprint arXiv:2305.14726. find
  23. Hong Jun Jeon, Jason D Lee, Qi Lei, and Benjamin Van Roy. 2024. An information-theoretic analysis of in-context learning. arXiv preprint arXiv:2401.15530. — [arXiv:2401.15530](https://arxiv.org/abs/2401.15530) find
  24. Jiale Kang, Ziyin Yue, Qingyu Yin, Jiang Rui, Weile Li, Zening Lu, and Zhouran Ji. 2025. Modrwkv: Transformer multimodality in linear time. arXiv preprint arXiv:2505.14505. find
  25. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33:2611–2624.
  26. Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? Advances in Neural Information Processing Systems, 37:87874–87907.
  27. Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, et al. 2024a. A survey on benchmarks of multimodal large language models. arXiv preprint arXiv:2408.08632. — [arXiv:2408.08632](https://arxiv.org/abs/2408.08632) find
  28. Jiaqian Li, Qisheng Hu, Jing Li, and Wenya Wang. 2025a. Stare at the structure: Steering icl exemplar selection with structural alignment. arXiv preprint arXiv:2508.20944. find
  29. Yanshu Li, Yi Cao, Hongyang He, Qisen Cheng, Xiang Fu, Xi Xiao, Tianyang Wang, and Ruixiang Tang. 2025c. M 2 iv: Towards efficient and fine-grained multimodal in-context learning via representation engineering. In Second Conference on Language Modeling. find
  30. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pag…
  31. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804.
  32. Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi. 2024a. In-context learning with retrieved demonstrations for language models: a survey. arXiv preprint arXiv:2401.11624.
  33. Meng Luo, Hao Fei, Bobo Li, Shengqiong Wu, Qian Liu, Soujanya Poria, Erik Cambria, Mong-Li Lee, and Wynne Hsu. 2024b. Panosent: A panoptic sextuple extraction benchmark for multimodal conversational aspect-based sentiment analysis. In Proceedings of the 32nd ACM International Conference on Multimedi… find
  34. Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Z-icl: zero-shot in-context learning with pseudo-demonstrations. arXiv preprint arXiv:2212.09865. find
  35. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204. find
  36. Reid McIlroy-Young, Katrina Brown, Conlan Olson, Linjun Zhang, and Cynthia Dwork. 2024. Order-independence without fine tuning. Advances in Neural Information Processing Systems, 37:72818– 72839. find
  37. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: what makes in-context learning work? arXiv preprint arXiv:2202.12837.
  38. Shiwen Ni, Dingwei Chen, Chengming Li, Xiping Hu, Ruifeng Xu, and Min Yang. 2023. Forgetting before learning: Utilizing parametric arithmetic for knowledge updating in large language models. arXiv preprint arXiv:2311.08011. nostalgebraist. 2020. Interpreting gpt: the logit lens. LessWrong. find
  39. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. In-context learning and induction heads. arXiv preprint arXiv:2209.11895.
  40. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  41. Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. 2023. What in-context learning “learns” in-context: Disentangling task recognition and task learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8298–8319, Toronto, Canada. Association for Computational Linguistics…
  42. Zhenyu Pan, Yutong Zhang, Jianshu Zhang, Haoran Lu, Haozheng Luo, Yuwei Han, Philip S Yu, Manling Li, and Han Liu. 2025. Fairreason: Balancing reasoning and social bias in mllms. arXiv preprint arXiv:2507.23067. — [arXiv:2507.23067](https://arxiv.org/abs/2507.23067) find
  43. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146–162. Springer.
  44. Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. 2023. Multimodal neurons in pretrained text-only transformers. In 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 2854–2859. find
  45. Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2024. Emu: generative pretraining in multimodality. arXiv preprint arXiv:2307.05222. — [arXiv:2307.05222](https://arxiv.org/abs/2307.05222) find
  46. Minh-Hao Van, Xintao Wu, et al. 2024. In-context learning demonstration selection via influence analysis. arXiv preprint arXiv:2402.11750. — [arXiv:2402.11750](https://arxiv.org/abs/2402.11750) find
  47. Liang Wang, Nan Yang, and Furu Wei. 2024. Learning to retrieve in-context examples for large language models. arXiv preprint arXiv:2307.07164.
  48. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846.
  49. Yang Wu, Shilong Wang, Hao Yang, Tian Zheng, Hongbo Zhang, Yanyan Zhao, and Bing Qin. 2023a. An early evaluation of gpt-4v(ision). arXiv preprint arXiv:2310.16534. — [arXiv:2310.16534](https://arxiv.org/abs/2310.16534) find
  50. Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. 2023b. Self-adaptive in-context learning: an information compression perspective for in-context example selection and ordering. arXiv preprint arXiv:2212.10375. find
  51. Xu Yang, Yingzhe Peng, Haoxuan Ma, Shuo Xu, Chi Zhang, Yucheng Han, and Hanwang Zhang. 2024. Lever lm: configuring in-context sequence to lever large vision language models. arXiv preprint arXiv:2312.10104. — [arXiv:2312.10104](https://arxiv.org/abs/2312.10104) find
  52. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. find
  53. Yu Yuan, Lili Zhao, Kai Zhang, Guangting Zheng, and Qi Liu. 2024. Do llms overcome shortcut learning? an evaluation of shortcut challenges in large language models. arXiv preprint arXiv:2410.13343. find
  54. Anhao Zhao, Fanghua Ye, Jinlan Fu, and Xiaoyu Shen. 2024. Unveiling in-context learning: a coordinate system to understand its working mechanism. arXiv preprint arXiv:2407.17011. find
  55. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625.
  56. Yucheng Zhou, Xiang Li, Qianning Wang, and Jianbing Shen. 2024. Visual in-context learning for large vision-language models. arXiv preprint arXiv:2402.11574. find
  57. Yongshuo Zong, Ondrej Bohdal, and Timothy Hospedales. 2024. Vl-icl bench: the devil in the details of multimodal in-context learning. arXiv preprint arXiv:2403.13164. — [arXiv:2403.13164](https://arxiv.org/abs/2403.13164) find
  58. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2025-taco-enhancing-multimodal-context-learning/