Otter: A Multi-Modal Model with In-Context Instruction Tuning
Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2305.03726
Summary
Most large multimodal models (LMMs) are tuned to answer one instruction about one image, or at most to draw on earlier dialogue turns. Few use paired images and text as in-context demonstrations. Otter addresses this gap. It is built on the Flamingo design, via OpenFlamingo, with its Perceiver resampler. The model is instruction-tuned so that each query arrives with related multimodal examples, a format that mirrors the interleaved image-text data Flamingo saw during pretraining. To support this, the authors assemble MIMIC-IT, a resource of more than three million instruction–response pairs (roughly 2.2 million distinct) covering both images and videos. Every entry is linked to curated contextual examples. Otter accepts text, several images or video as input. The authors report that training with in-context examples speeds up convergence and improves generalization. They also find that MIMIC-IT's broad coverage helps Otter reason over video and multiple images. The work matters because it provides a large instruction dataset and shows that matching the tuning format to the pretraining format pays off.
Key points
- Otter extends Flamingo/OpenFlamingo. A Perceiver resampler converts images into visual tokens, and the model is trained with token-level supervised fine-tuning on (image, instruction, response) triples.
- MIMIC-IT (MultI-Modal In-Context Instruction Tuning) contains over 3M instruction–response pairs with about 2.2M unique instructions. Each instance has attached multimodal context.
- Context types include retrieved image-text examples, task descriptions, multi-turn conversations and related video clips. Sources and subsets include LA-I2I, LA-T2T, Dense Caption, TV Shows, Ego4D, VST, Scene Navigation and Spot Difference.
- The design aligns with Flamingo's upstream pretraining on interleaved corpora such as MMC4 and M3W.
- The authors are from S-Lab, Nanyang Technological University, and Microsoft Research (arXiv:2305.03726).
Cited by
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
- ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
- Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
- How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
- A Survey on In-context Learning
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
- Evaluating Object Hallucination in Large Vision-Language Models
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
- SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
- Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
- Aligning Large Multimodal Models with Factually Augmented RLHF
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
- Instruction Tuning for Large Language Models: A Survey
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
- Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
Cite
@misc{li2023otter,
title = {{Otter: A Multi-Modal Model with In-Context Instruction Tuning}},
author = {Li, Bo and Zhang, Yuanhan and Chen, Liangyu and Wang, Jinghao and Pu, Fanyi and Cahyono, Joshua Adrian and Yang, Jingkang and Liu, Ziwei},
year = {2023},
doi = {10.48550/arxiv.2305.03726},
eprint = {2305.03726},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2305.03726}
}References
- Common crawl. https://commoncrawl.org/. Accessed: 2023-11-11. 3 Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661, 2022. 2 Jean-Baptiste Alayrac, Jeff Donahue, Pa… ↗ our card