A Survey on Multimodal Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2306.13549

Summary

This survey by Yin and colleagues (USTC and Tencent YouTu Lab) maps the fast-moving field of Multimodal Large Language Models (MLLMs). These systems use a pre-trained LLM as the reasoning core and attach perception components so the model can accept, and sometimes generate, non-text signals. The motivation is that two kinds of models have complementary gaps: text-only LLMs cannot perceive images, while large vision models see well but reason weakly.

The authors describe theirs as the first survey on the topic. They organise the literature around three areas:

• architecture: modality encoder, connector, LLM and an optional output generator
• training strategy and data
• evaluation practice

They then cover extensions toward finer region-level control, extra modalities and languages beyond English (e.g. Chinese with limited corpora), and new domains such as medicine, documents and agents. Further sections address multimodal hallucination and three techniques: M-ICL, M-CoT and LLM-Aided Visual Reasoning (LAVR).

The paper introduces no new model or dataset. Its value is a structured reference plus a continuously updated paper list, which is useful when planning vision-language work for under-resourced languages like Ukrainian.

Key points

• Connectors fall into three families: projection-based, query-based (e.g. Q-Former) and fusion-based. The first two merge features at the token level, while fusion-based connectors combine them at the feature level inside the LLM.
• The image encoders surveyed range from CLIP-ViT-L/14 (OpenAI WIT, 224/336 px, ~304M parameters) to OpenCLIP-ViT-bigG/14 (LAION-2B, ~1.84B parameters). Fuyu-8B skips the encoder entirely, and ImageBind-LLM also handles audio, depth, thermal and IMU input.
• Empirical studies cited in the survey find that input resolution matters more than encoder size or pretraining data mix. Resolution is raised in two ways:
  • direct scaling, e.g. CogAgent's dual encoder
  • patch division, e.g. Monkey and SPHINX
• Common LLM backbones include Flan-T5 (used in BLIP-2 and InstructBLIP) and LLaMA/Vicuna, which are mostly English-centric, and Qwen, which is bilingual in Chinese and English. Liu et al. report broad gains from scaling the LLM from 7B to 13B.
• The survey is maintained over time (v4, Nov 2024), with a companion paper list on GitHub: BradyFU/Awesome-Multimodal-Large-Language-Models.