📄 LaMI: Augmenting Large Language Models via Late Multi-Image Fusion

Summary

Text-only LLMs often fail at visual commonsense questions, such as the colour of an animal's belly. VLMs handle these questions better, but they need expensive multimodal training and often lose ground on purely textual reasoning. LaMI (Late Multi-Image fusion) adds visual knowledge to an existing LLM while keeping it unchanged.

The LLM and a vision encoder both stay frozen. Only two small parts are trained: a Visual Token Projector, which turns image patch features into pseudo-text embeddings, and a Late Fusion Attention Layer. In that layer, the LLM's final hidden states attend to the visual tokens once, just before the prediction head.

Real images are not available at test time, so a distilled text-to-image model generates several images from the prompt in one parallel batch. Each image produces its own prediction, and these are blended with the text-only prediction. The weighting uses CLIP image–text alignment, so poorly matching images give way to the text-only path.

The method beats earlier visually-augmented LMs on object commonsense, rivals VLMs on visual tasks, and also slightly improves text benchmarks for models such as Llama3. This makes it a cheap way to add visual grounding to newly released LLMs.

Key points

• Object commonsense with GPT-2: For a fair comparison, LaMI was trained only on Visual Genome. It reaches 72.5 on Memory Color, 69.2 on Color Terms, 66.8 on shape and 85.5 on relative size, compared with 32.4, 34.6, 44.5 and 43.1 for plain GPT-2. On relative size it only marginally beats VaLM (85.0).
• Ablation: Late fusion outperforms both early and intermediate fusion. Using multiple images helps under every fusion strategy, and combining late fusion with multiple images gives the best results.
• Comparison with VLMs (average over VC/CR/RC):
  • Llama3-8B-Instruct improves from 61.2 to 62.7, ahead of Llava-Next at 60.7.
  • Qwen-2.5-7B-Instruct improves from 64.4 to 66.0, ahead of Qwen2-VL-7B-Instruct at 63.7.
  • Vicuna-7B improves from 53.4 to 55.1, ahead of InstructBLIP at 52.1.
• Number of images: Gains level off at around k≈6, and k=3 already helps. With Gemma-2B, inference time is 389 ms with generated images, 58 ms with CLIP text embeddings and 20 ms for the baseline. Generated images score better than CLIP embeddings.
• Compute versus visual evidence: On visual commonsense with Gemma-2B, a Best-of-N baseline with a matched compute budget reaches 47.8, while LaMI reaches 50.1. This suggests the gains come from visual evidence rather than extra compute.
• Failure mode: When generated images are plausible but wrong, they can override a correct text prior. This is most likely for abstract or legendary concepts.