📄 Retrieval-Augmented Multimodal Language Modeling

Summary

Large multimodal generators such as DALL-E and CM3 keep all of their world knowledge inside their weights. Covering more entities therefore means ever bigger models and datasets, and the knowledge is hard to update. The authors carry retrieval augmentation, already successful in text-only NLP, over to mixed image–text data. Their system, RA-CM3, has two parts. The retriever is a dense retriever built on frozen CLIP encoders: each document's text and image are embedded separately, and the two vectors are averaged. The generator is a CM3 Transformer that receives the retrieved documents prepended to its input. It is trained on LAION with a token-prediction loss over both the main document and the retrieved ones. RA-CM3 is presented as the first model that can both retrieve and generate text and images. On MS-COCO it clearly beats a CM3 baseline without retrieval and also outperforms DALL-E and Flamingo, while using far fewer resources. The work matters because external memory makes entity-heavy generation more faithful. It also gives the model multimodal in-context learning, including control of image generation through demonstration images.

Key points

• Training and memory data: 150M image–text pairs from LAION. Images are converted into 1024 tokens with VQGAN, and documents are formatted as HTML-like sequences in the CM3 style.
• Retriever design: Dense Retrieval (bi-encoder) using frozen CLIP text and image encoders. The two embeddings are mean-pooled and L2-normalized, and candidates are found with MIPS. The method also selects retrieved documents for diversity.
• MS-COCO results: RA-CM3 improves on vanilla CM3 by about 12 FID points in text-to-image generation and about 17 CIDEr points in captioning. It outperforms DALL-E and Flamingo with under 30% of their parameters and training compute.
• New capabilities:
  • higher-fidelity images of uncommon entities, such as a Ming Dynasty vase or the Oriental Pearl tower, and of unusual combinations (e.g., French flag on the moon);
  • knowledge-aware image infilling and editing;
  • style control by placing example images in the context.
• Few-shot classification: On ImageNet-based binary tasks with non-semantic labels ("animal X/Y"), RA-CM3 reaches 0.78, 0.79, 0.86 and 0.90 accuracy at k = 1, 2, 4 and 8 shots. Baseline CM3 stays at roughly 0.50–0.56.