📄 Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

Summary

Visual question answering and phrase grounding both require fusing an image vector with a text vector. Typical fusion methods such as concatenation, element-wise sum or element-wise product may miss rich cross-modal interactions. A full outer product captures every pairwise interaction, but it is far too large to learn on top of. The authors propose Multimodal Compact Bilinear pooling (MCB), which approximates the outer product cheaply. Each modality's vector is hashed into a higher-dimensional space with Count Sketch, and the two sketches are then convolved by multiplying them element-wise in FFT space.

For VQA, their architecture applies MCB twice: once to compute spatial attention over ResNet-152 feature maps, and once to merge the attended image features with an LSTM question encoding. A third MCB handles multiple-choice answers. For grounding, MCB scores bounding-box proposals against a phrase. MCB beats concatenation and element-wise baselines throughout. It reaches state-of-the-art results on the VQA challenge, Visual7W, Flickr30k Entities and ReferItGame. This makes it a strong general-purpose fusion operator.

Key points

• Cost of a full bilinear layer. With 2048-D inputs and a 3000-D output, an explicit bilinear layer would need about 12.5 billion parameters. MCB avoids this by never computing the outer product directly.
• VQA setup:
  • Image features come from a ResNet-152 pretrained on ImageNet, using 448×448 inputs and 2048-D L2-normalized features.
  • Questions are encoded with a 2-layer LSTM of 1024 units per layer, whose outputs are concatenated into a 2048-D vector.
  • MCB produces a 16,000-D vector, followed by signed square-root and L2 normalization.
  • A classifier then chooses among the 3,000 most frequent answers.
• VQA results:
  • A 7-model ensemble, with some members trained on extra Visual Genome data and some with two attention maps, beat the next best entry by 1.8 points on open-ended and 0.8 points on multiple-choice (Test-dev).
  • The best single model ("MCB + Genome + Att. + GloVe") scored 65.4% versus 64.9% for the next best result.
• Grounding results (IOU > 50%):
  • On Flickr30k Entities, MCB reached 48.69%, compared with 46.50% for concatenation, 47.41% for element-wise product and 47.86% for element-wise product plus conv.
  • On ReferItGame, MCB reached 28.91%, compared with 25.48%, 27.80% and 27.98% for the same baselines.
  • For grounding, MCB used d = 2048 with Adam at a learning rate of 0.0001.
• Code: released at github.com/akirafukui/vqa-mcb.

Interesting ideas

• I1. Cheap outer product via sketch and FFT. Fixed random hashing plus FFT-based convolution approximates a full outer product at low cost. It could replace concatenation wherever two embeddings must interact, such as text paired with metadata or with geographic features.
• I2. The same fusion operator, reused. The operator appears once to compute attention weights and again to produce the final joint representation. A single strong fusion primitive can serve several roles in one network.
• I3. Extends to more than two inputs. In the frequency domain, fusing an extra modality is just one more element-wise product. Adding a third input source therefore stays cheap.
• I4. Fair ablations against parameter-matched baselines. MCB was compared with element-wise product plus an added convolution, which has more parameters than MCB yet still lost. This kind of comparison is a useful template for showing that a gain comes from the operator itself rather than from extra capacity.
• I5. Normalization after fusion. Applying signed square-root and L2 normalization to the pooled vector is a simple trick that may help stabilize other high-dimensional pooled representations.