📄 Analyzing the Behavior of Visual Question Answering Models

Summary

Most deep-learning systems for Visual Question Answering (VQA) score in a narrow 60–70% band, so headline accuracy says little about how they differ or why they fail. The authors propose diagnostic probes that examine model behaviour along three axes. The first asks whether a model generalizes to test cases unlike its training data. The second asks whether it uses the whole question. The third asks whether its answers actually depend on the image. They apply these probes, on the VQA validation set, to three models trained on the VQA training set: a non-attention CNN+LSTM, a hierarchical co-attention model (ATT) and the VQA Challenge 2016 winner (MCB). The models share three weaknesses. They fail on unfamiliar inputs or answers. They often settle on an answer after reading only half the question. They tend to give the same answer for a question regardless of the picture, although attention-based models do this less. The study shows that aggregate accuracy hides these behaviours, and it reveals label biases in the dataset. It points future work toward compositional generalization and less biased data.

Key points

• Data and models: The VQA dataset has ~0.25M images, ~0.76M questions and ~10M answers. Validation accuracies are 54.13% for CNN+LSTM, 57.02% for ATT and 60.36% for MCB.
• Novel inputs:
  • Accuracy correlates negatively with the average distance from a test question-image pair to its k-NN training pairs in the joint embedding space: −0.41 for CNN+LSTM and −0.42 for ATT, but only −0.14 for MCB.
  • These distances flag 67.5%, 66.7% and 55.08% of mistakes, respectively.
• Novel answers: Accuracy also correlates with the Word2Vec distance between the ground-truth answer and the answers of the nearest training neighbours: −0.62 for CNN+LSTM and ATT, and −0.47 for MCB. This points to models repeating answers they saw in training and to a lack of compositionality.
• Partial questions: After seeing half the question, the final answer is already fixed for 40% (CNN+LSTM), 49% (ATT) and 45% (MCB) of questions. Removing wh-words changes answers the most, and removing pronouns changes them the least.
• Image insensitivity: The authors examined 263 questions, each asked about at least 25 images. CNN+LSTM gives one answer for at least half of a question's images on 56% of these questions, versus 42% for ATT and 40% for MCB. This constant behaviour is often rewarded because of label bias.

Interesting ideas

• I1. Neighbour distance as a failure signal: Distance to the nearest training examples in a model's own embedding space can work as a cheap confidence signal. It would let a system decline to answer inputs that are too unfamiliar, and the idea transfers to any classifier or QA system.
• I2. Truncated-input probe: Feeding progressively longer prefixes of the input shows when a model commits to its prediction. This could reveal shortcut reliance on early tokens in text classifiers or NLP models more generally.
• I3. Ablation by part of speech: Dropping all words of one POS category and counting how many predictions change is a simple, model-agnostic way to see which word types actually drive decisions.
• I4. Constant-output probe: Holding one input fixed while varying the other, and measuring how often the output stays the same, quantifies whether a multimodal model really uses both inputs. The same probe also surfaces dataset label biases.
• I5. Separating input novelty from output novelty: Measuring these two kinds of novelty independently distinguishes poor handling of unseen inputs from an inability to produce unseen answer combinations. This makes it a useful check for compositional generalization.