Elamur Research
☰
home / papers / 2017 / FiLM: Visual Reasoning with a General Conditioning Layer

FiLM: Visual Reasoning with a General Conditioning Layer

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1709.07871

Summary

Standard visual question answering models often fail on CLEVR, a benchmark of questions that need multi-step reasoning about rendered scenes. They tend to exploit dataset biases instead. The authors avoid hard-coding reasoning structure into the architecture. They propose FiLM (Feature-wise Linear Modulation), a generic conditioning layer in which one network predicts a scale (γ) and a shift (β) for every feature map of another network. In their model, a GRU reads the question and outputs these parameters for several residual blocks of an image CNN, so language steers visual processing directly. The model is trained end-to-end only on CLEVR image-question-answer triplets, and it roughly halves the previous best error on CLEVR. Ablations show that the gains survive many design changes and do not require placing the affine transformation right after normalization. On CLEVR-CoGenT, FiLM adapts well from little fine-tuning data, and arithmetic on its predicted parameters enables some zero-shot generalization. The work suggests that a cheap, general-purpose conditioning mechanism can rival architectures built specifically for reasoning.

Key points

  • Architecture:
    • The FiLM generator is a GRU with 4096 hidden units over 200-dimensional word embeddings.
    • The visual pipeline has 4 FiLM-ed ResBlocks with 128 feature maps each.
    • Image features come either from a CNN trained from scratch or from a frozen ResNet-101 conv4 extractor pre-trained on ImageNet.
  • Efficiency: FiLM needs only two parameters per modulated feature map, and its cost does not grow with image resolution.
  • Relation to prior methods: FiLM generalizes Conditional Normalization variants such as Conditional Instance Norm, Adaptive Instance Norm, Dynamic Layer Norm and Conditional Batch Norm. The experiments indicate that the normalization step is not essential.
  • Sample efficiency on CLEVR-CoGenT: FiLM reaches the prior state of the art with one third of the fine-tuning data. Even so, fine-tuning leaves the model prone to losing much of what it had previously learned.
  • Zero-shot method: Questions are combined linearly in (γ, β) space, analogous to word-embedding analogies.
    • Training on condition A and testing zero-shot on B gives a 3.2% overall accuracy gain.
    • The method applies to about 1/3 of the questions in B, where accuracy rises from 71.5% to 80.7%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

164 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 158 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1709.07871 · arXiv 1709.07871

@misc{perez2017film,
  title = {{FiLM: Visual Reasoning with a General Conditioning Layer}},
  author = {Perez, Ethan and Strub, Florian and Vries, Harm de and Dumoulin, Vincent and Courville, Aaron},
  year = {2017},
  doi = {10.48550/arxiv.1709.07871},
  eprint = {1709.07871},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1709.07871}
}

References

38 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; and Zhang, L. 2017. Bottom-up and top-down attention for image captioning and vqa. In VQA Workshop at CVPR. find
  2. Andreas, J.; Marcus, R.; Darrell, T.; and Klein, D. 2016a. Learning to compose neural networks for question answering. In NAACL. find our card
  3. Andreas, J.; Rohrbach, M.; Darrell, T.; and Klein, D. 2016b. Neural module networks. In CVPR. find our card
  4. Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Zitnick, C. L.; and Parikh, D. 2015. find
  5. VQA: Visual Question Answering. In ICCV. Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; and Yakhnenko, O. 2013. Translating embeddings for modeling multi-relational data. In Burges, C. J. C.; Bottou, L.; Welling, M.; Ghahramani, Z.; and Weinberger, K. Q., eds., NIPS. Curran Associates, Inc.… find
  6. Dumoulin, V.; Shlens, J.; and Kudlur, M. 2017. A learned representation for artistic style. In ICLR. find
  7. Eigen, D.; Ranzato, M.; and Sutskever, I. 2014. Learning factored representations in a deep mixture of experts. In ICLR Workshops. find
  8. Gehring, J.; Auli, M.; Grangier, D.; Yarats, D.; and Dauphin, Y. N. 2017. Convolutional sequence to sequence learning. In ICML. find our card
  9. Geman, D.; Geman, S.; Hallonquist, N.; and Younes, L. 2015. Visual turing test for computer vision systems. volume 112, 3618– 3623. National Acad Sciences. find
  10. Ghiasi, G.; Lee, H.; Kudlur, M.; Dumoulin, V.; and Shlens, J. 2017. Exploring the structure of a real-time, arbitrary neural artistic stylization network. CoRR abs/1705.06830. — [arXiv:1705.06830](https://arxiv.org/abs/1705.06830) find
  11. Goyal, Y.; Khot, T.; Summers-Stay, D.; Batra, D.; and Parikh, D. 2017. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In CVPR. find our card
  12. Guu, K.; Miller, J.; and Liang, P. 2015. Traversing knowledge graphs in vector space. In EMNLP. find
  13. Ha, D.; Dai, A.; and Le, Q. 2016. Hypernetworks. In ICLR. find
  14. He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In CVPR. find
  15. Hochreiter, S., and Schmidhuber, J. 1997. Long short-term memory. Neural Comput. 9(8):1735–1780. find
  16. Hu, R.; Andreas, J.; Rohrbach, M.; Darrell, T.; and Saenko, K. 2017. Learning to reason: End-to-end module networks for visual question answering. In ICCV. find
  17. Hu, J.; Shen, L.; and Sun, G. 2017. find
  18. Squeeze-and-Excitation Networks. In ILSVRC 2017 Workshop at CVPR. find
  19. Huang, X., and Belongie, S. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV. find
  20. Ioffe, S., and Szegedy, C. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML. find
  21. Johnson, J.; Hariharan, B.; van der Maaten, L.; Fei-Fei, L.; Zitnick, C. L.; and Girshick, R. B. 2017a. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR. find our card
  22. Johnson, J.; Hariharan, B.; van der Maaten, L.; Hoffman, J.; Li, F.; Zitnick, C. L.; and Girshick, R. B. 2017b. Inferring and executing programs for visual reasoning. In ICCV. find
  23. Jordan, M. I., and Jacobs, R. A. 1994. Hierarchical mixtures of experts and the em algorithm. Neural Comput. 6(2):181–214. find
  24. Kim, T.; Song, I.; and Bengio, Y. 2017. Dynamic layer normalization for adaptive neural acoustic modeling in speech recognition. In InterSpeech. find
  25. Kingma, D. P., and Ba, J. 2015. Adam: A method for stochastic optimization. In ICLR. find
  26. Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; Hassabis, D.; Clopath, C.; Kumaran, D.; and Hadsell, R. 2017. Overcoming catastrophic forgetting in neural networks. National Academy of Sciences 114(13):35… find
  27. Lu, J.; Yang, J.; Batra, D.; and Parikh, D. 2016. Hierarchical question-image co-attention for visual question answering. In NIPS. find
  28. Malinowski, M., and Fritz, M. 2014. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS. find
  29. Malinowski, M.; Rohrbach, M.; and Fritz, M. 2015. Ask your neurons: A neural-based approach to answering questions about images. In ICCV. find
  30. Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In NIPS. find
  31. Oh, J.; Singh, S.; Lee, H.; and Kholi, P. 2017. Zero-shot task generalization with multi-task deep reinforcement learning. In ICML. find
  32. Perez, E.; de Vries, H.; Strub, F.; Dumoulin, V.; and Courville, A. C. 2017. Learning visual reasoning without strong priors. In MLSLP Workshop at ICML. find
  33. Radford, A.; Metz, L.; and Chintala, S. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR. find our card
  34. Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M. S.; Berg, A. C.; and Li, F. 2015. Imagenet large scale visual recognition challenge. IJCV 115(3):211–252. find our card
  35. Santoro, A.; Raposo, D.; Barrett, D. G.; Malinowski, M.; Pascanu, R.; Battaglia, P.; and Lillicrap, T. 2017. A simple neural network module for relational reasoning. CoRR abs/1706.01427. find our card
  36. Shazeer, N.; Mirhoseini, A.; Maziarz, K.; Davis, A.; Le, Q.; Hinton, G.; and Dean, J. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR. van den Oord, A.; Dieleman, S.; Zen, H.; Simonyan, K.; Vinyals, O.; Graves, A.; Kalchbrenner, N.; Senior, A.; and Kavu… find our card
  37. Watters, N.; Tacchetti, A.; Weber, T.; Pascanu, R.; Battaglia, P.; and Zoran, D. 2017. Visual interaction networks. CoRR abs/1706.01433. find
  38. Yang, Z.; He, X.; Gao, J.; Deng, L.; and Smola, A. J. 2016. Stacked attention networks for image question answering. In CVPR. find