Elamur Research⌕
☰
home / papers / 2017 / FOIL it! Find One mismatch between Image and Language caption

FOIL it! Find One mismatch between Image and Language caption

Paper · 2017 · Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich et al.

Summary

The paper asks whether language-and-vision (LaVi) models really link text to image content or mostly rely on language priors. The authors build FOIL-COCO, an automatic extension of MS-COCO. Each image is paired with its original human caption and with a "foil" version in which exactly one noun is swapped for a related but wrong object name. The swaps come from MS-COCO category labels within the same supercategory, and the target::foil pairs are split so that training and testing use different pairs.

Three tasks of increasing difficulty are defined:

  • T1: decide whether a caption is correct or foil.
  • T2: locate the wrong word.
  • T3: supply the right replacement.

Two VQA models (LSTM + norm I following Antol et al. and the attention-based HieCoAtt) and one image-captioning model (IC-Wang) perform close to chance. Humans, by contrast, solve the tasks almost effortlessly. Because only one word differs, the dataset limits language bias and makes clear where a model fails. This makes it a useful diagnostic benchmark, which is more informative than n-gram-based captioning metrics.

Key points

  • FOIL-COCO has 297,268 datapoints over 97,847 images: 197,788 datapoints and 65,697 images for training, and 99,480 datapoints and 32,150 images for testing.
  • 73 of the 91 MS-COCO categories are used, leaving out multiword ones such as traffic light. They yield 472 target::foil pairs, split into 256 for training and 216 for testing.
  • The models evaluated are LSTM + norm I, HieCoAtt and IC-Wang. IC-Wang does best on T3, which the authors link to the task being generative and to the model seeing the whole sentence around the missing word.
  • Human results:
    • T1: majority vote is correct in 914 of 984 cases (92.89%), and unanimous agreement is correct in 76.32% of cases.
    • T2: among correctly classified foil captions, humans find the foil word in 97% of cases (majority) and 73.6% (unanimity).
  • The two VQA models succeed and fail on different pairs. For example, boat::airplane is easy for LSTM + norm I but among the hardest for HieCoAtt.

Interesting ideas

  • I1. Single-edit perturbations of gold data. Changing exactly one controlled element of a human-written example produces hard, near-miss negatives with a known error location. The same recipe could build diagnostic sets for other grounding or NLU tasks.
  • I2. Disjoint replacement pairs between train and test. Using different substitution pairs in each split stops models from memorising specific swaps. This is a cheap safeguard against shortcut learning in any synthetic-negative dataset.
  • I3. A graded task ladder (detect → localise → correct). Each step tests a deeper ability, so the point where a model breaks reveals what it lacks: rough compatibility, fine-grained language structure, or visual grounding.
  • I4. Evaluate both correct and foil items. The authors note that recognising correct captions is much easier than spotting foils, so performance should be reported on both classes rather than as overall accuracy only.
  • I5. Read unanimous human errors qualitatively. Cases where every annotator was wrong exposed typos in the original captions and vague or metaphorical wording. This is a practical way to audit label noise in generated benchmarks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TrentoTrento · Italy62 works · 14,868 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto and Raffaella Bernardi. 2017. FOIL it! Find One mismatch between Image and Language caption. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1024

@inproceedings{shekhar2017foil,
  title = {{FOIL it! Find One mismatch between Image and Language caption}},
  author = {Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto and Raffaella Bernardi},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1024},
  url = {https://doi.org/10.18653/v1/p17-1024}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016a. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. arXiv preprint arXiv:1612.00837 .
  2. Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016. Analyzing the behavior of visual question answering models. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP).
  3. Yash Goyal, Akrit Mohapatra, Devi Parikh, and Dhruv Batra. 2016b. Towards Transparent AI Systems: Interpreting Visual Question Answering Models . In In Proceedings of ICML Visualization Workshop. find
  4. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: Semantic Propositional Image Caption Evaluation. In In Proceedings of the European Conference on Computer Vision (ECCV).
  5. Micah Hodosh and Julia Hockenmaier. 2016. Focused evaluation for image description with binary forced-choice tasks. In Proceedings of the 5th Workshop on Vision and Language (VL’16). find
  6. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV). https://github.com/ VT-vision-lab/VQA_LSTM_CNN. ↗
  7. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research 47:853–899. find
  8. Raffaella Bernardi, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. 2016. Automatic description generation from images: A survey of models, datasets, and evaluation measures. J. Artif. Intell. Res.(JAIR) 55:409–442. find
  9. Allan Jabri, Armand Joulin, and Laurens van der Maaten. 2016. Revisiting visual question answering baselines. In Proceedings of the European Conference on Computer Vision (ECCV). pages 727–739. find
  10. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2016. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. ArXiv:1612.06890.
  11. Xinlei Chen and C Lawrence Zitnick. 2015. Mind’s eye: A recurrent visual representation for image caption generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 2422–2431. find
  12. Kushal Kafle and Christopher Kanan. 2016. Visual question answering: Datasets, algorithms, and future challenges. arXiv preprint arXiv:1610.01465 . find
  13. Nan Ding, Sebastian Goodman, Fei Sha, and Radu Soricut. 2016. Understanding image and text simultaneously: a dual vision-language machine comprehension task. arXiv preprint arXiv:1612.07833 . — [arXiv:1612.07833](https://arxiv.org/abs/1612.07833) find
  14. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 3128–3137.
  15. Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition…
  16. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision. Springer, pages 740–755.
  17. Desmond Elliott and Frank Keller. 2014. Comparing automatic evaluation measures for image description. In In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: Short Papers. pages 452–457. find
  18. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical question-image co-attention for visual question answering. In In Proceedings of NIPS 2016. https://github.com/jiasenlu/HieCoAttenVQA. ↗
  19. Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. 2015. From captions to visual concepts and back. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 1473–1482… find
  20. Mateusz Malinowski and Mario Fritz. 2014. A multi-world approach to question answering about real-world scenes based on uncertain input. In Advances in Neural Information Processing Systems. pages 1682–1690. find
  21. Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. 2015. Are you talking to a machine? dataset and methods for multilingual image question. In Advances in Neural Information Processing Systems. pages 2296–2304. find
  22. Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. 2015. Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE International Conference on Computer Vision. pages 1–9. find
  23. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 .
  24. Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Trevor Darrell Bernt Schiele, and Marcus Rohrbach. 2016. Attentive explanations: Justifying decisions and pointing to the evidence. ArXiv:1612.04757. — [arXiv:1612.04757](https://arxiv.org/abs/1612.04757) find
  25. Mengye Ren, Ryan Kiros, and Richard Zemel. 2015. Exploring models and data for image question answering. In Advances in Neural Information Processing Systems (NIPS 2015). find
  26. Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. 2016. Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. ArXiv:1610.02391v2.
  27. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 3156–3164.
  28. Cheng Wang, Haojin Yang, Christian Bartz, and Christoph Meinel. 2016. Image captioning with deep bidirectional LSTMs. In Proceedings of the 2016 ACM on Multimedia Conference. ACM, pages 988–997. find
  29. Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton van den Hengel. 2016. Visual question answering: A survey of methods and datasets. arXiv preprint arXiv:1607.05910 . — [arXiv:1607.05910](https://arxiv.org/abs/1607.05910) find
  30. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2:67–78. find
  31. Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016. Yin and yang: Balancing and answering binary visual questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 5014–5022. find
  32. Bolei Zhou, Yuandong Tian, Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. 2015. Simple baseline for visual question answering. arXiv preprint arXiv:1512.02167 . find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shekhar-2017-foil-it-find-one-mismatch/