Elamur Research
☰
home / papers / 2018 / Object Hallucination in Image Captioning

Object Hallucination in Image Captioning

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1437.pdf

Summary

Modern image captioning systems often mention objects that are absent from the picture. Standard metrics compare generated text only with reference captions, so they can miss this error. To catch it, the authors propose a metric that compares every object a caption names against the 80 MSCOCO categories, and they call it CHAIR, short for "Caption Hallucination Assessment with Image Relevance." Ground truth comes from both MSCOCO segmentation labels and MSCOCO Captions, with synonym mapping and handling of compounds like "hot dog". CHAIR has two variants: CHAIRi measures the share of mentioned objects that are hallucinated, and CHAIRs measures the share of sentences containing at least one hallucination.

They apply CHAIR to many models with different architectures (FC, LRCN, Att2In, TopDown, TopDown-BB, NBT, GAN) and training losses (cross-entropy, self-critical). To trace causes, they compare errors with an image-only classifier and an LSTM language model. Models that hallucinate more tend to produce errors that agree with language priors. Top scores on standard metrics do not guarantee fewer hallucinations. The work argues that captioning should be evaluated and developed with image relevance in mind.

Key points

  • Current models hallucinate between 5.5% and 13.1% of mentioned MSCOCO objects. Self-critical (CIDEr-optimizing) training raises CIDEr but also increases hallucination.
  • Attention reduces hallucination, but most of the gain seems to come from access to convolutional features rather than from the attention mechanism itself. Models with stronger image consistency hallucinate less.
  • Among standard metrics, SPICE correlates best with hallucination-free output (0.27 for FC vs. about 0.2 for METEOR and CIDEr). Even so, models with similar scores can differ noticeably in hallucination rates.
  • In an AMT study, workers rated captions from 5 models on 500 images, with 3 annotators per image. Adding 1−CHAIR to a sentence metric raised correlation with human scores, for example CIDEr from 0.282 to 0.322.
  • A hallucinated word changes the subsequent words 47.3% of the time, and one hallucination can trigger another (e.g., "dog" leading to "frisbee").

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

214 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 195 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1437

@inproceedings{rohrbach2018object,
  title = {{Object Hallucination in Image Captioning}},
  author = {Rohrbach, Anna and Hendricks, Lisa Anne and Burns, Kaylee and Darrell, Trevor and Saenko, Kate},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1437},
  url = {https://doi.org/10.18653/v1/d18-1437}
}

References

21 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In European Conference on Computer Vision, pages 382–398. Springer. find our card
  2. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and vqa. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. find
  3. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72. metrics for image captio… find our card
  4. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32… find our card
  5. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer. find our card
  6. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2018. Neural baby talk. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. find
  7. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. find our card
  8. Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich. 2018. Discriminability objective for training descriptive captions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. find
  9. Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge Belongie. 2018. Learning to evaluate image captioning. In CVPR. find
  10. Haley MacLeod, Cynthia L. Bennett, Meredith Ringel Morris, and Edward Cutrell. 2017. Understanding blind peoples experiences with computer-generated captions of social media images. In Proceedings of the 2017 SIGCHI Conference on Human Factors in Computing Systems. find
  11. Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition… find our card
  12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. find
  13. Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Darrell. 2016. Deep compositional captioning: Describing novel object categories without paired training data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages… find
  14. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  15. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137. find our card
  16. Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2016. Re-evaluating automatic Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision a… find our card
  17. Anna Rohrbach, Makarand Tapaswi, Atousa Torabi, Tegan Maharaj, Marcus Rohrbach, Sanja Fidler Christopher Pal, and Bernt Schiele. 2017a. The Joint Video and Language Understanding Workshop: MovieQA and The Large Scale Movie Description Challenge (LSMDC). https://sites.google.com/site/ describingmovie… ↗
  18. Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. 2017b. Movie description. International Journal of Computer Vision, 123(1):94–120. find
  19. Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. 2017. Speaking the same language: Matching machine to human captions by adversarial training. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). find
  20. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4566–4575. find our card
  21. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048–2057. find our card