Elamur Research
☰
home / papers / 2018 / Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1238.pdf

Summary

Image captioning models had been trained mostly on MS-COCO. That dataset is hand-curated and fairly small for deep networks, at around 120K images. The authors address this with Conceptual Captions, a dataset of about 3.3M image–caption pairs harvested automatically from the web. A Flume pipeline started from the Alt-text attributes of over 5 billion images on roughly 1 billion English webpages. It filtered the candidates by image properties, by text quality and by checking that image and text agree. It then rewrote the surviving captions through hypernymization: proper names, places and dates were replaced with general terms (for example, actor names became "actors") or removed, using the Google Knowledge Graph. The result covers a wider range of images and caption styles than COCO, such as product shots, cartoons and drawings. Captioning models were built on Inception-ResNet-v2 image features, paired with either an RNN or a Transformer decoder. In human evaluation, models trained on Conceptual Captions did better and hallucinated objects less often than COCO-trained ones. Transformer decoders also beat RNN decoders. Standard automatic metrics often disagreed with these human judgments, which led the authors to question how reliable those metrics are.

Key points

  • Scale: about 3.3M pairs, roughly ten times as many images as COCO. Only about 0.2% of the original web candidates made it through every filter. The dataset is published at github.com/google-research-datasets/conceptual-captions.
  • Filtering stages:
    • Image filter: keeps only JPEGs with both sides over 400 px and an aspect ratio of at most 2, and removes offensive images. This drops more than 65% of candidates.
    • Text heuristics: use Google Cloud Natural Language annotations (POS tags, sentiment, capitalization, a Wikipedia-based vocabulary) and let only about 3% through.
    • Image–text match: compares Cloud Vision labels (on the order of 10^5 classes) with the caption words and removes about 60% of what remains.
  • Hypernymization combines named-entity recognition, dependency parses and Knowledge Graph hypernyms. It removes details that a model could not learn from pixels alone, such as who a person is or where a photo was taken.
  • The best architecture was T2T8x8, which uses Inception-ResNet-v2 features with a Transformer decoder. The text reports a CIDEr score of up to 1.468 on the 22.5K-image Conceptual Captions test set, although the extracted Table 6 shows higher values (up to 1.676). I couldn't resolve this because part of the paper was missing from the extracted text.
  • On COCO and Flickr test sets, CIDEr, ROUGE-L, METEOR and SPICE favored COCO-trained and RNN models, which contradicts the human ratings. The authors argue these metrics do not penalize hallucinated content enough.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

474 papers of our library cite it (of the 140673 whose bibliographies we read); 42 of them have a page here, 432 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1238

@inproceedings{sharma2018conceptual,
  title = {{Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning}},
  author = {Sharma, Piyush and Ding, Nan and Goodman, Sebastian and Soricut, Radu},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1238},
  url = {https://doi.org/10.18653/v1/p18-1238}
}

References

23 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John Platt, et al. 2015. From captions to visual concepts and back. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  2. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: semantic propositional image caption evaluation. In ECCV. find our card
  3. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735– 1780. find
  4. D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR. find our card
  5. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. JAIR . find
  6. Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, and Kevin Murphy. 2016. Speed/accuracy trade-offs for modern convolutional object detectors. CoRR abs/1611.10012. — [arXiv:1611.10012](https://arxiv.org/abs/1611.10012) find
  7. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find our card
  8. Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. 2015. Unifying visual-semantic embeddings with multimodal neural language models. Transactions of the Association for Computational Linguistics . find our card
  9. A. Krizhevsky, I. Sutskever, and G. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In NIPS. find
  10. Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of ACL. find our card
  11. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. CoRR abs/1405.0312. find our card
  12. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find our card
  13. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015a. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 3156–3164. find our card
  14. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015b. Show and tell: A neural image caption generator. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find our card
  15. Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proc. of the 32nd International Conference on Machine Learning (ICML). find our card
  16. Z. Yang, Y. Yuan, Y. Wu, R. Salakhutdinov, and W. W. Cohen. 2016. Review networks for caption generation. In NIPS. find
  17. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL 2:67–78. find
  18. Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. 2017. Optimization of image description metrics using policy gradient methods. In International Conference on Computer Vision (ICCV). find
  19. Junhua Mao, Jiajing Xu, Yushi Jing, and Alan Yuille. 2016. Training and evaluating multimodal word embeddings with large-scale web annotated images. In NIPS. find
  20. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. CoRR abs/1511.06732. find our card
  21. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. pages 3104–3112. find
  22. Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke. 2016. Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR abs/1602.07261. — [arXiv:1602.07261](https://arxiv.org/abs/1602.07261) find
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. find