Elamur Research⌕
☰
home / papers / 2017 / Guided Open Vocabulary Image Captioning with Constrained Beam Search

Guided Open Vocabulary Image Captioning with Constrained Beam Search

Summary

Neural image captioners trained on datasets such as MSCOCO cover a limited set of objects, so they often fail on images showing unfamiliar concepts. The authors propose adding outside knowledge only at decoding time, without retraining the captioner. The knowledge comes as image tags or labels, from a tagger or from existing annotations. Their constrained beam search encodes the requirements as a finite-state machine and keeps a separate beam for each machine state. This guarantees that chosen words or phrases appear in the caption, while the RNN still decides sentence structure and other content. To handle tag words absent from the training vocabulary, the model's input and output layers use fixed pretrained word embeddings, and the vocabulary is expanded when needed. On a held-out version of MSCOCO, using tags predicted by the DCC tagger, the method reaches state-of-the-art out-of-domain captioning and also improves in-domain results. It outperforms approaches such as DCC and NOC, which build the same tags into training. Because it works with any beam-searchable CNN-RNN model, the method is a cheap, modular way to extend captioning to new concepts.

Key points

  • Method: Constrained beam search can enforce any constraint expressible as a finite-state machine, which includes anything a regular expression can describe.
    • The algorithm keeps one beam per machine state.
    • Decoding stops once an accepting state holds a finished caption that scores higher than every unfinished candidate.
  • Base model: LRCN, with a CNN feature extractor and two LSTM layers of 1,000 hidden units each.
  • ImageNet test set: 5,000 images from ILSVRC 2012, five from each of the 1,000 classes.
    • The model used a ResNet-50 encoder and was trained on about 155k images from MSCOCO train/val plus Flickr 30k.
    • Each caption was forced to contain a phrase from the ground-truth WordNet synset.
  • Human evaluation (Amazon Mechanical Turk): Constrained captions were rated equal to or better than human captions in 22% of comparisons (12% better, 10% equally good). The unconstrained base model reached 11%.
    • The best results came for super-categories that the captioning training set already covered to some extent, including birds, mammals and dogs.
  • Release: Code and data are available from the project page (panderson.me/constrained-beam-search).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

85 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 76 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1098 · arXiv 1612.00576

@inproceedings{anderson2017guided,
  title = {{Guided Open Vocabulary Image Captioning with Constrained Beam Search}},
  author = {Anderson, Peter and Fernando, Basura and Johnson, Mark and Gould, Stephen},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1098},
  eprint = {1612.00576},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d17-1098}
}

References

32 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cyril Allauzen, Bill Byrne, Adrià de Gispert, Gonzalo Iglesias, and Michael Riley. 2014. Pushdown automata in statistical machine translation. Computational Linguistics 40(3):687–723. find
  2. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: Semantic propositional image caption evaluation. In ECCV. find our card
  3. Minmin Chen, Alice X Zheng, and Kilian Q Weinberger. 2013. Fast image tagging. In ICML. find
  4. Xinlei Chen, Tsung-Yi Lin Hao Fang, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 . find our card
  5. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation. find our card
  6. Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffrey Zweig, and Margaret Mitchell. 2015. Language models for image captioning: The quirks and what works. In ACL. find
  7. Jeffrey Donahue, Lisa A. Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent convolutional networks for visual recognition and description. In CVPR. find our card
  8. Desmond Elliot and Arjen P. de Vries. 2015. Describing images using inferred visual dependency representations. In ACL. find
  9. Michael Sipser. 2012. Introduction to the Theory of Computation. Cengage Learning, 3rd edition. find
  10. Kenneth Tran, Xiaodong He, Lei Zhang, Jian Sun, Cornelia Carapcea, Chris Thrasher, Chris Buehler, and Chris Sienkiewicz. 2016. Rich image captioning in the wild. In CVPR Workshop. find
  11. Hao Fang, Saurabh Gupta, Forrest N. Iandola, Rupesh Srivastava, Li Deng, Piotr Dollar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick, and Geoffrey Zweig. 2015. From captions to visual concepts and back. In CVPR. find
  12. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In CVPR. find our card
  13. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Bradford Books. find
  14. Subhashini Venugopalan, Lisa Anne Hendricks, Marcus Rohrbach, Raymond J. Mooney, Trevor Darrell, and Kate Saenko. 2016. Captioning images with diverse objects. arXiv preprint arXiv:1606.07770 . — [arXiv:1606.07770](https://arxiv.org/abs/1606.07770) find
  15. Marjan Ghazvininejad, Xing Shi, Yejin Choi, and Kevin Knight. 2016. Generating topical poetry. In EMNLP. find
  16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. find
  17. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In CVPR. find our card
  18. Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Darrell. 2016. Deep compositional captioning: Describing novel object categories without paired training data. In CVPR. find
  19. Q. Wu, C. Shen, L. Liu, A. Dick, and A. van den Hengel. 2016. What Value Do Explicit High Level Concepts Have in Vision to Language Problems? In CVPR. find
  20. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL . find
  21. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8). find
  22. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 . find our card
  23. Yang Zhang, Boqing Gong, and Mubarak Shah. 2016. Fast zero-shot image tagging. In CVPR. find
  24. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In CVPR. find our card
  25. Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York, NY, USA, 1st edition. find
  26. Jonathan Krause, Benjamin Sapp, Andrew Howard, Howard Zhou, Alexander Toshev, Tom Duerig, James Philbin, and Li Fei-Fei. 2016. The unreasonable effectiveness of noisy data for fine-grained recognition. In ECCV. find
  27. T.Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick. 2014. Microsoft COCO: Common objects in context. In ECCV. find our card
  28. Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan L. Yuille. 2015. Deep captioning with multimodal recurrent neural networks (m-RNN). In ICLR. find
  29. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In EMNLP. find
  30. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS. find our card
  31. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV) 115(3):211–252… find our card
  32. Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In ICLR. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/anderson-2017-guided-open-vocabulary-image-captioning/