Elamur Research⌕
☰
home / papers / 2016 / Visual Storytelling

Visual Storytelling

Summary

Image captioning research mostly produces literal descriptions of single photos. This paper asks whether systems can instead produce narratives for photo sequences, using social, evaluative and figurative language grounded in what the images show. The authors build SIND v.1, described as the first dataset for sequential vision-to-language.

They mined "storyable" event types from Flickr and collected short-timespan albums. Crowd workers on Mechanical Turk then selected and ordered photos and wrote a story for them. Every sequence carries three aligned tiers of text: DII (each image described alone), DIS (images described within the sequence) and SIS (stories). This design separates the effect of temporal context from the effect of narrative style.

The authors compared automatic metrics against human ratings and propose METEOR as the most suitable one. They then trained a GRU-based sequence-to-sequence baseline. Simple decoding heuristics raised its METEOR from about 23 to over 31. The results suggest that story generation poses different challenges than captioning.

Key points

  • Dataset scale: SIND v.1 contains 81,743 unique photos in 20,211 sequences. These come from 10,117 CC-licensed Flickr albums with 210,819 photos in total. Albums hold 10–50 photos taken within 48 hours, and each album averages 20.8 photos.

  • Story collection: Two workers per album performed storytelling, choosing at least five photos and writing one line per photo. Three more workers re-told existing sequences, for a total of 1,907 workers.

  • Post-processing and splits: Person names were replaced with MALE/FEMALE tokens and other named entities with their type. The data is split 80/10/10.

  • Metric choice: METEOR correlated best with human judgments (Pearson r = 0.22). Skip-Thoughts reached 0.18 and BLEU only 0.08.

  • Baseline results (METEOR on stories):

    • beam search with size 10: 23.13
    • greedy decoding: 27.76
    • greedy without duplicate content words: 30.11
    • greedy plus caption-based grounding: 31.42

    For per-image captions the effect reversed: beam search scored 23.55 and greedy 19.10.

Interesting ideas

  • I1. Tiered annotation of the same inputs. Collecting literal, in-context and narrative text for identical images lets you isolate what context adds and what style adds. A similar design could separate register or genre effects in any corpus-building project.
  • I2. Mining event types from possessive phrases. The authors extracted heads of phrases like "someone's X" from photo titles and descriptions, then filtered them with WordNet EVENT classes. This is a cheap way to discover culturally relevant event categories for targeted data collection.
  • I3. Validating the metric before using it. Several metrics were correlated against human ratings before one was adopted. This is a good habit whenever a new generation task lacks an established evaluation.
  • I4. Decoding choices are task-dependent. Greedy decoding helped stories but hurt captions, so beam search settings tuned for one generation task should not be assumed to transfer to another.
  • I5. Licensing words with a second model. Words more typical of captions than of stories were allowed in a story only if a caption model proposed them for those photos. This is a lightweight grounding constraint that could curb hallucinated content in other conditional generation systems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

126 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 123 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1147

@inproceedings{huang2016visual,
  title = {{Visual Storytelling}},
  author = {Huang, Ting-Hao Kenneth and Ferraro, Francis and Mostafazadeh, Nasrin and Misra, Ishan and Agrawal, Aishwarya and Devlin, Jacob and Girshick, Ross and He, Xiaodong and Kohli, Pushmeet and Batra, Dhruv and Zitnick, C. Lawrence and Parikh, Devi and Vanderwende, Lucy and Galley, Michel and Mitchell, Margaret},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1147},
  url = {https://doi.org/10.18653/v1/n16-1147}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In International Conference on Computer Vision (ICCV). find
  2. Jianfu Chen, Polina Kuznetsova, David Warren, and Yejin Choi. 2015. Déjà image-captions: A corpus of expressive descriptions in repetition. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 504–5… find
  3. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. find
  4. CoRR. Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffrey Zweig, and Margaret Mitchell. 2015. Language models for image captioning: The quirks and what works. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Internati… find
  5. Desmond Elliott and Frank Keller. 2013. Image description using visual dependency representations. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1292– 1302, Seattle, Washington, USA, October. Association for Computational Linguistics. Hao Fang, Saur… find
  6. Francis Ferraro, Nasrin Mostafazadeh, Ting-Hao K. Huang, Lucy Vanderwende, Jacob Devlin, Michel Galley, and Margaret Mitchell. 2015. A survey of current datasets for vision and language research. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 207–213… find
  7. Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. 2015. Are you talking to a machine? dataset and methods for multilingual image question. In C. Cortes, N.D. Lawrence, D.D. Lee, M. Sugiyama, R. Garnett, and R. Garnett, editors, Advances in Neural Information Processing Systems… find
  8. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June. find our card
  9. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In C. Cortes, N.D. Lawrence, D.D. Lee, M. Sugiyama, R. Garnett, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 3276– 328… find
  10. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalanditis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. 2016. Visual genome: Connecting language and vision using crowdsourced dense image annotations. find our card
  11. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. find our card
  12. Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42Nd Annual Meeting on Association for Computational Linguistics, ACL ’04, Stroudsburg, PA, USA. Association for Computationa… find our card
  13. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014, pages 740–755. Springer. find our card
  14. Mateusz Malinowski and Mario Fritz. 2014. A multi-world approach to question answering about real-world scenes based on uncertain input. In Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 1682–1690. Curran… find
  15. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. find our card
  16. Mitchell Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor. 1999. Brown corpus, treebank-3. find
  17. Vignesh Ramanathan, Percy Liang, and Li Fei-Fei. 2013. Video event understanding using natural language descriptions. In Computer Vision (ICCV), 2013 IEEE International Conference on, pages 905–912. find
  18. IEEE. Mengye Ren, Ryan Kiros, and Richard Zemel. 2015. Exploring models and data for image question answering. In C. Cortes, N.D. Lawrence, D.D. Lee, M. Sugiyama, R. Garnett, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2935– 2943. find
  19. Curran Associates, Inc. Mohammad Amin Sadeghi and Ali Farhadi. 2011. Recognition using visual phrases. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1745–1752. find
  20. IEEE. Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2015. The new data and new challenges in multimedia research. arXiv preprint arXiv:1503.01817. find
  21. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2014. Show and tell: a neural image caption generator. In CVPR. find our card
  22. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Computer Vision and Pattern Recognition. find our card
  23. Polly W Wiessner. 2014. Embers of society: Firelight talk among the ju/hoansi bushmen. Proceedings of the National Academy of Sciences, 111(39):14027–14035. find
  24. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044. find our card
  25. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2016-visual-storytelling/