📄 Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences

Summary

Visual story generation models are often trained on image sequences such as VIST's Flickr albums. These sequences tend to lack a real plot, and the people in them change from picture to picture, so the stories written for them are weak training data. The authors introduce the notion of "visual tellability": a good image prompt should be coherent, diverse, and visually grounded. To get such prompts, they built an automatic pipeline that draws shots from MovieNet scenes tied to a single synopsis paragraph and filters out unsuitable genres, near-duplicate frames, and adult content. Crowdworkers then wrote stories for these sequences while seeing the main characters' names and cropped images. The resulting dataset, Visual Writing Prompts (VWP), yields stories that are more coherent, diverse, and grounded than earlier collections. The authors also propose CharGrid, a generation model with a coherence component modeled on entity grids that tracks character continuity. CharGrid outperforms the previous state of the art, TAPM, on both automatic and human evaluations. This shows that careful data curation and explicit character modeling both improve image-based storytelling.

Key points

• Dataset size: VWP contains almost 2K movie-shot sequences of 5–10 images each, paired with about 12K crowdsourced stories of 50–300 words collected on Amazon Mechanical Turk.
• Character grounding: Each story is linked to at most five main characters, and every mention of a character is tied to a bounding box marking that character in the matching image (taken from MovieNet annotations).
• Filtering pipeline:
  • Fantasy, sci-fi, horror, and animated films were excluded.
  • Near-duplicate frames were removed when the cosine similarity of their ResNet-50 features exceeded 0.89.
• Automatic results: On the VWP test set, CharGrid reaches a CIDEr score of about 4.83, compared with roughly 1.4–1.9 for TAPM variants. Baselines also include GPT-2 with object and character features.
• Human evaluation: 28 workers compared 50 story pairs. CharGrid beat TAPM + char significantly on every criterion: +8.41 points on coherence, +11.06 on diversity, +6.49 on grammaticality, and +6.25 on visual groundedness. Code and data are available at vwprompt.github.io.