“Caption” as a Coherence Relation: Evidence and Implications

Summary

The paper asks whether standard image–caption datasets can support general models that connect pictures and language. The authors argue that they cannot, because captioning is a specific coherence relation in the sense of Hobbs (1990), with its own limits on content and perspective. They compare verb usage across five image–text corpora: crowdsourced COCO, Flickr30K and VIST, plus Conceptual Captions and the Recipe dataset, which pair images with text written for other purposes. The reference points are ANC and Facebook children's stories. Verbs, tense and aspect were extracted with SpaCy-based rules, and 500 sentences per corpus were hand-annotated for lexical aspect.

Caption corpora use few verbs, and those verbs are repetitive. They favour present progressive forms, visible events and atelic or stative descriptions. Models trained on COCO inherit this narrow pattern. The Recipe data, where text and images illustrate each other, looks much more like ordinary English. The authors conclude that retrieval, illustrated-document generation and grounded semantics need more naturally occurring data, and that the different text–image relations should be modelled explicitly.

Key points

• Verb density: Verbs make up 0.184 of ANC tokens. The caption corpora fall well below this, with CC at 0.065, COCO at 0.026, VIST at 0.017 and Flickr at just 0.012.
• Verb concentration: The 10 most frequent verb lemmas account for about 0.60 of verb tokens in COCO and Flickr. The figure is 0.443 in ANC and 0.392 in Recipe.
• Frequent Caption Verbs (FCVs): The authors list 17 such verbs, e.g. sitting, holding, wearing. They make up 0.535–0.638 of the verbs output by eight captioning models trained on COCO.
• Telicity: Telic verbs are scarce in Flickr and COCO, at about 28–29 per 500 sentences. Recipe and ANC have about 300 each, and the CC telic rate is 0.252. Flickr≈COCO and Recipe≈ANC, while all other pairs differ significantly. Annotator agreement was Cohen's κ = 0.77.
• Atelicity is systematic: In Flickr and COCO, FCVs yield atelic readings 96% of the time, compared with 39% in ANC. The annotations are released on GitHub (malihealikhani/Captions).