Elamur Research⌕
☰
home / papers / 2016 / Pragmatic Factors in Image Description: The Case of Negations

Pragmatic Factors in Image Description: The Case of Negations

Paper · 2016 · Emiel van Miltenburg, Roser Morante and Desmond Elliott

Summary

The paper looks at why crowdworkers sometimes use negation when describing photos, even though they were told to describe only what is visible. The authors searched Flickr30K captions for non-affixal negators and implicit negatives using regular expressions. The implicit negatives were without, sans, minus and verbs such as lack and miss. They ran the same search on MS COCO. Negation turned out to be rare but stable: about half a percent of captions in both corpora. Usually only one of an image's five descriptions contains it, which suggests it is a subjective choice.

The authors read all Flickr30K hits twice, first to build a taxonomy of negation uses and then to apply it. They then double-annotated the corpus with these categories and reached moderate agreement. Most negations mark a salient absence, especially missing clothing. The authors argue that negation reflects a gap between expectation and reality. Generating it would therefore require background knowledge: norms, scripts, action recognition, gaze and scene types. This makes negation a useful window into subjective language in vision-and-language data.

Key points

  • Flickr30K search results:
    • 896 sentences with negations, 892 of them unique, including 31 false positives.
    • The most frequent terms were no (371), not (198), without (141), miss (69) and n't (68).
  • Comparison across corpora:
    • MS COCO train and validation splits gave 3339 negated sentences, 3232 of them unique.
    • The negation rate was 0.56% in Flickr30K and 0.54% in MS COCO, so the phenomenon looks robust across datasets.
    • A single negated description per image was the norm: 86.25% of cases in Flickr30K and 72.05% in MS COCO.
  • Taxonomy: salient absence; negation of action/behavior; negation of property; negation of attitude; outside the frame; (preventing) future events; quotes and idioms; other. A Meta class covers comments about the task itself.
  • Annotation:
    • Two annotators reached Cohen's κ=0.67, with 77% raw agreement.
    • Salient absence was by far the largest category, with 488 sentences.
    • 39 sentences also used negation to set up a contrast.
  • Clothing skew: 282 negations (32%) are about shirtlessness and 59 (7%) about missing shoes. This skew may make it hard for systems to learn other kinds of negation. Code, data and guidelines are released on GitHub (annotating-negations).

Interesting ideas

  • I1. Treat a negation as evidence that the speaker held an expectation which the scene broke, as in a graduate without a cap; mined at scale, such sentences expose default script knowledge.
  • I2. A negated part of a set licenses an inference about the rest: "two of three without shoes" implies the third wears shoes, a scalar implicature drawn from the negation.
  • I3. Separate negations that are mentioned (tattoo text, game names, idioms like "path to nowhere") from negations that are actually asserted about the scene.
  • I4. Compare several independent descriptions of the same item to tell objective content from subjective choices: if only one describer uses a construction, it is optional.
  • I5. A negation sounds natural only when it cancels a plausible expectation, which is why "the Queen isn't wearing jeans" is odd.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): negated clauses in texts could be harvested as candidate defaults for frames and scripts, e.g. "graduation → cap", and stored as typical-but-cancellable properties in the knowledge graph.
  • I2 (frames): the implicature module could add a rule that a negated subset quantifier asserts the complementary property for the remaining members of the group.
  • I3 (frames): quoted or idiomatic negators should be flagged so they do not flip the polarity of a frame or delete a state in the world of the text.
  • I4 (learn): when witness parsers or LLM teacher tiers describe the same text, content produced by only one of them could be marked as subjective or low-confidence instead of being merged into silver data.
  • I5 (back): the generator should emit a negative statement only when the world model holds a default that the current state contradicts; otherwise it should leave the absence unmentioned.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands89 works · 81,337 citations🎓Vrije Universiteit AmsterdamAmsterdam · Netherlands33 works · 3,294 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Emiel van Miltenburg, Roser Morante and Desmond Elliott. 2016. Pragmatic Factors in Image Description: The Case of Negations. Proceedings of the 5th Workshop on Vision and Language.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-3207 · arXiv 1606.06164

@inproceedings{miltenburg2016pragmatic,
  title = {{Pragmatic Factors in Image Description: The Case of Negations}},
  author = {Emiel {van Miltenburg}, Roser Morante and Desmond Elliott},
  booktitle = {Proceedings of the 5th Workshop on Vision and Language},
  year = {2016},
  doi = {10.18653/v1/w16-3207},
  eprint = {1606.06164},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w16-3207}
}

References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The berkeley framenet project. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics -Volume 1, ACL ’98, pages 86–90, Stroudsburg, PA, USA. Association for Computational Linguistics.
  2. Camiel J Beukeboom, Catrin Finkenauer, and Daniël HJ Wigboldus. 2010. The negation bias: when negations signal stereotypic expectancies. Journal of personality and social psychology, 99(6):978. ↗
  3. Nathanael Chambers and Dan Jurafsky. 2009. Unsupervised learning of narrative schemas and their participants. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2, ACL ’09, pages 602–610, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  4. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325. ↗
  5. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. Multi30K: Multilingual English-German Image Descriptions. CoRR, abs/1605.00459. ↗
  6. Ethan Fast, William McGrath, Pranav Rajpurkar, and Michael S. Bernstein. 2016. Augur: Mining human behaviors from fiction to power interactive systems. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, pages 237–247, New York, NY, USA. ACM. find
  7. Charles J Fillmore. 1976. Frame semantics and the nature of language. Annals of the New York Academy of Sciences, 280(1):20–32. ↗
  8. James J. Gibson. 1977. The theory of affordances. In R. E. Shaw and J. Bransford, editors, Perceiving, Acting, and Knowing. Hillsdale, NJ: Lawrence Erlbaum Associates. ↗
  9. Laurence R. Horn. 1972. On the Semantic Properties of Logical Operators in English. Ph.D. thesis, UCLA, Los Angeles. find
  10. Laurence R. Horn. 1989. A natural history of negation. CSLI Publications. find
  11. Geoffrey Leech. 1983. Principles of pragmatics. London and New York: Longman. find
  12. Aude Oliva and Antonio Torralba. 2001. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision, 42(3):145–175. ↗
  13. Marco Pennacchiotti, Diego De Cao, Roberto Basili, Danilo Croce, and Michael Roth. 2008. Automatic induction of framenet lexical units. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 457–465. Association for Computational Linguistics.
  14. Ronald Poppe. 2010. A survey on vision-based human action recognition. Image and Vision Computing, 28(6):976 – 990. ↗
  15. Roger C. Schank and Robert P. Abelson. 1977. Scripts, Plans, Goals, and Understanding: An Inquiry Into Human Knowledge Structures. L. Erlbaum Associates. find
  16. Antonio Torralba, Aude Oliva, Monica S Castelhano, and John M Henderson. 2006. Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychological review, 113(4):766. ↗
  17. Gunnel Tottie. 1980. Affixal and non-affixal negation in English: Two systems in (almost) complementary distribution. Studia linguistica, 34(2):101–123. ↗
  18. Roberto Valenti, Nicu Sebe, and Theo Gevers. 2012. Combining head pose and eye location information for gaze estimation. IEEE Transactions on Image Processing, 21(2):802–815, Feb. ↗
  19. Emiel van Miltenburg. 2016. Stereotyping and bias in the Flickr30K dataset. In Proceedings of the 11 th Workshop on Multimodal Corpora (MMC2016), pages 1–4. ↗
  20. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67– 78. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/miltenburg-2016-pragmatic-factors-image-description-case/