Elamur Research⌕
☰
home / papers / 2023 / Pragmatic Inference with a CLIP Listener for Contrastive Captioning

Pragmatic Inference with a CLIP Listener for Contrastive Captioning

Paper · 2023 · Jiefu Ou, Benno Krojer and Daniel Fried

Summary

The paper addresses contrastive (discriminative) captioning: describing a target image so that a person can pick it out from a set of nearly identical distractors. Earlier pragmatic methods use the captioning model itself as both speaker and listener. These methods are fragile with respect to the informativity weight, and when pushed toward informativity they produce garbled text. The authors propose PICL, which keeps the Rational Speech Acts style of reasoning but uses an off-the-shelf CLIP model as an independent listener. At each beam-search step, an OFA captioner fine-tuned on MSCOCO proposes candidate partial captions. CLIP then scores each candidate against all images in the set, and the two distributions are mixed through a weight λ to rerank the candidates. The method needs no training. Experiments use the static-image part of ImageCoDe, the first time this dataset has been used for captioning. Evaluation combines ALBEF retrieval, GPT-2 perplexity and human judgments. PICL tolerates the choice of λ much better than earlier methods, so λ can be tuned automatically. Under this automatic tuning, humans identify the target 11–15% more accurately than with baseline captions.

Key points

  • Each ImageCoDe set contains one target and 9 highly similar distractors. Only the static partition was used, because the video-frame partition remains too hard for all methods.
  • The speaker is OFA-base fine-tuned on MSCOCO captioning. The listener is zero-shot CLIP, whose cosine similarities are softmax-normalized over the images in the set. The pragmatic score combines the listener score raised to λ with the speaker score raised to 1−λ.
  • Scoring the whole vocabulary with CLIP at every step is too costly. Instead, N candidates are drawn from the speaker by beam search, rescored, and pruned to the top B (N > B).
  • In the automated ablation, full PICL scores 77.3. Reranking only complete captions ("- incremental") drops this to 65.4, and dropping distractors ("- distractors") drops it to 57.5.
  • Human evaluation covered 100 test image sets. ALBEF accuracy tracked human accuracy, except for disfluent baseline captions (E-S, Incre-RSA) and human-written captions. GPT-2 perplexity correlated strongly with human fluency ratings.

Interesting ideas

  • I1. Let a model that is independent of the generator act as the judge, because a generator grading its own outputs rewards text that only it finds distinctive.
  • I2. Score a description against every candidate referent and normalize over the set, so the score measures how much the description singles out one item.
  • I3. Apply the discriminative check to partial outputs during decoding, using cheap proposals from the base model and expensive reranking of a small subset.
  • I4. Push hard on a model-predicted target only together with a fluency control, and compare systems at matched perplexity. Maximizing model-judged informativity alone yielded captions that misled humans.
  • I5. Ablate the contrast set itself, for example by running with the target only and no distractors, to show that gains come from the contrast and not from reranking alone.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): when generating text from the meaning graph, have a separate parser read each candidate sentence back and check that it recovers the intended graph rather than a nearby one.
  • I2 (frames): in coreference, rank a mention by how strongly it selects one entity over the competing entities in the text world, not by raw compatibility with each entity in isolation.
  • I3 (back): when building referring expressions for an entity, prune growing phrases step by step by how well they separate the entity from similar ones in the world, keeping the cost bounded.
  • I4 (learn): when tuning against a single automatic witness, add a fluency or sanity constraint and report results at equal constraint levels, so the system does not learn to game the witness.
  • I5 (learn): add negative controls that remove the competing alternatives, such as distractor answers or rival entities, to confirm that a component's improvement truly depends on contrastive reasoning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,225 citations🎓McGill UniversityMontreal · Canada91 works · 11,178 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Jiefu Ou, Benno Krojer and Daniel Fried. 2023. Pragmatic Inference with a CLIP Listener for Contrastive Captioning. Findings of the Association for Computational Linguistics: ACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-acl.120 · arXiv 2306.08818

@inproceedings{ou2023pragmatic,
  title = {{Pragmatic Inference with a CLIP Listener for Contrastive Captioning}},
  author = {Jiefu Ou, Benno Krojer and Daniel Fried},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.120},
  eprint = {2306.08818},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.120}
}

References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173– 1182, Austin, Texas. Association for Computational Linguistics. ↗
  2. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint, arXiv:1504.00325. ↗
  3. Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal. 2022. Fine-grained image captioning with CLIP reward. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 517–527, Seattle, United States. Association for Computational Linguistics. find
  4. Reuben Cohn-Gordon and Noah Goodman. 2019. Lost in machine translation: A method to reduce meaning loss. ↗
  5. Reuben Cohn-Gordon, Noah Goodman, and Christopher Potts. 2018. Pragmatically informative image captioning with character-level inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 439–443, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  6. Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. 2022. Image retrieval from contextual descriptions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3426–3440, Dublin, Ireland. Association for Computational Linguistics. ↗
  7. Roberto Dessì, Eleonora Gualdoni, Francesca Franzon, Gemma Boleda, and Marco Baroni. 2022. Communication breakdown: On the low mutual intelligibility between human and neural captioning. In Proceedings of EMNLP. find
  8. Angeliki Lazaridou, Anna Potapenko, and Olivier Tiele-man. 2020. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7663–7674, Online. Association for Computational Linguistics. find
  9. Michael C Frank and Noah D Goodman. 2012. Predicting Pragmatic Reasoning in Language Games. Science, 336(6084):998–998. ↗
  10. Daniel Fried, Jacob Andreas, and Dan Klein. 2018. Unified pragmatic models for generating and following instructions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1951–1963, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  11. Daniel Fried, Justin Chiu, and Dan Klein. 2021. Reference-centric models for grounded collaborative dialogue. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2130–2147, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  12. Noah D Goodman and Michael C Frank. 2016. Pragmatic Language Interpretation as Probabilistic Inference. Trends in Cognitive Sciences, 20(11):818–829. ↗
  13. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514–7528, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  14. Ukyo Honda, Taro Watanabe, and Yuji Matsumoto. 2022. Switching to discriminative image captioning by relieving a bottleneck of reinforcement learning. pages 1124–1134. find
  15. Hyunwoo Kim, Byeongchang Kim, and Gunhee Kim. 2020. Will I sound like me? improving persona consistency in dialogues through pragmatic Self-Consciousness.
  16. Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh. 2019. CoDraw: Collaborative drawing as a testbed for grounded goal-driven communication. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6495–6513, Florence, Italy. Association for Computational Linguistics.
  17. Junnan Li, Yongkang Wong, Qi Zhao, and M. Kankan-halli. 2020. Video storytelling: Textual summaries for events. IEEE Transactions on Multimedia, 22:554–565. ↗
  18. Lixin Liu, Jiajun Tang, Xiaojun Wan, and Zongming Guo. 2019. Generating diverse and descriptive image captions using visual paraphrases. pages 4239–4248. find
  19. Xihui Liu, Hongsheng Li, Jing Shao, Dapeng Chen, and Xiaogang Wang. 2018. Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XV, pages 353–369. find
  20. Ruotian Luo, Brian L. Price, Scott D. Cohen, and Gregory Shakhnarovich. 2018. Discriminability objective for training descriptive captions. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6964–6974. find
  21. Ruotian Luo and Gregory Shakhnarovich. 2017. Comprehension-guided referring expressions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7102–7111. find
  22. Yangjun Mao, Long Chen, Zhihong Jiang, Dong Zhang, Zhimeng Zhang, Jian Shao, and Jun Xiao. 2022. Rethinking the reference-based distinctive image captioning. find
  23. Will Monroe, Robert X.D. Hawkins, Noah D. Goodman, and Christopher Potts. 2017. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5:325–338. ↗
  24. Benjamin Newman, Reuben Cohn-Gordon, and Christopher Potts. 2020. Communication-based evaluation for natural language generation. In Proceedings of the Society for Computation in Linguistics 2020, pages 116–126, New York, New York. Association for Computational Linguistics. ↗
  25. Allen Nie, Reuben Cohn-Gordon, and Christopher Potts. 2020. Pragmatic issue-sensitive image captioning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1924–1938, Online. Association for Computational Linguistics.
  26. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. ↗
  27. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  28. Sheng Shen, Daniel Fried, Jacob Andreas, and Dan Klein. 2019. Pragmatically informative text generation. In Proceedings of NAACL. ↗
  29. Ece Takmaz, Sandro Pezzelle, and Raquel Fernández. 2022. Less descriptive yet discriminative: Quantifying the properties of multimodal referring utterances via CLIP. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 36–42, Dublin, Ireland. Association for Computational Linguistics. Searching Method We conduct all the hyperparameter searching via coarse-to-fine search, with step sizes 0.1, 0.01, and 0.001 respectively. A.2.2 Beam Search Parameters For beam search parameters B, N discussed in Section 3.2, we set B = 16 and N = 256. A.3 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ou-2023-pragmatic-inference-clip-listener-contrastive/