Elamur Research⌕
☰
home / papers / 2023 / Evaluating pragmatic abilities of image captioners on A3DS

Evaluating pragmatic abilities of image captioners on A3DS

Paper · 2023 · Polina Tsvilodub and Michael Franke

Summary

Standard caption metrics such as BLEU, ROUGE, CIDEr and METEOR compare surface strings. They cannot tell whether a caption does its communicative job, for example picking out a target image among distractors without saying too much. To support that kind of check, the authors built A3DS (Annotated 3D Shapes). It adds language annotations to the 480,000 synthetic images of 3DShapes. A hand-written grammar produces the captions from the dataset's numeric feature labels, so every phrase maps to a known feature value. This lets the authors measure, feature by feature, what a caption mentions: contrastive, redundant or false features. That works even when the output is ungrammatical. They define four Grice-inspired metrics. They then compare a task-neutral LSTM captioner with versions fine-tuned via reinforcement learning in a two-agent reference game. Fine-tuned speakers say much less and are far more efficient and relevant, though less often discriminative. Like humans, they show a bias toward shape and other object features. The work offers a cheap way to audit pragmatic behaviour without collecting human judgments.

Key points

  • Dataset. A3DS covers 480,000 images that vary along six features:
    • shape (4 values), object colour (10), scale (8), orientation (15), wall colour (10) and floor colour (10).
    • It provides over 9 million exhaustive captions (all six features) and over 12 million short captions (2–3 features).
  • Metrics. The authors define discriminativity, contrastive efficiency, relevance and optimal discriminativity. These are computed from:
    • contrastive features mentioned,
    • total features mentioned,
    • the true number of contrastive features.
  • Setup.
    • The baseline was pretrained on 150,000 exhaustive-caption pairs drawn from 30,000 images, then fine-tuned on another 150,000 pairs in a reference game with reward +1/−1.
    • Both agents use a frozen ResNet-50, a 512-d projection and a one-layer LSTM (h = 512).
    • There are two fine-tuning regimes: random pairs (RP) and similar pairs (SP).
  • Results on one-feature pairs.
    • The baseline is almost always discriminative (0.999) but mentions about 5.9 features, with efficiency of 0.198.
    • RP reaches discriminativity 0.822, efficiency 0.879 and relevance 0.668, using about 2.9 features.
    • RP's listener accuracy is 0.919.
  • False features and the shape shortcut.
    • Average false features fall from 0.720 for the baseline to 0.139 for RP and 0.316 for SP.
    • Discriminativity collapses when shape is not the contrastive feature. On three-feature pairs matched on object features, it drops to 0.087 for RP and 0.164 for SP.

Interesting ideas

  • I1. Because a grammar generates every caption from labels, each phrase can be traced back to a feature, so outputs can be scored by function rather than string overlap.
  • I2. Every mentioned property is sorted into contrastive, redundant or false, and separate scores tie to Quantity and Relevance.
  • I3. An ambiguous colour word only counts if it attaches to the correct head noun, such as floor, wall or the object.
  • I4. Test sets are split by which features the two images share, and this exposed a hidden reliance on shape that averages would hide.
  • I5. Training on exhaustive captions rules out thin data as the reason a model fails to be contrastive, isolating the behaviour under test.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): generated text from the meaning graph could be scored by which graph facts its phrases recover, not by n-gram match to a reference.
  • I2 (frames): the implicature module could grade referring expressions in a text the same way, flagging overinformative mentions as candidates for implied contrast.
  • I3 (tree): an amod/nmod arc to the right head is an explainable way to make that binding check, instead of word co-occurrence.
  • I4 (learn): build evaluation splits that hold specific cues constant, as negative controls against shortcut heuristics in the readers.
  • I5 (world): describing a text world's entities exhaustively before testing selection lets errors be blamed on choosing what to say, not on missing knowledge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TübingenTübingen · Germany75 works · 41,081 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Polina Tsvilodub and Michael Franke. 2023. Evaluating pragmatic abilities of image captioners on A3DS. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-short.110 · arXiv 2305.12777

@inproceedings{tsvilodub2023evaluating,
  title = {{Evaluating pragmatic abilities of image captioners on A3DS}},
  author = {Polina Tsvilodub and Michael Franke},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-short.110},
  eprint = {2305.12777},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-short.110}
}

References

26 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173– 1182, Austin, Texas. Association for Computational Linguistics. ↗
  2. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  3. Chris Burgess and Hyunjik Kim. 2018. 3d shapes dataset. https://github.com/deepmind/3dshapes-dataset/. ↗
  4. Reuben Cohn-Gordon, Noah Goodman, and Christopher Potts. 2018. Pragmatically informative image captioning with character-level inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 439–443, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  5. Judith Degen, Robert D Hawkins, Caroline Graf, Elisa Kreiss, and Noah D Goodman. 2020. When redundancy is useful: A Bayesian approach to “overinformative” referring expressions. Psychological Review, 127(4):591. find
  6. Noah D Goodman and Michael C Frank. 2016. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20(11):818–829. ↗
  7. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. ↗
  8. Serhii Havrylov and Ivan Titov. 2017. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. Advances in neural information processing systems, 30. find
  9. Andrej Karpathy, Armand Joulin, and Li F Fei-Fei. 2014. Deep fragment embeddings for bidirectional image sentence mapping. Advances in neural information processing systems, 27. ↗
  10. Hyunjik Kim and Andriy Mnih. 2018. Disentangling by factorising. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2649–2658. PMLR. ↗
  11. Emiel Kramer and Kees van Deemter. 2012. Computational generation of referring expressions: A survey. Computational Linguistics, 38(1):173–218.
  12. Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. 2017. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253. ↗
  13. Angeliki Lazaridou and Marco Baroni. 2020. Emergent multi-agent communication in the deep learning era. ArXiv, abs/2006.02419. ↗
  14. Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. 2016. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182. ↗
  15. Angeliki Lazaridou, Anna Potapenko, and Olivier Tiele-man. 2020. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7663–7674, Online. Association for Computational Linguistics. find
  16. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  18. Gregory Scontras, Judith Degen, and Noah D Goodman. 2017. Subjectivity predicts adjective ordering preferences. Open Mind, 1(1):53–66. ↗
  19. John R Searle. 1969. Speech acts: An essay in the philosophy of language, volume 626. Cambridge university press. find
  20. Sheng Shen, Daniel Fried, Jacob Andreas, and Dan Klein. 2019. Pragmatically informative text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4060–4067, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  21. Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 251–260. find
  22. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575. ↗
  23. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2016. Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge. IEEE transactions on pattern analysis and machine intelligence, 39(4):652–663. ↗
  24. Sina Zarrieß, Hendrik Buschmeier, Ting Han, and Simeon Schüz. 2021. Decoding, fast and slow: A case study on balancing trade-offs in incremental, character-level pragmatic reasoning. In Proceedings of the 14th International Conference on Natural Language Generation, pages 371–376, Aberdeen, Scotland, UK. Association for Computational Linguistics.
  25. Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and VQA. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13041–13049. find
  26. ACL 2023 Responsible NLP Checklist find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tsvilodub-2023-evaluating-pragmatic-abilities-image-captioners/