Elamur Research
☰
home / papers / 2018 / Pragmatically Informative Image Captioning with Character-Level Inference

Pragmatically Informative Image Captioning with Character-Level Inference

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N18-2070.pdf

Summary

Standard neural image captioners describe an image accurately but often fail to say what sets it apart from similar pictures. This paper adds a Rational Speech Acts (RSA) model to a captioner. RSA has the speaker reason about a listener who must pick out the target image from a set of distractors. Earlier neural RSA work had to normalize over all possible captions, which is intractable, so it relied on a small sampled set of candidate utterances. The authors instead apply RSA incrementally, one character at a time, using a character-level LSTM in a CNN-RNN captioner. At each step the speaker chooses among roughly 30 symbols, and the listener's beliefs about the images carry over from step to step. The models are trained on Visual Genome regions and MSCOCO. Evaluation is automatic: a separately trained listener model tries to identify the target among 10 similar images. The pragmatic character-level speaker identifies targets far more reliably than the literal captioner and also beats a word-level RSA variant. This shows that local character choices can produce caption-level pragmatic effects cheaply.

Key points

  • Accuracy on TS1 / TS2 (each 100 clusters of 10 images):
    • Char S0: 48.9 / 47.5
    • Char S1: 68.0 / 65.9
    • Word S0: 57.6 / 53.4
    • Word S1: 60.6 / 57.6
  • The word S0 is stronger than the char S0, but the char S1 beats the word S1. The authors attribute this to the many more decision points where pragmatic reasoning can act.
  • The search space per step is about 30 characters, compared with about 20,000 words. Settings used: beam width 10 and rationality α = 5.0.
  • Consider the cases where S1 succeeded and S0 failed. In 66% of them, the S1 caption was not among S0's top-50 beam outputs, so a sampling-based (non-incremental) RSA could not have produced it.
  • Training data were split in half, one part for the caption generator and one for the evaluation listener L_eval. The code is released as Recurrent-RSA on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

41 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 34 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-2070

@inproceedings{cohngordon2018pragmatically,
  title = {{Pragmatically Informative Image Captioning with Character-Level Inference}},
  author = {Cohn-Gordon, Reuben and Goodman, Noah and Potts, Christopher},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-2070},
  url = {https://doi.org/10.18653/v1/n18-2070}
}

References

10 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173–1182. Association for Computational Linguistics. find our card
  2. Will Monroe, Robert X. D. Hawkins, Noah D. Goodman, and Christopher Potts. 2017. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5:325–338. find our card
  3. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. ↗ our card
  4. Will Monroe and Christopher Potts. 2015. Learning in the Rational Speech Acts model. In Proceedings of 20th Amsterdam Colloquium, Amsterdam. ILLC. find
  5. Robert Dale and Ehud Reiter. 1995. Computational interpretations of the Gricean maxims in the generation of referring expressions. Cognitive Science, 19(2):233–263. ↗ our card
  6. Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. arXiv preprint arXiv:1701.02870. ↗
  7. Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010. Every picture tells a story: Generating sentences from images. In European Conference on Computer Vision, pages 15–29. Springer. find
  8. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3156–3164. find our card
  9. Michael C. Frank and Noah D. Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998. find our card
  10. Noah D Goodman and Andreas Stuhlmüller. 2013. Knowledge and implicature: Modeling language understanding as social cognition. Topics in Cognitive Science, 5(1):173–184. find