Elamur Research
☰
home / papers / 2019 / Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories

Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories

What it is about

A speaker must refer to an object in an image whose category the speaker does not know, for example a bus when only cars were seen in training. A neural literal speaker trained on known categories tends to name the wrong category. The paper adds a pragmatic decoder in the style of Rational Speech Acts: the listener keeps a belief over the object's category, and the speaker chooses each word by a score combining the literal speaker and the listener, with weights that favour words not yet used. Uncertainty is set to uniform over the categories. Six medium-frequency categories (cat, horse, cup, bottle, bus, train) are held out from the RefCOCO training data. In Table 1, the share of expressions that contain a wrong category noun falls for all six categories: for example from 39.8% to 16.6% for bottle and from 62.7% to 33.2% for cup. The share of expressions with no noun at all rises, from 27.5% to 56.2% for bottle. The reference-resolution test uses a listener that knows the true categories, to avoid training the evaluator on the same mistakes. The results are mixed across the two test sets; the authors call them positive evidence for communicative success.

Ideas

For Mova

The pragmatic speaker belongs with prag/src/rsa.rs and prag/src/roles.rs. Unknown-word handling relates to en/src/dict.rs and languages/en/19-lexicon-mwe-register.md. Reference and the choice of noun belong in languages/en/35-reference-and-coreference.md, and the evaluation idea fits coref/src/eval.rs. Measures: the wrong-category rate and the no-noun rate on held-out lessons, the resolution accuracy with a listener that knows the categories, run twice for determinism.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1063.pdf

Summary

Zarrieß and Schlangen study a specific problem: how a referring expression generator should describe an object whose category it never saw in training. A plain neural speaker in this situation tends to confidently use the wrong object name. The authors frame this as a "zero-shot reference game" and build a pragmatic speaker inspired by Rational Speech Acts (RSA). A standard CNN-LSTM generator serves as the literal speaker. It is decoded word by word through a Bayesian listener that holds latent beliefs about the target's category. Word probabilities conditioned on categories, estimated from training data, tell the speaker which words are category-specific and therefore risky. When the category is uncertain, the speaker shifts toward attributes such as colour or location. Experiments use RefCOCO, holding out six MSCOCO categories. Compared with the literal speaker, the pragmatic speaker produces fewer wrong nouns and more noun-free expressions. A competent automatic listener resolves its expressions more accurately in most settings with similar-category distractors. The work recasts zero-shot learning as a problem of communicating under uncertainty rather than a pure labelling task.

Key points

  • Data and held-out categories: The experiments use RefCOCO referring expressions over MSCOCO images. Six of the 90 MSCOCO categories (cat, horse, cup, bottle, bus, train) are held out. For each one, every training image containing that category is moved to the test set.
  • Literal speaker (S0): A CNN-LSTM in the style of the Yu et al. (2016) baseline, using VGG fc features. Settings: embedding size 512, hidden state 1024, Adam optimizer, learning rate 0.0004, batch size 32, 5 epochs.
  • Pragmatic speaker (S1): Decoding is greedy and incremental. It weights S0 by the listener's score raised to α+β, where α=2 and β=2 applies only to words already present in the prefix.
  • Fewer wrong names: The share of outputs containing a wrong (distractor) category noun drops for every held-out category. For example, cup falls from 0.627 to 0.332 and bottle from 0.398 to 0.166. Noun-free outputs rise, e.g. bottle goes from 0.275 to 0.562. This behaviour emerges purely from the Bayesian decoding, with no explicit part-of-speech knowledge.
  • Communicative success: The evaluation listener has access to ground-truth categories.
    • On TS-distractors (target plus four similar-category objects), S1 beats S0 for all categories except bus (e.g. train 0.202 → 0.305).
    • On TS-image (other objects from the same image), accuracies are higher overall, but S1 versus S0 results are mixed.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1063

@inproceedings{zarries2019know,
  title = {{Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories}},
  author = {Zarrieß, Sina and Schlangen, David},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1063},
  url = {https://doi.org/10.18653/v1/p19-1063}
}

References

23 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emiel Krahmer and Kees Van Deemter. 2012. Computational generation of referring expressions: A survey. Computational Linguistics, 38(1):173–218. find
  2. Harsh Agrawal, Karan Desai, Xinlei Chen, Rishabh Jain, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2018. nocaps: novel object captioning at scale. arXiv preprint arXiv:1812.08658. ↗
  3. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73. find our card
  4. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided open vocabulary image captioning with constrained beam search. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 936–945, Copenhagen, Denmark. Association for Computational Linguistics. find
  5. Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. 2009. Learning to detect unseen object classes by between-class attribute transfer. In IEEE Computer Vision and Pattern Recognition, pages 951–958. IEEE. find
  6. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173–1182, Austin, Texas. Association for Computational Linguistics. ↗ our card
  7. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C.Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision – ECCV 2014, volume 8693, pages 740–755. Springer International Publishing. find our card
  8. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. ↗ our card
  9. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In CVPR 2016. find our card
  10. Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22:1– 39. find
  11. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. ↗ our card
  12. Reuben Cohn-Gordon, Noah Goodman, and Christopher Potts. 2018. Pragmatically informative image captioning with character-level inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 439–443. Association for Computational Linguistics. find our card
  13. Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. 2013. Zero-shot learning through cross-modal transfer. In Advances in neural information processing systems, pages 935–943. find
  14. Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. In Computer Vision and Pattern Recognition (CVPR), volume 3. find
  15. Robert Dale and Ehud Reiter. 1995. Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive Science, 19(2):233–263. ↗ our card
  16. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09. find
  17. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Computer Vision and Pattern Recognition. find our card
  18. Rui Fang, Malcolm Doering, and Joyce Y Chai. 2014. Collaborative models for referring expression generation in situated dialogue. In Twenty-Eighth AAAI Conference on Artificial Intelligence. find
  19. Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. 2016. Modeling Context in Referring Expressions, pages 69–85. Springer International Publishing, Cham. find our card
  20. Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998. ↗ our card
  21. Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. 2017. A joint speakerlistener-reinforcer model for referring expressions. In Computer Vision and Pattern Recognition (CVPR), volume 2. find
  22. Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Darrell. 2016. Deep compositional captioning: Describing novel object categories without paired training data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–10. find
  23. Sina Zarrieß and David Schlangen. 2018. Decoding strategies for neural referring expression generation. Proceedings of INLG 2018. find