Elamur Research⌕
☰
home / papers / 2021 / Decoupling Pragmatics: Discriminative Decoding for Referring Expression Generation

Decoupling Pragmatics: Discriminative Decoding for Referring Expression Generation

Paper · 2021 · Simeon Schüz and Sina Zarrieß

Summary

Neural Referring Expression Generation (REG) handles realistic photo scenes, but its pragmatic behaviour is buried inside architectures and training objectives. The authors propose to keep a context-agnostic literal generator and add pragmatic reasoning only at inference time. This mirrors classical algorithms such as the Incremental Algorithm, which operate over explicit symbols.

They took the captioning model of Lu et al. (2017), extended it with location features, and trained it on RefCOCO, RefCOCO+ and RefCOCOg. They then decoded it four ways:

  • greedy search
  • beam search
  • Emitter-Suppressor (ES)
  • RSA decoding, where every other object in the image serves as a distractor

The outputs were scored for likelihood (BLEU-1, CIDEr), diversity (type-token ratio, vocabulary coverage) and informativity. Informativity was measured by whether a pre-trained comprehension model locates the intended object.

Discriminative decoding lowered n-gram overlap and raised diversity. It also improved informativity, but only modestly, by at most about 10% relative. The authors suspect that REG training data, written specifically to disambiguate, already teaches the base model much of the needed pragmatics. This raises the open question of whether pragmatics must be modelled explicitly at all.

Key points

  • Data: RefCOCO, RefCOCO+ (interactive collection) and RefCOCOg (non-interactive), all built on MSCOCO images.
    • testA covers people and testB covers other objects.
    • Images containing a single object were dropped, leaving roughly 1950 (testA), 1750 (testB) and 4000 (testg) test objects.
  • Base model: the adaptive-attention captioner of Lu et al. (2017), extended with 7 location features.
  • Decoding: beam size k = 5.
    • ES was tested with λ ∈ {0.7, 0.5, 0.3}; smaller λ gives the suppressor more weight.
    • RSA was tested with α ∈ {0.5, 1.0, 5.0}.
    • Both discriminative methods run inside beam search and support a variable number of distractors (n − 1 for n objects).
  • Informativity metric: detection precision of the Luo et al. (2020) comprehension model, counting a hit when IoU > 0.5.
    • On testA, greedy reaches 75.1, ES λ0.5 reaches 80.6, and human references reach 84.4.
    • On testB+, human references reach only 57.9, which shows how hard the comprehension task itself is.
  • Fluency and diversity trade-off: ES λ0.3 sharply degrades BLEU-1 and CIDEr (e.g. testA BLEU-1 falls to 35.8 from 53.2). It also raises diversity; testg coverage rises from 10.7 (greedy) to 23.6.
    • Human references remain far more diverse, with 44.4 coverage on testg.

Interesting ideas

  • I1. Separate a context-free literal speaker from a pragmatic reranker applied only while decoding, so the pragmatic step stays explicit and adjustable.
  • I2. Measure informativity by having an independent listener model resolve the generated expression and checking whether it lands on the intended referent.
  • I3. Define the distractor set as every other object in the same scene, so the number of distractors varies per instance instead of being fixed.
  • I4. Sweep the rationality weight instead of maximising it, because overly strong pragmatic pressure broke fluency and even lowered informativity.
  • I5. Before crediting a reasoning layer, check whether a strong plain baseline already gets most of the gain from data that was written pragmatically; here greedy search was already strong.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Mova's graph→text realizer could propose literal mentions of an entity, then rerank them by how well they exclude other entities in the world of the text.
  • I2 (back): a round-trip test could feed each generated referring expression into Mova's own coreference and entity resolution and count how often it picks the right entity.
  • I3 (world): candidate distractors for any mention can be drawn from all co-present entities of the same type in the text world, making context sets naturally sized.
  • I4 (learn): any tunable pragmatic or repair weight should be evaluated across a range, with fluency and task metrics tracked together to find the peak.
  • I5 (learn): evaluations of prag modules should include a no-reasoning baseline as a negative control, to show the gain comes from the module rather than the training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Bielefeld UniversityBielefeld · Germany19 works · 968 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Text generation · 145 works

Cite

Cite as Simeon Schüz and Sina Zarrieß. 2021. Decoupling Pragmatics: Discriminative Decoding for Referring Expression Generation. Proceedings of the Reasoning and Interaction Conference (ReInAct 2021).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{schuz2021decoupling,
  title = {{Decoupling Pragmatics: Discriminative Decoding for Referring Expression Generation}},
  author = {Simeon Schüz and Sina Zarrieß},
  booktitle = {Proceedings of the Reasoning and Interaction Conference (ReInAct 2021)},
  year = {2021},
  url = {https://aclanthology.org/2021.reinact-1.7/}
}

References

36 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. on Computational Linguistics, pages 1952–1962, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  2. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173–1182, Austin, Texas. Association for Computational Linguistics. ↗
  3. Emiel Krahmer and Kees van Deemter. 2019. Computational Generation of Referring Expressions: An Updated Survey. find
  4. Emiel Krahmer, Sebastiaan van Erk, and André Verleg. 2003. Graph-based generation of referring expressions. Computational Linguistics, 29(1):53–72. ↗
  5. Alasdair DF Clarke, Micha Elsner, and Hannah Rohde. 2013. Where’s wally: the influence of visual salience on referring expression generation. Frontiers in psychology, 4. ↗
  6. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer. ↗
  7. Reuben Cohn-Gordon, Noah Goodman, and Christopher Potts. 2018. Pragmatically informative image captioning with character-level inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 439–443, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  8. Jingyu Liu, Liang Wang, and Ming-Hsuan Yang. 2017. Referring expression generation and comprehension via attributes. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE. find
  9. Jingyu Liu, Wei Wang, Liang Wang, and Ming-Hsuan Yang. 2020. Attribute-guided attention for referring expression generation and comprehension. IEEE Transactions on Image Processing, 29:5244–5258. ↗
  10. Robert Dale. 1989. Cooking up referring expressions. In 27th Annual Meeting of the Association for Computational Linguistics, pages 68–75, Vancouver, British Columbia, Canada. Association for Computational Linguistics. find
  11. Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 375–383. find
  12. Robert Dale and Ehud Reiter. 1995. Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive Science, 19(2):233–263. ↗
  13. Kees van Deemter, Albert Gatt, Ielka van der Sluis, and Richard Power. 2012. Generation of referring expressions: Assessing the incremental algorithm. Cognitive Science, 36(5):799–836. ↗
  14. Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. 2020. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  15. Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998. R. ↗
  16. Dimitra Gkatzia, Verena Rieser, Phil Bartie, and William Mackaness. 2015. From the virtual to the real world: Referring to objects in real-world spatial scenes. In Proceedings of EMNLP 2015. Association for Computational Linguistics. find
  17. Luo and Gregory Shakhnarovich. 2017. Comprehension-guided referring expressions. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3125–3134. find
  18. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11–20.
  19. H. P. Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Semantics: Vol. 3: Speech Acts, pages 41–58. Academic Press, New York. ↗
  20. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2015. Generation and comprehension of unambiguous object descriptions. ArXiv / CoRR, abs/1511.02283. ↗
  21. Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L Berg. 2014. ReferItGame: Referring to Objects in Photographs of Natural Scenes. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), pages 787–798, Doha, Qatar.
  22. Emiel van Miltenburg, Desmond Elliott, and Piek Vossen. 2018. Measuring the diversity of automatic image descriptions. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1730–1741, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  23. Jungjun Kim, Hanbin Ko, and Jialin Wu. 2020. Co-NAN: A complementary neighboring-based attention network for referring expression generation. In Proceedings of the 28th International Conference Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. 2017. A joint speaker-listener-reinforcer model for referring expressions. In Computer Vision and Pattern Recognition (CVPR), volume 2. find
  24. Margaret Mitchell, Kees van Deemter, and Ehud Reiter. 2010. Natural reference to objects in a visual domain. In Proceedings of the 6th international natural language generation conference, pages 95–104. Association for Computational Linguistics. find
  25. Sina Zarrieß and David Schlangen. 2016. Easy things first: Installments improve referring expression generation for objects in photographs. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 610–620, Berlin, Germany. Association for Computational Linguistics. find
  26. Aditya Mogadala, Marimuthu Kalimuthu, and Dietrich Klakow. 2019. Trends in integration of vision and language research: A survey of tasks, datasets, and methods. CoRR, abs/1907.09358. ↗
  27. Nikolaos Panagiaris, Emma Hart, and Dimitra Gkatzia. 2020. Improving the naturalness and diversity of referring expression generation models using minimum risk training. In Proceedings of the 13th International Conference on Natural Language Generation, pages 41–51, Dublin, Ireland. Association for Computational Linguistics. find
  28. Sina Zarrieß and David Schlangen. 2018. Decoding strategies for neural referring expression generation. In Proceedings of the 11th International Conference on Natural Language Generation, pages 503–512, Tilburg University, The Netherlands. Association for Computational Linguistics. find
  29. Nikolaos Panagiaris, Emma Hart, and Dimitra Gkatzia. 2021. Generating unambiguous and diverse referring expressions. Computer Speech & Language, 68:101184. ↗
  30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  31. Ehud Reiter and Robert Dale. 2000. Building natural language generation systems. Cambridge university press. ↗
  32. Simeon Schüz, Ting Han, and Sina Zarrieß. 2021. Diversity as a by-product: Goal-oriented language generation leads to linguistic variation. In Proceedings of the 22th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 175–185. Association for Computational Linguistics. find
  33. M. Tanaka, Takayuki Itamochi, K. Narioka, Ikuro Sato, Y. Ushiku, and T. Harada. 2019. Generating easy-to-understand referring expressions for target identifications. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 5793–5802. ↗
  34. Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 251–260. find
  35. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575. ↗
  36. Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. 2016. Modeling context in referring expressions. In Computer Vision – ECCV 2016, pages 69–85, Cham. Springer International Publishing.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schuz-2021-decoupling-pragmatics-discriminative-decoding-referring/