Elamur Research⌕
☰
home / papers / 2014 / Information Structure Prediction for Visual-world Referring Expressions

Information Structure Prediction for Visual-world Referring Expressions

Paper · 2014 · Micha Elsner, Hannah Rohde and Alasdair Clarke

Summary

Referring expression generation in visual scenes has mostly studied content selection, meaning which landmarks to mention. Surface ordering has usually been left to fixed templates. This paper asks what decides the order in which a speaker mentions the target ("anchor") and a helper object ("landmark") in a relational description. The authors use the Mechanical Turk corpus of "Where's Wally" descriptions from Clarke et al. (2013). They annotate each anchor–landmark pair with one of three orderings:

  • PRECEDE: the landmark comes first.
  • FOLLOW: the landmark comes after the anchor.
  • INTER: the landmark sits between pieces of the anchor description.

Each pair also gets an ESTABLISH flag, set when the landmark is first introduced on its own. Image regions are treated separately from real objects. Logistic regression shows that large, visually prominent objects tend to be mentioned early, the way given information is in discourse. A maximum-entropy classifier beats simple baselines, with the clearest gains on non-canonical orders. The main claim is that visual prominence can put an entity into common ground without any prior mention. Template realizers therefore miss real variation in how people order these descriptions.

Key points

  • Data: 1672 descriptions from 152 participants, covering 11 Wally scenes with 16 designated targets per scene. Annotations mark which substrings refer to the target, to other objects, and to image regions.
  • Method: a maxent multilabel classifier, trained with Theano, predicts an (ESTABLISH, direction) pair for each anchor–landmark link.
    • Features include area, salience, distance to the image centre and inter-object distance.
    • These values are discretized into training-set quartiles, which worked better than continuous values.
  • Evaluation setup: three images were held out for development. Testing was leave-one-image-out over the remaining 8, with training on the other 10 images, giving 2382 test pairs.
  • Results:
    • Overall pair accuracy rose from 43% (a "regions precede, landmarks follow" baseline) to 52%.
    • Direction accuracy rose from roughly 42–44% to 57%.
    • ESTABLISH reached a 60% F-score, about equal to a lower-bound estimate of inter-annotator agreement.
    • The ceiling from inter-subject agreement is about 74% pair accuracy.
    • All gains over baselines are significant (sign test, p < 0.01).
  • Negative findings:
    • Adding the Torralba et al. (2006) visual distinctiveness model did not help.
    • A parser-like structured classifier did no better than independent pair classification, possibly because the description trees are shallow.

Interesting ideas

  • I1. Things that are perceptually or contextually obvious behave as "given" even when nothing has mentioned them, so givenness should not be tied only to earlier text.
  • I2. The authors describe a description's structure as an order between object references (precede, follow, interleaved, established first) rather than as word order, which gives a compact and interpretable target to predict.
  • I3. Continuous cues are binned into training-set quartiles before a linear model sees them, which captures nonlinear effects cheaply and keeps the features readable.
  • I4. Systems are compared against both trivial baselines and a measure of agreement between human speakers, so that 52% can be read against a ceiling of about 74%.
  • I5. The authors report honestly that a joint structured model gave no gain over independent pair decisions when the structures were shallow.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): coreference and information-status tagging in prag could treat situationally prominent entities (for example a story's protagonist or a well-known name) as old information, which would help explain why they appear in subject or fronted position.
  • I2 (back): the generator could first choose an ordering strategy for each pair of entities in the world graph, then realize the text, so that every ordering decision can be explained.
  • I3 (learn): perceptron-style models in Mova could quartile-bin numeric features such as distance or length, keeping them human-inspectable while handling nonlinear effects.
  • I4 (learn): Mova's evaluations could report inter-witness or inter-annotator agreement next to negative-control baselines, so that scores have both a floor and a ceiling.
  • I5 (tree): before adding joint or global decoding to gparse-related components, check how deep the target structures really are, because local classifiers may already be enough.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations🎓The Ohio State UniversityColumbus · United States57 works · 17,793 citations

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 works

Cite

Cite as Micha Elsner, Hannah Rohde and Alasdair Clarke. 2014. Information Structure Prediction for Visual-world Referring Expressions. Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/e14-1055

@inproceedings{elsner2014information,
  title = {{Information Structure Prediction for Visual-world Referring Expressions}},
  author = {Micha Elsner, Hannah Rohde and Alasdair Clarke},
  booktitle = {Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.3115/v1/e14-1055},
  url = {https://doi.org/10.3115/v1/e14-1055}
}

References

48 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. A. Baldwin. 1995. Understanding the link between joint attention and language. In Joint attention: its origins and role in development. Lawrence Erlbaum Assoc., Hillsdale, NJ. find
  2. D. Bates, M. Maechler, and B. Bolker. 2011. lme4: Linear mixed-effects models using s4 classes. Comprehensive R Archive Network: cran.r-project.org. find
  3. David L. Bean and Ellen Riloff. 1999. Corpus-based identification of non-anaphoric noun phrases. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics (ACL’99), pages 373–380, Morristown, NJ, USA. Association for Computational Linguistics.
  4. Anja Belz and Albert Gatt. 2008. Intrinsic vs. extrinsic evaluation measures for referring expression generation. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, pages 197–200. Association for Computational Linguistics. find
  5. Yoshua Bengio. 2009. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1):1–127. Also published as a book. ↗
  6. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June. Oral Presentation. find
  7. Robbert-Jan Beun and Anita H.M. Cremers. 1998. Object reference in a shared domain of conversation. Pragmatics and Cognition, 6(1-2):121–152. ↗
  8. Joan Bresnan, Anna Cueni, Tatiana Nikitina, and R. Harald Baayen. 2007. Predicting the dative alternation. Cognitive Foundations of Interpretation, pages 69–94. find
  9. Ivo Brugman, Mariët Theune, Emiel Krahmer, and Jette Viethen. 2009. Realizing the costs: template-based surface realisation in the graph approach to referring expression generation. In Proceedings of the 12th European Workshop on Natural Language Generation, ENLG ’09, pages 183–184, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  10. M. Carpenter, K. Nagell, and M. Tomasello. 1998. Social cognition, joint attention, and communicative competence from 9 to 15 months of age. Monographs of the Society for Research in Child Development, 63(4). ↗
  11. Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22(1):1–39. ↗
  12. Herbert H. Clark. 1996. Using language. Cambridge University Press, Cambridge. find
  13. Alasdair D. F. Clarke, Micha Elsner, and Hannah Rohde. 2013. Where’s Wally: The influence of visual salience on referring expression generation. Frontiers in Psychology (Perception Science), Issue on Scene Understanding: Behavioral and computational perspectives. ↗
  14. Michael Crawley. 2007. Blackwell, Hoboken, NJ. The R Book. find
  15. Wiley-Robert Dale and Nicholas J. Haddock. 1991. Generating referring expressions involving relations. In EACL, pages 161–166. find
  16. Giuseppe Di Fabbrizio, Amanda J. Stent, and Srinivas Bangalore. 2008. Referring expression generation using speaker-based attribute selection and trainable realization (ATTR). In Proceedings of the 5th International Conference on Natural Language Generation (INLG), Salt Fork, OH. find
  17. Manjuan Duan, Micha Elsner, and Marie-Catherine de Marneffe. 2013. Visual and linguistic predictors for the definiteness of referring expressions. In Proceedings of the 17th Workshop on the Semantics and Pragmatics of Dialogue (SemDial), Amsterdam. find
  18. Matt Duckham, Stephan Winter, and Michelle Robinson. 2010. Including landmarks in routing instructions. Journal of Location Based Services, 4(1):28– 52. ↗
  19. Rui Fang, Changsong Liu, Lanbo She, and Joyce Y. Chai. 2013. Towards situated dialogue: Revisiting referring expression generation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 392–402, Seattle, Washington, USA, October. Association for Computational Linguistics. ↗
  20. Katja Filippova and Michael Strube. 2007. Generating constituent order in German clauses. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 320–327, Prague, Czech Republic, June. Association for Computational Linguistics. find
  21. William Ford and David Olson. 1975. The elaboration of the noun phrase in children’s description of objects. Journal of Experimental Child Psychology, 19:371–382. ↗
  22. Albert Gatt and Anja Belz. 2010. Introducing shared task evaluation to NLG: The TUNA shared task evaluation challenges. In E. Krahmer and M. Theune, editors, Empirical Methods in Natural Language Generation. Springer, Berlin and Heidelberg. find
  23. Albert Gatt, Anja Belz, and Eric Kow. 2008. The TUNA-REG challenge 2008: Overview and evaluation results. In Proceedings of the 5th International Conference on Natural Language Generation (INLG), Salt Fork, OH. find
  24. Albert Gatt, Anja Belz, and Eric Kow. 2009. The TUNA-REG challenge 2009: Overview and evaluation results. In Proceedings of the 12th European Workshop on Natural Language Generation (ENLG), Athens. find
  25. A. Gatt, E. Krahmer, R. P. G. van Gompel, and K. van Deemter. 2012. Does domain size impact speech onset time during reference production? In Proceedings of the 34th Annual Meeting of the Cognitive Science Society, pages 1584–1589, Sapporo, Japan. find
  26. Dave Golland, Percy Liang, and Dan Klein. 2010. A game-theoretic approach to generating spatial descriptions. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 410–419, Cambridge, MA, October. Association for Computational Linguistics. find
  27. Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1995. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2):203–225.
  28. M. Handford. 1987. Where’s Wally? Walker Books, London, 3 edition. find
  29. M. Handford. 1988. Where’s Wally Now? Books, London, 4 edition. find
  30. Walker L. Itti and C. Koch. 2000. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision research, 40(10-12):1489–1506. ↗
  31. John D. Kelleher and Geert-Jan M. Kruijff. 2006. Incremental generation of spatial referring expressions in situated dialog. In ACL. find
  32. J. Kelleher, F. Costello, and J. van Genabith. 2005. Dynamically structuring, updating and interrelating representations of visual and linguistic discourse context. Artificial Intelligence, 167(12):62 – 102. Connecting Language to the World. ↗
  33. Emiel Krahmer and Kees van Deemter. 2012. Computational generation of referring expressions: A survey. Computational Linguistics, 38(1):173–218, March. ↗
  34. Claudia Maienborn. 2001. On the position and interpretation of locative modifiers. Natural Language Semantics, 9(2):191–240. ↗
  35. Crystal Nakatsu and Michael White. 2010. Generating with discourse combinatory categorial grammar. Linguistic Issues in Language Technology, 4(1). ↗
  36. T. Pechmann. 2009. Incremental speech production and referential overspecification. Linguistics, 27(1):89–110. find
  37. Ellen Prince. 1981. Toward a taxonomy of given-new information. In Peter Cole, editor, Radical Pragmatics, pages 223–255. Academic Press, New York. find
  38. Ruslan Salakhutdinov and Geoffrey Hinton. 2009. Replicated softmax: an undirected topic model. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 1607– 1614. find
  39. Richard Socher, Cliff C. Lin, Andrew Y. Ng, and Christopher D. Manning. 2011. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 26th International Conference on Machine Learning (ICML). find
  40. A. Toet. 2011. Computational versus psychophysical bottom-up image saliency: A comparative evaluation study. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 33(11):2131 –2146. ↗
  41. A. Torralba, A. Oliva, M. Castelhano, and J. M. Henderson. 2006. Contextual guidance of attention in natural scenes: The role of global features on object search. Psychological Review, 113:766–786. find
  42. Jette Viethen and Robert Dale. 2008. The use of spatial relations in referring expressions. In Proceedings of the 5th International Conference on Natural Language Generation, Salt Fork, Ohio, USA. find
  43. Gregory Ward and Betty Birner. 1995. Definiteness and the English existential. Language, 71(4):722– 742, December. ↗
  44. Gregory Ward and Betty Birner. 2001. Discourse and information structure. In Deborah Schiffrin, Deborah Tannen, and Heidi Hamilton, editors, Handbook of discourse analysis, pages 119–137. Basil Blackwell, Oxford. find
  45. Bonnie L. Webber. 2004. D-ltag: extending lexicalized tag to discourse. Cognitive Science, 28(5):751– 779. ↗
  46. Michael White and Rajakrishnan Rajkumar. 2012. Minimal dependency length in realization ranking. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 244–255, Jeju Island, Korea, July. Association for Computational Linguistics. find
  47. Jeremy M. Wolfe. 2012. Visual search. In P. Todd, T. Holls, and T. Robbins, editors, Cognitive Search: Evolution, Algorithms and the Brain, pages 159 – 175. MIT Press, Cambridge, MA, USA. find
  48. Sina Zarrieß, Aoife Cahill, and Jonas Kuhn. 2012. To what extent does sentence-internal realisation reflect discourse context? a study on word order. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 767–776, Avignon, France, April. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elsner-2014-information-structure-prediction-visual-world/