Elamur Research⌕
☰
home / papers / 2019 / What You See is What You Get: Visual Pronoun Coreference Resolution in Dialogues

What You See is What You Get: Visual Pronoun Coreference Resolution in Dialogues

Summary

In conversations about a scene that every participant can see, speakers often use pronouns for objects that are never named in the dialogue. Coreference systems that rely only on the text cannot resolve such pronouns. The authors define this setting as visual pronoun coreference resolution (Visual PCR) and build VisPro, a dataset based on image-grounded dialogues from VisDial v1.0. They used the Stanford Parser to extract candidate noun phrases from the dialogues and image captions. MTurk workers then marked the antecedents of third-person personal and possessive pronouns, and the final coreference clusters were settled by majority vote. The authors also propose VisCoref, a model that matches textual mentions to objects detected in the image and combines this visual signal with contextual span representations to choose antecedents. According to the authors' experiments and case studies, VisCoref works better than the text-only End-to-end baseline. For example, it links "they" to "2 zebras" rather than "the people" when the image contains no humans. The work shows that the shared visual context is important for understanding dialogue.

Key points

  • VisPro contains 29,722 pronouns from 5,000 dialogues taken from VisDial v1.0. Each selected dialogue has 4–10 pronouns (5.94 on average), and the data is split 4,000/500/500 into train/val/test.
  • At least four of the 186 qualified MTurk workers annotated each dialogue, producing 122,443 annotations at a cost of USD 3,270.80. After workers who failed the checkpoint questions were filtered out, 116,300 annotations (95%) remained.
  • Of all pronouns, 73.43% are anaphoric to noun phrases, 5.67% have no suitable antecedent, and 20.90% are non-referential. Among the anaphoric ones, 13.45% have no antecedent in the dialogue itself, only in the caption.
  • Inter-annotator agreement, measured as the average MUC score against the adjudicated result, is 72.4.
  • The dataset, code, and models are released at github.com/HKUST-KnowComp/Visual_PCR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1516 · arXiv 1909.00421

@inproceedings{yu2019what,
  title = {{What You See is What You Get: Visual Pronoun Coreference Resolution in Dialogues}},
  author = {Yu, Xintong and Zhang, Hongming and Song, Yangqiu and Song, Yan and Zhang, Changshui},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1516},
  eprint = {1909.00421},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1516}
}

References

29 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question answering. In Proceedings of ICCV, pages 2425– 2433. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR. find our card
  3. Hong Chen, Zhenhua Fan, Hao Lu, Alan L. Yuille, and Shu Rong. 2018. Preco: A large-scale dataset in preschool vocabulary for coreference resolution. In Proceedings of EMNLP, pages 172–181. find our card
  4. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of ACL, pages 1405–1415. find our card
  5. Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of EMNLP, pages 2256–2262. find our card
  6. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of CVPR, pages 1080–1089. find
  7. Richard J. Edens, Helen L. Gaylard, Gareth J. F. Jones, and Adenike M. Lam-Adesina. 2003. An investigation of broad coverage automatic pronoun resolution for information retrieval. In Proceedings of SIGIR, pages 381–382. find
  8. Kate Ehrlich. 1981. Search and inference strategies in pronoun resolution: an experimental study. In Proceedings of ACL, pages 89–93. find
  9. Liane Guillou. 2012. Improving pronoun translation for statistical machine translation. In Proceedings of EACL, pages 1–10. find
  10. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR, pages 770–778. find
  11. Jerry R Hobbs. 1978. Resolving pronoun references. Lingua, 44(4):311–338. find our card
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  13. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR. find
  14. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of ACL, pages 423–430. find our card
  15. Satwik Kottur, José M. F. Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2018. Visual coreference resolution in visual dialog using neural module networks. In Proceedings of ECCV, pages 160–178. find
  16. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of NAACL-HLT, pages 687–692. find our card
  17. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In Proceedings of CVPR, pages 11–20. find our card
  18. Vincent Ng. 2005. Supervised ranking for pronoun resolution: Some recent improvements. In Proceedings of AAAI, pages 1081–1086. find
  19. US NIST. 2003. The ace 2003 evaluation plan. US National Institute for Standards and Technology (NIST), pages 2003–08. find
  20. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532–1543. find
  21. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237. find
  22. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Proceedings of EMNLP-CoNLL, pages 1–40. find our card
  23. Karthik Raghunathan, Heeyoung Lee, Sudarshan Rangarajan, Nate Chambers, Mihai Surdeanu, Dan Jurafsky, and Christopher D. Manning. 2010. A multi-pass sieve for coreference resolution. In Proceedings of EMNLP, pages 492–501. find our card
  24. Josef Steinberger, Massimo Poesio, Mijail A. Kabadjov, and Karel Jezek. 2007. Two uses of anaphora resolution in summarization. Inf. Process. Manage., 43(6):1663–1680. find
  25. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL, pages 656–664. find our card
  26. Michael Strube and Christoph Müller. 2003. A machine learning approach to pronoun resolution in spoken dialogue. In Proceedings of ACL, pages 168–175. find our card
  27. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of ICML, pages 2048–2057. find our card
  28. Hongming Zhang, Yan Song, and Yangqiu Song. 2019a. Incorporating context and external knowledge for pronoun coreference resolution. In Proceedings of NAACL-HLT, pages 872–881. find our card
  29. Hongming Zhang, Yan Song, Yangqiu Song, and Dong Yu. 2019b. Knowledge-aware pronoun coreference resolution. In Proceedings of ACL, pages 867–876. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2019-what-you-see-is-what/