Elamur Research⌕
☰
home / papers / 2019 / CITE: A Corpus of Image-Text Discourse Relations

CITE: A Corpus of Image-Text Discourse Relations

Summary

Discourse corpora such as the Penn Discourse Treebank describe how text segments cohere. Before this paper, no comparable resource covered how photographs and accompanying text combine into a coherent presentation. Text-oriented annotation schemes depend on linguistic signals such as connectives, which images lack. The authors therefore skip a predefined relation taxonomy. Instead, crowd workers answer targeted questions about specific inferences: temporal (preparation, execution, result), logical (one of several actions, repeated action), and what the image depicts, omits or adds relative to the text.

Using recipes from RecipeQA, they built CITE, a crowdsourced corpus of image–text pairs from instructional documents, and obtained high inter-annotator agreement. The analysis shows that images often serve as background rather than essential instruction. A single photo can also support several overlapping relations at once, which is more complex than typical text-to-text coherence. Bag-of-words classifiers can predict many of these inferences from the text alone.

The work shows that cross-modal discourse annotation is feasible with non-experts. This matters for systems that understand or generate multimodal documents.

Key points

  • Data: from 1,690 single-image-per-step RecipeQA training recipes, the authors kept those with at most 70 words per step. The result is 516 documents with 2,047 image–text pairs, annotated via Amazon Mechanical Turk using 10 questions (text highlighting plus True/False items).
  • Agreement: Cohen's κ = 0.844 on 150 pairs with two annotators each, and average Fleiss's κ = 0.736 on 50 pairs with five annotators each.
  • Findings:
    • About 17% of images gave no information about their step.
    • Images were judged necessary for carrying out the step in only about 6% of cases.
    • 12% of temporally related images showed more than one temporal relation.
  • Classification: SVMs with bag-of-words features reached F1 between 0.64 and 0.92 across questions. Naive Bayes and NBSVM features show that action verbs align with depicted content, while numbers and units signal text-only information.
  • The CITE dataset and code are released at github.com/malihealikhani/CITE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1056 · arXiv 1904.06286

@inproceedings{alikhani2019cite,
  title = {{CITE: A Corpus of Image-Text Discourse Relations}},
  author = {Alikhani, Malihe and Chowdhury, Sreyasi Nag and Melo, Gerard de and Stone, Matthew},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1056},
  eprint = {1904.06286},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1056}
}

References

24 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Hunter, N. Asher, and A. Lascarides. 2015. Integrating non-linguistic events into discourse structure. In Proceedings of the 11th International Conference on Computational Semantics (IWCS), pages 184–194, London. find
  2. Alex Lascarides and Matthew Stone. 2009. Discourse coherence and gesture interpretation. Gesture, 9(2):147–180. find
  3. Dorit Abusch. 2013. Applying discourse semantics and pragmatics to co-reference in picture sequences. In Proceedings of Sinn und Bedeutung 17, pages 9– 25, Paris. find
  4. M. Alikhani and M. Stone. 2018a. Exploring coherence in visual explanations. In 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pages 272–277. find
  5. Scott McCloud. 1993. Understanding comics: The invisible art. William Morrow. find
  6. Malihe Alikhani and Matthew Stone. 2018b. Arrows are the verbs of diagrams. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3552–3563. find
  7. Eleni Miltsakaki, Rashmi Prasad, Aravind K. Joshi, and Bonnie L. Webber. 2004. The Penn Discourse Treebank. In LREC. European Language Resources Association. find our card
  8. Malihe Alikhani and Matthew Stone. 2019. “caption” as a coherence relation: Evidence and implications. In Second Workshop on Shortcomings in Vision and Language (SiVL). find our card
  9. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind K. Joshi, and Bonnie L. Webber. 2008. The Penn Discourse TreeBank 2.0. In LREC. European Language Resources Association. find our card
  10. Nicholas Asher and Alex Lascarides. 2003. Logics of conversation. Cambridge University Press. find
  11. Mousumi Banerjee, Michelle Capozzoli, Laura Mc-Sweeney, and Debajyoti Sinha. 1999. Beyond kappa: A review of interrater agreement measures. Canadian journal of statistics, 27(1):3–23. find
  12. Philippe Schlenker and Emmanuel Chemla. 2017. Gestural agreement. Natural Language & Linguistic Theory, pages 1–39. find
  13. Anne Cocos, Aaron Masino, Ting Qian, Ellie Pavlick, and Chris Callison-Burch. 2015. Effectively crowdsourcing radiology report annotations. In Proceedings of the Sixth International Workshop on Health Text Mining and Information Analysis, pages 109– 114. find
  14. Matthew Stone and Una Stojnic. 2015. Meaning and demonstration. Review of Philosophy and Psychology, 6(1):69–97. find
  15. Anthony J Viera, Joanne M Garrett, et al. 2005. Understanding interobserver agreement: the kappa statistic. Fam Med, 37(5):360–363. find
  16. Neil Cohn. 2013. Visual narrative structure. Cognitive science, 37(3):413–452. find
  17. Heriberto Cuayáhuitl, Simon Keizer, and Oliver Lemon. 2015. Strategic dialogue management via deep reinforcement learning. In NIPS Workshop on Deep Reinforcement Learning. ArXiv:1511.08099. — [arXiv:1511.08099](https://arxiv.org/abs/1511.08099) find
  18. Sida Wang and Christopher D Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2, pages 90–94. Association for Computational Linguistics. find
  19. Samuel Cumming, Gabriel Greenberg, and Rory Kelly. 2017. Conventions of viewpoint coherence in film. Philosophers’ Imprint, 17(1):1–29. find
  20. Bonnie Webber. 2009. Genre distinctions for discourse in the penn treebank. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, pages 674–682. Association for Computati… find
  21. Steven K Feiner and Kathleen R McKeown. 1991. Automating the generation of coordinated multimedia explanations. Computer, 24(10):33–41. find
  22. Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613– 619. find
  23. Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. 2018. RecipeQA: A challenge dataset for multimodal comprehension of cooking recipes. In EMNLP, pages 1358–1368. Association for Computational Linguistics. find
  24. Tuomo Hiippala and Serafina Orekhova. 2018. Enhancing the ai2 diagrams dataset using rhetorical structure theory. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/alikhani-2019-cite-corpus-image-text-discourse/