Elamur Research⌕
☰
home / papers / 2013 / Grounding Action Descriptions in Videos

Grounding Action Descriptions in Videos

Paper · 2013 · Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel et al.

Summary

Text-only distributional models describe word meaning incompletely, and earlier multimodal work grounded semantics only in still images. This paper grounds descriptions of actions in video instead. The authors build TACOS (Saarbrücken Corpus of Textually Annotated Cooking Scenes) on top of the MPII Composites cooking videos. Mechanical Turk workers wrote several descriptions for each video, and every sentence was linked to a video segment and to the existing low-level activity annotations. The authors also create ASim, a gold standard of paired action descriptions with their video segments, rated for similarity in the style of the Usage Similarity dataset of Erk et al. (2009). They then test existing textual and visual similarity measures against ASim, both alone and combined by averaging z-scores. Video-based measures correlate significantly with human judgments, and mixing the two modalities works best, coming close to inter-annotator agreement. Visual features help mostly when pairs share an object but describe different actions, which suggests they capture action similarity rather than object similarity. All data are publicly released.

Key points

  • MPII Composites provides 212 high-resolution videos (4.5 min on average) of 41 cooking tasks performed by 22 subjects. It includes 8818 segments annotated with 60 low-level activity labels (e.g. PEEL, STIR).
  • For TACOS, 127 videos covering 26 ingredient-manipulation tasks each received 20 MTurk descriptions, giving 2540 assignments. Each description contained 5–15 sentences and was collected with an adapted Vatic tool.
  • On the full ASim set, the ALL COMBINED model (textual and visual measures together) reached the best correlation with human ratings, 0.55.
  • Visual models beat textual ones on the SAME OBJECT subset but added nothing on the SAME VERB subset. Raw visual features alone account for most of the gain, without needing the supervised classifier.
  • The released resources are the videos, low-level annotations, aligned descriptions, the ASim dataset, and the visual features. Apart from the visual classifier, all models are unsupervised and could transfer to other domains with observable actions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany102 works · 17,447 citations🔬Max Planck Institute for InformaticsSaarbrücken · Germany11 works · 4,289 citations🔬Max Planck SocietyMunich · Germany10 works · 42,142 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele and Manfred Pinkal. 2013. Grounding Action Descriptions in Videos. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00207

@article{regneri2013grounding,
  title = {{Grounding Action Descriptions in Videos}},
  author = {Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele and Manfred Pinkal},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2013},
  doi = {10.1162/tacl\_a\_00207},
  url = {https://doi.org/10.1162/tacl_a_00207}
}

References

25 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of SIGCHI 2004. find
  2. Elia Bruni, Giang Binh Tran, and Marco Baroni. 2011. Distributional semantics from text and images. In Proceedings of GEMS 2011. find
  3. David L. Chen and William B. Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of ACL 2011.
  4. Timothee Cour, Chris Jordan, Eleni Miltsakaki, and Ben Taskar. 2008. Movie/script: Alignment and parsing of video and text transcription. In Computer Vision – ECCV 2008, volume 5305 of Lecture Notes in Computer Science, pages 158–171. find
  5. Springer Berlin Heidelberg. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Proceedings of MLCW 2005. Jesse Dodge, Amit Goyal, Xufeng Han, Alyssa Mensch, Margaret Mitchell, Karl Stratos, Kota Yamaguchi, Yejin Choi, Hal Daumé III, Alexand… find
  6. Katrin Erk and Diana McCarthy. 2009. Graded word sense assignment. In Proceedings of EMNLP 2009. find
  7. Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2009. Investigations on word senses and word usages. In Proceedings of ACL/AFNLP 2009. find
  8. Katrin Erk, Diana McCarthy, and Nick Gaylord. 2012. Measuring word meaning in context.
  9. CL. Yansong Feng and Mirella Lapata. 2010. Visual information in semantic representation. In Proceedings of HLT-NAACL 2010. find
  10. A. M. Glenberg. 2002. Grounding language in action. find
  11. Psychonomic Bulletin & Review. Sonal Gupta and Raymond J. Mooney. 2010. Using closed captions as supervision for video activity recognition. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI-2010), pages 1083–1088, Atlanta, GA, July. find
  12. Abhinav Gupta, Praveen Srinivasan, Jianbo Shi, and Larry S. Davis. 2009. Understanding videos, constructing plots learning a visually grounded storyline model from annotated videos. In Proceedings of CVPR 2009. find
  13. Steve R. Howell, Damian Jankowicz, and Suzanna Becker. 2005. A model of grounded language acquisition: Sensorimotor features improve lexical and grammatical learning. find
  14. JML. S. Mathe, A. Fazly, S. Dickinson, and S. Stevenson. 2008. Learning the abstract motion semantics of verbs from captioned videos. pages 1–8. find
  15. Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In Proceedings of ACL 2008. find
  16. Tanvi S. Motwani and Raymond J. Mooney. 2012. Improving video activity recognition using object recognition and text mining. In Proceedings of the 20th European Conference on Artificial Intelligence (ECAI-2012), pages 600–605, August. find
  17. Jeff Orkin and Deb Roy. 2009. Automatic learning and generation of social behavior from collective human gameplay. In Proceedings of AAMAS 2009. find
  18. Hilke Reckman, Jeff Orkin, and Deb Roy. 2011. Extracting aspects of determiner meaning from dialogue in a virtual world environment. In Proceedings of CCS 2011, IWCS ’11. find
  19. Marcus Rohrbach, Sikandar Amin, Mykhaylo Andriluka, and Bernt Schiele. 2012a. A database for fine grained activity detection of cooking activities. In Proceedings of CVPR 2012. find
  20. Marcus Rohrbach, Michaela Regneri, Micha Andriluka, Sikandar Amin, Manfred Pinkal, and Bernt Schiele. 2012b. Script data for attribute-based recognition of composite activities. In Proceedings of ECCV 2012. find
  21. Carina Silberer and Mirella Lapata. 2012. Grounded models of semantic representation. In Proceedings of EMNLP-CoNLL 2012. find
  22. Mark Steyvers. 2010. Combining feature norms and text data with topic models. Acta Psychologica, 133(3):234 – 243. ¡ce:title¿Formal modeling of semantic concepts¡/ce:title¿. Stefan Thater, Hagen Fürstenau, and Manfred Pinkal. 2011. Word meaning in context: A simple and effective vector model. In Pr… find
  23. Peter D. Turney and Patrick Pantel. 2010. From frequency to meaning. vector space models for semantics. find
  24. JAIR. E. Tzoukermann, J. Neumann, J. Kosecka, C. Fermuller, I. Perera, F. Ferraro, B. Sapp, R. Chaudhry, and G. Singh. 2011. Language models for semantic extraction and filtering in video action recognition. In AAAI Workshop on Language-Action Tools for Cognitive Artificial Agents. find
  25. Carl Vondrick, Donald Patterson, and Deva Ramanan. 2012. Efficiently scaling up crowdsourced video annotation. IJCV. Heng Wang, Alexander Kläser, Cordelia Schmid, and Cheng-Lin Liu. 2011. Action Recognition by Dense Trajectories. In Proceedings of CVPR 2011. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/regneri-2013-grounding-action-descriptions-videos/