Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

814 papers of our library cite it (of the 140673 whose bibliographies we read); 68 of them have a page here, 746 not yet.

• Towards a Cleaner Document-Oriented Multilingual Crawled Corpus
• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
• The Falcon Series of Open Language Models
• AraBERT: Transformer-based Model for Arabic Language Understanding
• MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950)
• Layer Normalization
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• Recognition of They/Them as Singular Personal Pronouns in Coreference Resolution
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• What you can cram into a single \$&!\#* vector: Probing sentence embeddings for linguistic properties
• Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning
• 8-bit Optimizers via Block-wise Quantization
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• English Machine Reading Comprehension Datasets: A Survey
• From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models
• Penn-Helsinki Parsed Corpus of Early Modern English: First Parsing Results and Analysis
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
• Learning to Write with Cooperative Discriminators
• Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition
• Challenges and Applications of Large Language Models
• GeDi: Generative Discriminator Guided Sequence Generation
• MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
• Parsing Early Modern English for Linguistic Search
• HeRo: RoBERTa and Longformer Hebrew Language Models
• Contrastive Decoding: Open-ended Text Generation as Optimization
• Few-shot Learning with Multilingual Language Models
• Linguistic Knowledge and Transferability of Contextual Representations
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• LLM-Pruner: On the Structural Pruning of Large Language Models
• EuroLLM: Multilingual Language Models for Europe
• StereoSet: Measuring stereotypical bias in pretrained language models
• CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
• CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages
• Representation Learning with Contrastive Predictive Coding
• The LAMBADA dataset: Word prediction requiring a broad discourse context
• ToTTo: A Controlled Table-To-Text Generation Dataset
• Language Models as Knowledge Bases?
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• A Transformer-based approach to Irony and Sarcasm detection
• Compressive Transformers for Long-Range Sequence Modelling
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• SocialIQA: Commonsense Reasoning about Social Interactions
• Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
• What makes a good conversation? How controllable attributes affect human judgments
• Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models
• RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation
• Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
• MPNet: Masked and Permuted Pre-training for Language Understanding
• Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• What do you learn from context? Probing for sentence structure in contextualized word representations
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue
• CLUE: A Chinese Language Understanding Evaluation Benchmark
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Large Language Model Unlearning
• LinkBERT: Pretraining Language Models with Document Links
• ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
• Big Bird: Transformers for Longer Sequences
• A Survey of Large Language Models

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1506.06724 · arXiv 1506.06724

@misc{zhu2015aligning,
  title = {{Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books}},
  author = {Zhu, Yukun and Kiros, Ryan and Zemel, Richard and Salakhutdinov, Ruslan and Urtasun, Raquel and Torralba, Antonio and Fidler, Sanja},
  year = {2015},
  doi = {10.48550/arxiv.1506.06724},
  eprint = {1506.06724},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1506.06724}
}

References

38 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015. 4 find our card
2. K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. EMNLP, 2014. 4 find
3. J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. 4 find
4. T. Cour, C. Jordan, E. Miltsakaki, and B. Taskar. Movie/script: Alignment and parsing of video and text transcription. In ECCV, 2008. 2 find
5. M. Everingham, J. Sivic, and A. Zisserman. “Hello! My name is... Buffy” – Automatic Naming of Characters in TV Video. BMVC, pages 899–908, 2006. 2 find
6. A. Farhadi, M. Hejrati, M. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences for images. In ECCV, 2010. 2 find
7. S. Fidler, A. Sharma, and R. Urtasun. A sentence is worth a thousand pixels. In CVPR, 2013. 2 find
8. A. Gupta and L. Davis. Beyond nouns: Exploiting prepositions and comparative adjectives for learning visual classifiers. In ECCV, 2008. 1 find
9. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 4 find
10. N. Kalchbrenner and P. Blunsom. Recurrent continuous translation models. In EMNLP, pages 1700–1709, 2013. 4 find
11. A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015. 1, 2 find our card
12. D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 5 find
13. R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. CoRR, abs/1411.2539, 2014. 1, 2, 3, 5, 9, 10 find our card
14. R. Kiros, Y. Zhu, R. Salakhutdinov, R. S. Zemel, A. Torralba, R. Urtasun, and S. Fidler. Skip-Thought Vectors. In Arxiv, 2015. 3, 4 find
15. C. Kong, D. Lin, M. Bansal, R. Urtasun, and S. Fidler. What are you talking about? text-to-image coreference. In CVPR, 2014. 1, 2 find
16. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. Berg, and T. Berg. Baby talk: Understanding and generating simple image descriptions. In CVPR, 2011. 2 find
17. D. Lin, S. Fidler, C. Kong, and R. Urtasun. Visual Semantic Search: Retrieving Videos via Complex Textual Queries. CVPR, pages 2657–2664, 2014. 1, 2 find
18. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. 2014. 1, 19 find our card
19. X. Lin and D. Parikh. Don’t just listen, use your imagination: Leveraging visual common sense for non-visual tasks. In CVPR, 2015. 1 find
20. M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, 2014. 1 find
21. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Explain images with multimodal recurrent neural networks. In arXiv:1410.1090, 2014. 1, 2 — [arXiv:1410.1090](https://arxiv.org/abs/1410.1090) find
22. T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. 4 find
23. K. Papineni, S. Roukos, T. Ward, and W. J. Zhu. BLEU: a method for automatic evaluation of machine translation. In ACL, pages 311–318, 2002. 6 find our card
24. H. Pirsiavash, C. Vondrick, and A. Torralba. Inferring the why in images. arXiv.org, jun 2014. 2 find
25. V. Ramanathan, A. Joulin, P. Liang, and L. Fei-Fei. Linking People in Videos with “Their” Names Using Coreference Resolution. In ECCV, pages 95–110. 2014. 2 find
26. V. Ramanathan, P. Liang, and L. Fei-Fei. Video event understanding using natural language descriptions. In ICCV, 2013. 1 find
27. A. Rohrbach, M. Rohrbach, N. Tandon, and B. Schiele. A dataset for movie description. In CVPR, 2015. 2, 5 find
28. P. Sankar, C. V. Jawahar, and A. Zisserman. Subtitle-free Movie to Script Alignment. In BMVC, 2009. 2 find
29. A. Schwing, T. Hazan, M. Pollefeys, and R. Urtasun. Efficient Structured Prediction with Latent Variables for General Graphical Models. In ICML, 2012. 6 find
30. J. Sivic, M. Everingham, and A. Zisserman. “Who are you?” - Learning person specific classifiers from video. CVPR, pages 1145–1152, 2009. 2 find
31. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In NIPS, 2014. 4 find
32. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. 5 — [arXiv:1409.4842](https://arxiv.org/abs/1409.4842) find
33. M. Tapaswi, M. Bauml, and R. Stiefelhagen. Book2Movie: Aligning Video scenes with Book chapters. In CVPR, 2015. 2 find
34. M. Tapaswi, M. Buml, and R. Stiefelhagen. Aligning Plot Synopses to Videos for Story-based Retrieval. IJMIR, 4:3– 16, 2015. 1, 2, 6 find
35. S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. J. Mooney, and K. Saenko. Translating Videos to Natural Language Using Deep Recurrent Neural Networks. CoRR abs/1312.6229, cs.CV, 2014. 1, 2 find
36. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In arXiv:1411.4555, 2014. 1, 2 find our card
37. K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In arXiv:1502.03044, 2015. 2 find our card
38. B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning Deep Features for Scene Recognition using Places Database. In NIPS, 2014. 5, 7 find