Elamur Research⌕
☰
home / papers / 2018 / Whodunnit? Crime Drama as a Case for Natural Language Understanding

Whodunnit? Crime Drama as a Case for Natural Language Understanding

Paper · 2018 · Lea Frermann, Shay B. Cohen and Mirella Lapata

Summary

Frermann, Cohen and Lapata propose crime drama as a testbed for natural language understanding. Unlike most comprehension benchmarks, it combines several modalities, spoken dialogue and reasoning that unfolds over time. Every CSI episode asks who committed the crime and later reveals the answer, so the supervision signal comes built in.

The authors pair videos from CSI seasons 1–5 with downloaded screenplays to build a new corpus of 39 episodes. The corpus includes gold annotations of perpetrator mentions and records of when human viewers guessed the culprit while watching. They cast perpetrator identification as sequence labeling: for each incoming input, a model decides whether the perpetrator is being mentioned. Their model is a one-directional LSTM with a binary softmax output, fed with fused textual, visual and acoustic features.

Experiments show that combining modalities beats text alone, and that incremental access to earlier context is crucial. Compared with humans, the LSTM starts guessing earlier but is less consistent. The work matters because it frames understanding as an online, situated, multi-modal process rather than offline reading.

Key points

  • The dataset contains 39 CSI episodes of about 43 minutes each, released at github.com/EdinburghNLP/csi-corpus.
    • The episodes cover 59 cases: 19 episodes have one case and 20 have two.
    • Most cases are murders (51).
    • A case averages roughly 689 sentences, of which about 89 mention the perpetrator.
  • The task is binary sequence labeling of perpetrator mentions. It is designed to be independent of the number of culprits, the specific episode or the crime series.
  • The model is a unidirectional LSTM over multi-modal input (text, video, audio). Both multi-modal fusion and an incremental inference strategy proved key to accuracy.
  • Comparisons with human viewers used five cross-validation splits covering 30 test episodes.
    • The LSTM behaves like an eager viewer: on average it first identifies the culprit earlier than humans do.
    • However, it changes its mind more often than humans.
  • On an episode where the victim's death was a suicide (no perpetrator), a human viewer stopped flagging suspects after about two thirds of the episode. The LSTM kept predicting perpetrator mentions until the end.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom381 works · 80,747 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Lea Frermann, Shay B. Cohen and Mirella Lapata. 2018. Whodunnit? Crime Drama as a Case for Natural Language Understanding. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00001

@article{frermann2018whodunnit,
  title = {{Whodunnit? Crime Drama as a Case for Natural Language Understanding}},
  author = {Lea Frermann, Shay B. Cohen and Mirella Lapata},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.1162/tacl\_a\_00001},
  url = {https://doi.org/10.1162/tacl_a_00001}
}

References

45 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, M̃argaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2425– 2433, Santiago, Chile. find
  2. Piotr Bojanowski, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic. 2013. Finding actors and actions in movies. In The IEEE International Conference on Computer Vision (ICCV), pages 2280– 2287, Sydney, Australia. find
  3. John S. Boreczky and Lynn D. Wilcox. 1998. A hidden Markov model framework for video segmentation using audio and image features. In Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3741– 3744, Seattle, Washington, USA. find
  4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632– 642, Lisbon, Portugal.
  5. Elia Bruni, Nam Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49(1):1–47, January. find
  6. Sachin Chachada and C.-C. Jay Kuo. 2014. Environmental sound recognition: A survey. APSIPA Transactions on Signal and Information Processing, 3. find
  7. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  8. Timothee Cour, Chris Jordan, Eleni Miltsakaki, and Ben Taskar. 2008. Movie/script: Alignment and parsing of video and text transcription. In Proceedings of the 10th European Conference on Computer Vision, pages 158–171, Marseille, France. find
  9. Steven B. Davis and Paul Mermelstein. 1990. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. In Alex Waibel and Kai-Fu Lee, editors, Readings in Speech Recognition, pages 65–74. find
  10. Morgan Kaufmann Publishers Inc., San Francisco, California, USA. Nevenka Dimitrova, Lalitha Agnihotri, and Gang Wei. 2000. Video classification based on HMM using text and faces. In Proceedings of the 10th European Signal Processing Conference (EUSIPCO), pages 1–4. find
  11. IEEE. Desmond Elliott and Frank Keller. 2013. Image description using visual dependency representations. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1292– 1302, Seattle, Washington, USA. find
  12. Philip John Gorinski and Mirella Lapata. 2015. Movie script summarization as graph-based scene extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1066–1076, Denver, Colorado, USA.
  13. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28,…
  14. Curran Associates, Inc. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The Goldilocks principle: Reading children’s books with explicit memory representations. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, California, USA.
  15. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780, November. find
  16. Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A Shamma, Michael S Bernstein, and Li Fei-Fei. 2015. Image retrieval using scene graphs. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3668–3678, Boston, Massachusetts, USA. find
  17. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3128– 3137, Boston, Massachusetts. Douwe Kiela and Léon Bottou. 2014. Learning image embeddings using conv…
  18. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the 18th International Conference on Machine Learning, pages 282–289, San Francisco, CA, USA. find
  19. Morgan Kaufmann Publishers Inc. Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. 2015. Combining language and vision with a multimodal skip-gram model. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi…
  20. Dahua Lin, Sanja Fidler, Chen Kong, and Raquel Urtasun. 2014. Visual semantic search: Retrieving videos via complex textual queries. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2657–2664, Columbus, Ohio, USA. find
  21. Cory S. Myers and Lawrence R. Rabiner. 1981. A comparative study of several dynamic time-warping algorithms for connected word recognition. The Bell System Technical Journal, 60(7):1389–1409. find
  22. Milind R. Naphide and Thomas S. Huang. 2001. A probabilistic framework for semantic video indexing, filtering, and retrieval. IEEE Transactions on Multimedia, 3(1):141–151. find
  23. Luis Gilberto Mateos Ortiz, Clemens Wolff, and Mirella Lapata. 2015. Learning to interpret and describe abstract scenes. In Proceedings of the 2015 NAACL: Human Language Technologies, pages 1505–1515, Denver, Colorado, USA. find
  24. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar.
  25. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, USA.
  26. Zeeshan Rasheed, Yaser Sheikh, and Mubarak Shah. 2005. On the use of computable features for film classification. IEEE Transactions on Circuits and Systems for Video Technology, 15(1):52–64. find
  27. Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193–203, Seattle, Washington, USA. Tim Rocktäschel, Ed…
  28. Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. 2017. Movie description. International Journal of Computer Vision, 123(1):94–120. find
  29. Md Sahidullah and Goutam Saha. 2012. Design, analysis and experimental evaluation of block based transformation in MFCC computation for speaker recognition. Speech Communication, 54(4):543–565. find
  30. Jitao Sang and Changsheng Xu. 2010. Character-based movie summarization. In Proceedings of the 18th ACM International Conference on Multimedia, pages 855–858, Firenze, Italy. find
  31. Carina Silberer, Vittorio Ferrari, and Mirella Lapata. 2016. Visually grounded meaning representations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 99. find
  32. Josef Sivic, Mark Everingham, and Andrew Zisserman. 2009. “Who are you?” – Learning person specific classifiers from video. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1145– 1152, Miami, Florida, USA. find
  33. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS’14, pages 3104–3112, Cambridge, MA, USA.
  34. MIT Press. Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke. 2016. Inception-v4, Inception-ResNet and the impact of residual connections on learning. CoRR, abs/1602.07261. Makarand Tapaswi, Martin Bäuml, and Rainer Stiefelhagen. 2015. Aligning plot synopses to videos for story-based retrieval… — [arXiv:1602.07261](https://arxiv.org/abs/1602.07261) find
  35. Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. MovieQA: Understanding stories in movies through question-answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4631–4640, Las Vegas, Nevada. find
  36. Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond J. Mooney, Trevor Darrell, and Kate Saenko. 2015a. Sequence to sequence – Video to text. In Proceedings of the 2015 International Conference on Computer Vision (ICCV), pages 4534–4542, Santiago, Chile. find
  37. Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko. 2015b. Translating videos to natural language using deep recurrent neural networks. In Proceedings the 2015 Conference of the North American Chapter of the Association for Computational Linguistics –… find
  38. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3156–3164.
  39. Ellen M. Voorhees and Dawn M. Tice. 2000. Building a question answering test collection. In ACM Special Interest Group on Information Retrieval (SIGIR), pages 200–207, Athens, Greece. find
  40. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015. Towards AI-complete question answering: A set of prerequisite toy tasks. CoRR, abs/1502.05698.
  41. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on Machine Learning, pages 2048–2057, Boston, M…
  42. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018, Lisbon, Portugal.
  43. Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual semantic role labeling for image understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5534–5542, Zurich, Switzerland. find
  44. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. CoRR, abs/1409.2329.
  45. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/frermann-2018-whodunnit-crime-drama-as-case/