Elamur Research⌕
☰
home / papers / 2023 / Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest

Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest

Paper · 2023 · Jack Hessel, Ana Marasovic, Jena D. Hwang et al.

Summary

The paper asks whether today's AI systems grasp humor or merely produce joke-like text. To probe this, the authors build three benchmarks from 14 years of the weekly New Yorker Cartoon Caption Contest, each harder than the last. The first is matching: pick the caption actually written for a cartoon out of five candidates. The second is quality ranking: tell a finalist apart from a non-finalist entry for the same cartoon. The third is explanation: write a free-text account of why a caption fits and is funny.

Models are tested in two regimes. In from pixels, they see only the image. In from description, they receive newly collected human annotations that stand in for perfect vision. These annotations cover the scene, its locations and entities, what is odd about it, and the joke itself.

All tested systems fall well short of people:

  • On matching, the best image-based model (fine-tuned CLIP ViT-L/14) gets 62%, while humans get 94%.
  • Human explanations beat 5-shot GPT-4 explanations in over two-thirds of direct comparisons.

The released corpus, models, code and leaderboard give the field a demanding test of multimodal, culture-dependent reasoning.

Key points

  • Data sources. Roughly 250 contests (from #508 on) come from Jain et al. (2020). These include about 1.5M captions and over 114M crowd ratings collected via the NEXT platform. The earlier contests, #1–#507, were drawn from two prior works, Shahaf et al. (2015) together with Radev et al. (2016), and supply about 2M captions without any crowd ratings. The authors discarded 55 contests whose images were too low-resolution.
  • Task sizes (train/val/test).
    • Matching: 1.6K / 538 / 538.
    • Quality ranking: 1.6K / 523 / 523.
    • Explanation: 391 / 130 / 130, drawn from 651 explanations, mostly written by one author.
  • Novelty of the tasks. Matching and explanation are new. Quality ranking builds on earlier formulations and is evaluated against two audiences: New Yorker editors and the crowd raters.
  • Headline gaps. CLIP ViT-L/14 reaches 62% on matching versus 94% for humans. Few-shot GPT-4 explanations lose to human ones in more than two-thirds of head-to-head judgments.
  • Release. Everything is available at capcon.dev. The authors note the corpus may contain offensive jokes; they hand-removed a few that targeted protected groups.

Interesting ideas

  • I1. Separating perception from reasoning. The same task is run once on raw images and once on gold human descriptions of those images. This shows whether failures come from vision or from reasoning, and the trick carries over to any multimodal benchmark, such as maps, charts or document scans.
  • I2. Strong distractors from other items. Wrong answers in matching are captions that won other contests, so they are high-quality text that simply belongs elsewhere. The distractors are also balanced so that a model looking only at the answer options cannot beat chance. Both choices block shortcut solutions and are cheap ways to harden multiple-choice datasets.
  • I3. A ladder of difficulty. The tasks step up from recognizing relevance, to judging quality, to producing an explanation. Arranging an evaluation this way reveals the level at which a model's competence breaks down, rather than giving a single pass/fail score.
  • I4. Pairwise human preference for generated text. Explanations are judged by asking people which of two they prefer. This sidesteps unreliable automatic metrics for open-ended output and could be reused for summaries, translations or commentary.
  • I5. Ranking models as reward signals. The authors suggest that their relevance and quality models could provide feedback in an RLHF loop for generating captions. More generally, a discriminative benchmark model can be repurposed as a training signal for a generator.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States263 works · 180,581 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States219 works · 89,712 citations🎓Cornell UniversityIthaca · United States89 works · 58,422 citations🎓University of UtahSalt Lake City · United States29 works · 5,781 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Computer Vision and Pattern Recognition · 1,282 works Psychology · 798 works Social Psychology · 129 works

🧭 Topics 1 Evaluation & benchmarks · 685 works

Cite

Cite as Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff and Yejin Choi. 2023. Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.41

@inproceedings{hessel2023do,
  title = {{Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest}},
  author = {Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff and Yejin Choi},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.41},
  url = {https://doi.org/10.18653/v1/2023.acl-long.41}
}

References

72 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Miriam Amin and Manuel Burghardt. 2020. A survey on approaches to computational humor generation. In The 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature. find
  2. Issa Annamoradnejad and Gohar Zoghi. 2020. Col-BERT: Using BERT sentence embedding for humor detection. arXiv preprint arXiv:2004.12765. — [arXiv:2004.12765](https://arxiv.org/abs/2004.12765) find
  3. Salvatore Attardo. 2008. A primer for the linguistics of humor. The primer of humor research, 8:101–55. find
  4. Michael Billig. 2005. Laughter and ridicule: Towards a social critique of humour. Sage. find
  5. Kim Binsted and Graeme Ritchie. 1994. An implemented model of punning riddles. In AAAI. find
  6. Vladislav Blinov, Valeria Bolotova-Baranova, and Pavel Braslavski. 2019. Large dataset and language model fun-tuning for humor recognition. In ACL. find
  7. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… our card
  8. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an Obviously perfect paper). In ACL. our card
  9. Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: figurative language understanding and textual explanations. In EMNLP. find
  10. Arjun Chandrasekaran, Devi Parikh, and Mohit Bansal. 2018. Punny captions: Witty wordplay in image descriptions. In NAACL. find
  11. Arjun Chandrasekaran, Ashwin K. Vijayakumar, Stanislaw Antol, Mohit Bansal, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2016. We are humor beings: Understanding and predicting visual humor. In CVPR. find
  12. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… our card
  13. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. our card
  14. Fallianda, Rani Yuni Astiti, and Zulvy Alivia Hanim. 2018. Analyzing humor in newspaper comic strips using verbal-visual analysis. Lingua Cultura, 12(4):383–388. find
  15. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Bradford Books. find
  16. Sigmund Freud. 1905. Jokes and their Relation to the Unconscious, volume 8 of The Standard Edition of the Complete Psychological Works of Sigmund Freud. Hogarth, London. find
  17. William F. Fry. 1963. Sweet madness: A study of humor. Pacific Books, Palo Alto. find
  18. Charles R. Gruner. 1978. Understanding laughter: The workings of wit & humor. Nelson-Hall, Chicago. find
  19. Kilem Gwet. 2014. Handbook of Inter-Rater reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters, 4th edition edition. Advanced Analytics, LLC. find
  20. Md Kamrul Hasan, Sangwu Lee, Wasifur Rahman, Amir Zadeh, Rada Mihalcea, Louis-Philippe Morency, and Ehsan Hoque. 2021. Humor knowledge enriched transformer for understanding multimodal humor. In AAAI. find
  21. Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, and Mohammed (Ehsan) Hoque. 2019. UR-FUNNY: a multimodal language dataset for understanding humor. In EMNLP. find
  22. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In ICLR. find
  23. Harvey Mindess. 1971. Nash. find
  24. Pamela Mishkin, Matt Daniels, Russell Goldenberg, Ilia Blinderman, and James Yu. 2022. The pudding caption contest experiments. https://pudding.cool/ projects/caption-contest/. Accessed: 2022-04-01. ↗
  25. Matthijs P. Mulder and Antinus Nijholt. 2002. Humour research: State of the art. Centre for Telematics and Information Technology, University of Twente. find
  26. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. our card
  27. OpenAI. 2023. Gpt-4 technical report. our card
  28. Lalit Jain, Kevin Jamieson, Robert Mankoff, Robert Nowak, and Scott Sievert. 2020. The New Yorker cartoon caption contest dataset. find
  29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL. our card
  30. Kevin G. Jamieson, Lalit Jain, Chris Fernandez, Nicholas J. Glattard, and Rob Nowak. 2015. NEXT: A system for real-world development, evaluation, and application of active learning. In NeurIPS. find
  31. Ben King, Rahul Jha, Dragomir Radev, and Robert Mankoff. 2013. Random walk factoid annotation for collective discourse. In ACL. find
  32. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In NeurIPS. our card
  33. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out. our card
  34. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and… our card
  35. Badri N. Patro, Mayank Lunayach, Deepankar Srivastava, Sarvesh, Hunar Singh, and Vinay P. Namboodiri. 2021. Multimodal humor dataset: Predicting laughter tracks for sitcoms. In WACV. find
  36. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In *SEM. find
  37. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In ECCV. our card
  38. Matt Post. 2018. A call for clarity in reporting BLEU scores. In WMT. our card
  39. Tania Lombrozo. 2006. The structure and function of explanations. Trends in Cognitive Sciences, 10(10):464–470. find
  40. Ana Marasović, Iz Beltagy, Doug Downey, and Matthew E. Peters. 2022. Few-shot self-rationalization with natural language prompts. In Findings of NAACL. our card
  41. Rada Mihalcea and Stephen Pulman. 2009. Characterizing humour: An exploration of features in humorous texts. In Proceedings of the 8th International Conference on Computational Linguistics and Intelligent Text Processing, page 337–347, Berlin, Heidelberg. Springer-Verlag. find
  42. Rada Mihalcea and Carlo Strapparava. 2005. Making computers laugh: Investigations in automatic humor recognition. In EMNLP. our card
  43. Rada Mihalcea and Carlo Strapparava. 2006. Technologies that make you smile: Adding humor to text-based applications. IEEE Intelligent Systems, 21(5):33–39. find
  44. Dragomir Radev, Amanda Stent, Joel Tetreault, Aasish Pappu, Aikaterini Iliakopoulou, Agustin Chanfreau, Paloma de Juan, Jordi Vallmitjana, Alejandro Jaimes, Rahul Jha, and Robert Mankoff. 2016. Humor in collective discourse: Unsupervised funniness detection in the New Yorker cartoon caption contest.… find
  45. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In ICML. our card
  46. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR. our card
  47. Victor Raskin. 1979. Semantic mechanisms of humor. In Annual Meeting of the Berkeley Linguistics Society, volume 5, pages 325–335. find
  48. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In KDD. find
  49. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In EMNLP. our card
  50. Arthur Schopenhauer. 1818. The world as will and idea, volume 1. find
  51. Dafna Shahaf, Eric Horvitz, and Robert Mankoff. 2015. Inside jokes: Identifying humorous cartoon captions. In KDD. find
  52. Erica K Shimomoto, Lincon S Souza, Bernardo B Gatto, and Kazuhiro Fukui. 2019. News2meme: An automatic content generator from news based on word subspaces from text and image. In Conference on Machine Vision Applications. find
  53. Thomas R Shultz. 1976. A cognitive-developmental analysis of humour. Transaction Publishers. find
  54. Oliviero Stock and Carlo Strapparava. 2003. Getting serious about the development of computational humor. In IJCAI. find
  55. Rajesh Shanmuga Sundaram. 2018. Generation of Humorous Caption for Cartoon Images Using Deep Learning. Ph.D. thesis, Texas A&M University-Commerce. find
  56. Chenhao Tan. 2022. On the diversity and limits of human explanations. In NAACL. find
  57. Ervin Tanczos, Robert Nowak, and Bob Mankoff. 2017. A KL-LUCB algorithm for large-scale crowdsourcing. In NeurIPS. find
  58. Paul Trichelair, Ali Emami, Adam Trischler, Kaheer Suleman, and Jackie Chi Kit Cheung. 2019. How reasonable are common-sense reasoning tasks: A case-study on the Winograd schema challenge and SWAG. In EMNLP. find
  59. Villy Tsakona. 2009. Language and image interaction in cartoons: Towards a multimodal theory of humor. Journal of Pragmatics, 41(6):1171–1188. find
  60. Alessandro Valitutti. 2011. How many jokes are really funny? In Human-Machine Interaction in Translation: Proceedings of the 8th International NLPCS Workshop. find
  61. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NeurIPS. our card
  62. Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML. our card
  63. William Yang Wang and Miaomiao Wen. 2015. I can has cheezburger? a nonparanormal approach to combining textual and visual information for predicting and generating popular meme descriptions. In NAACL. find
  64. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS. our card
  65. White, E. B. 1941. Preface. In E. B. White and Katherine S. White, editors, A Subtreasury Of American Humor, page xvii. The original version of this quote appeared as a preview in The Saturday Review (1941), credited to both Whites. But, the quote appears in the preface to A Subtreasury (1941) with… find
  66. Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022. Reframing human-AI collaboration for generating free-text explanations. In NAACL. find
  67. Hannah Wilson. 2019. Project four - nobody knows you’re a bot. find
  68. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… our card
  69. Kota Yoshida, Munetaka Minoguchi, Kenichiro Wani, Akio Nakamura, and Hirokatsu Kataoka. 2018. Neural joking machine: Humorous image captioning. In CVPR Language & Vision Workshop. find
  70. Michael Zelenko and Frank Bi. 2015. On the internet, nobody knows you’re a machine. find
  71. Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. 2022. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint… our card
  72. David Wallace. 2022. Lecture notes for MIT 2.00b toy product design: Innovation and associations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hessel-2023-do-androids-laugh-electric-sheep/