Elamur Research⌕
☰
home / papers / 2019 / Humor Detection: A Transformer Gets the Last Laugh

Humor Detection: A Transformer Gets the Last Laugh

Paper · 2019 · Orion Weller and Kevin Seppi

Summary

The paper moves past simply spotting jokes in text and asks a harder question: can a model predict whether a joke will actually be judged funny? The authors build a rated-joke dataset by repeatedly scraping Reddit's r/Jokes forum and updating upvote counts. They then call a joke "funny" if it passes a cutoff that falls at a natural gap in the score distribution. They fine-tune pre-trained BERT on this data, using the joke body only, the punchline only, or both. They compare it with a CNN with Highway Layers and with crowd-sourced human judgments. On full jokes the Transformer is the most accurate of the three. Humans recruited on Mechanical Turk agree with the Reddit labels less often than the model does. This suggests the model learns the taste of one particular audience. The same setup also beats earlier results on two established joke-identification benchmarks, Short Jokes and Pun of the Day. The work matters because it frames humor as audience-specific and shows that a single Transformer setup is strong across several humor tasks.

Key points

  • Reddit dataset:
    • Collected hourly through March–April 2019, with upvote scores refreshed on every run.
    • Scores range from 0 to 136,354.
    • A 200-upvote threshold yields 13,884 not-funny and 2,025 funny jokes.
    • Three variants are provided: body, punchline, and full.
  • Model and training:
    • BERT fine-tuned with learning rate 2e-05, max sequence length 128, and up to 7 epochs.
    • Stratified 75/25 split.
    • The minority class is upsampled to 50% in training, and the evaluation data is downsampled to a 50/50 split.
  • Reddit accuracy (full jokes): Transformer 0.724, CNN 0.688, MTurk humans 0.663. The human baseline used 199 participants, each rating about 30 jokes, combined by majority vote. Every system, humans included, scored higher on punchlines than on bodies.
  • Short Jokes:
    • Built from 231,657 Kaggle jokes plus non-joke sentences from the WMT16 news crawl, matched in word and character distribution.
    • The Transformer reaches 0.986 on accuracy, precision, recall and F1, versus 0.906 accuracy for CNN+F+HN.
  • Pun of the Day:
    • Contains 16,001 puns and 16,002 non-pun sentences.
    • The Transformer scores 0.930 accuracy and 0.931 F1, versus 0.894 and 0.901 for the best earlier CNN.
  • Availability: code and datasets are released publicly.

Interesting ideas

  • I1. Labels from community engagement: upvote counts work as large-scale, cheap supervision for a subjective quality. Choosing the cutoff at a visible break in the score distribution avoids an arbitrary threshold. The same approach could suit other popularity-driven labels.
  • I2. Re-scraping to let scores settle: the scraper revisits earlier posts and updates their counts. This reduces label noise from fresh posts that have not yet gathered votes, which is useful for any dataset built on social signals.
  • I3. Ablating the input by part: training separately on body, punchline and the full text reveals which segment drives the prediction. Similar part-wise ablations could probe structured texts such as headline vs. article or question vs. answer.
  • I4. Humans as an "outside audience" baseline: asking general crowd workers to rate the same items tests whether labels reflect a niche community's taste rather than a universal judgment. This is a useful sanity check for any subjective-label task.
  • I5. Matched negatives and balanced evaluation: - Non-joke sentences are chosen to match the jokes' length distribution, so the classifier cannot win on surface statistics. - The test set is downsampled to 50/50 so that accuracy is easy to interpret. - Both tricks carry over to building fair binary-classification benchmarks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Brigham Young UniversityProvo · United States15 works · 40,065 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Social Psychology · 129 works

Cite

Cite as Orion Weller and Kevin Seppi. 2019. Humor Detection: A Transformer Gets the Last Laugh. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1372

@inproceedings{weller2019humor,
  title = {{Humor Detection: A Transformer Gets the Last Laugh}},
  author = {Orion Weller and Kevin Seppi},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1372},
  url = {https://doi.org/10.18653/v1/d19-1372}
}

References

18 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway Networks. arXiv eprints, page arXiv:1505.00387. our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. International Conference on Learning Representations. our card
  3. Hareesh Bahuleyan. 2018. Natural Language Generation with Neural Variational Models. arXiv e-prints, page arXiv:1808.09012. — [arXiv:1808.09012](https://arxiv.org/abs/1808.09012) find
  4. Peng-Yu Chen and Von-Wun Soo. 2018. Humor recognition using deep learning. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). find
  5. Yishay Raz. 2012. Automatic humor classification on twitter. Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pages 66–70. find
  6. Julia M. Taylor and Lawrence J. Mazlack. 2004. Computationally recognizing wordplay in jokes. In Proceedings of CogSci 2004. find
  7. Alessandro Valitutti, Hannu Toivonen, Antoine Doucet, and Jukka M Toivanen. 2013. let everything turn well in your wife: Generation of adult humor using lexical constraints. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2,… find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. North American Chapter of the Association for Computational Linguistics. our card
  9. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. 31st Conference on Neural Information Processing Systems, pages 17–21. our card
  10. He He, Nanyun Peng, and Percy Liang. 2019. Pun generation with surprise. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1734–1744, Minneapolis, Minnesota. Assoc… find
  11. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. arXiv e-prints, page arXiv:1502.03044. our card
  12. Yann LeCun, Leon Bottou, Y Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86:2278 – 2324. find
  13. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating Wikipedia by Summarizing Long Sequences. International Conference on Learning Representations. our card
  14. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages pages 1412 – 1421. our card
  15. Diyi Yang, Alon Lavie, Chris Dyer, and Eduard Hovy. 2015. Humor recognition and humor anchor extraction. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2367–2376. our card
  16. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016. Hierarchical attention networks for document classification. HLT-NAACL. our card
  17. Renxian Zhang and Naishi Liu. 2014. Recognizing humor on twitter. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM ’14, pages 889–898, New York, NY, USA. ACM. find
  18. Rada Mihalcea and Carlo Strapparava. 2005. Making computers laugh: Investigations in automatic humor recognition. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT ’05, pages 531–538, Stroudsburg, PA, USA. Association for Computat… our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/weller-2019-humor-detection-transformer-gets-last/