Elamur Research⌕
☰
home / papers / 2021 / SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense

SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense

Paper · 2021 · J. A. Meaney, Steven Wilson, Luis Chiruzzo et al.

Summary

HaHackathon is the SemEval 2021 shared task that, for the first time, brought humor detection and offense detection together in one benchmark. The organizers assembled 10,000 English texts. Most came from US Twitter accounts and the rest from the Kaggle Short Jokes collection. Each text was labeled by 20 US-based native speakers, with five annotators from each of four age brackets. There were four subtasks: deciding whether a text is humorous, regressing its mean humor score, regressing its mean offense score, and a new controversy subtask that asks whether annotators disagreed strongly on how funny a joke is. Between 36 and 58 teams entered each subtask. Most relied on pre-trained transformers, often combined with extra techniques such as task-adaptive training, adversarial training or multi-task learning. Binary humor detection turned out to be largely tractable. Controversy prediction stayed hard, and a simple rule-based system placed in the top three. The dataset matters because it records funniness and offensiveness as subjective, demographically varied judgments rather than single fixed labels.

Key points

  • Data sources: 80% of the corpus is tweets and 20% comes from the Kaggle Short Jokes dataset (/r/jokes, /r/cleanjokes). The 8,000 tweets were filtered down from 76,542 crawled texts.
  • Annotation: Annotators were recruited on Prolific and split across four age groups (18–25, 26–40, 41–55, 56–70), giving 20 ratings per text. After 301 sessions were rejected, 2,062 annotation sessions remained.
  • Controversy label: A humorous text counts as controversial when the variance of its humor ratings is above the median variance in the training set.
  • Rule-based ceiling for controversy: If humor scores were known perfectly, labeling every text with a score above 2 as controversial would reach 0.580 accuracy. A cut-off of 1.45 would reach an F1 of 0.635. Both results would top the leaderboard.
  • Offense errors by source: Kaggle texts account for 44.8% of the hardest-to-predict offense items, although they are only 20% of the data. Many of these are ironic jokes that systems wrongly rated as highly offensive.

Interesting ideas

  • I1. Keyword decoupling: The organizers took 1,000 jokes in which a hate-speech keyword is the butt of the joke and 1,000 in which the keyword only appears in passing. This stops models from learning a shortcut based on keywords alone. The same contrastive sampling could help any toxicity or sentiment dataset where trigger words are confounded with labels.
  • I2. Disagreement as a target: The task turns rater variance into its own prediction label instead of averaging it away. This approach could be reused for any subjective annotation, such as politeness or sarcasm, where how much people disagree is informative in itself.
  • I3. Hard non-humorous negatives: Non-humorous texts were drawn from sources that resemble jokes in form, such as trivia Q&A accounts, movie-quote accounts and advocacy groups that mention the same target words. News was avoided because its style is too easy to tell apart. Choosing negatives this way makes classifiers learn content rather than genre style.
  • I4. Upper bound for simple heuristics: The organizers measured how well a threshold rule would do if it had perfect knowledge of the humor scores. This shows how much of a subtask a trivial baseline could solve, which is a useful sanity check when designing new subtasks.
  • I5. Matched annotators and texts: Texts and raters were both taken from the US to reduce misunderstandings caused by cultural gaps. This is relevant to any culturally loaded annotation, for example Ukrainian humor or slang, where raters should share the background of the texts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations🔬The Alan Turing InstituteLondon · United Kingdom19 works · 3,608 citations🎓Universidad de la República de UruguayMontevideo · Uruguay4 works · 429 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Social Psychology · 129 works

Cite

Cite as J. A. Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez and Walid Magdy. 2021. SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense. Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.semeval-1.9

@inproceedings{meaney2021semeval,
  title = {{SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense}},
  author = {J. A. Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez and Walid Magdy},
  booktitle = {Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)},
  year = {2021},
  doi = {10.18653/v1/2021.semeval-1.9},
  url = {https://doi.org/10.18653/v1/2021.semeval-1.9}
}

References

50 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805. our card
  2. Kabil Essefar, Abdellah El Mekki, Abdelkader El Mahdaouy, NABIL El Mamoun, and Ismail Berrada. 2021. CS-UM6P at SemEval-2021 Task 7: Deep Multi-Task Learning Model for Detecting and Rating Humor and Offense. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics a… find
  3. Hani Al-Omari, Isra’a AbedulNabi, and Rehab Duwairi. 2021. DLJUST at SemEval-2021 Task 7: Hahackathon: Linking Humor and Offense Across Different Age Groups. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu… find
  4. Dalya Faraj and Malak Abdullah. 2021. SarcasmDet at SemEval-2021 Task 7: Detect Humor and Offensive based on Demographic Factors using RoBERTa Pre-trained Model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on… find
  5. Miriam Amin and Manuel Burghardt. 2020. A survey on approaches to computational humor generation. In Proceedings of the The 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 29–41, Online. International Committee on Comput… find
  6. Zhengyi Guan. 2021. Tsia at SemEval-2021 Task 7:Detecting and Rating Humor and Offense. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. find
  7. Salvatore Attardo. 2008. A Primer for the Linguistics of Humor. The Primer of Humor Research, 8:101– 156. find
  8. Emran Al Bashabsheh and Sanaa Abu Alasal. 2021. ES-JUST at SemEval-2021 Task 7: Detecting and Rating Humor and Offensive Text Using Deep Learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Languag… find
  9. Elisa Bassignana, Valerio Basile, and Viviana Patti. 2018. Hurtlex: A multilingual Lexicon of Words to Hurt. In 5th Italian Conference on Computational Linguistics, CLiC-it 2018, volume 2253, pages 1–6. CEUR-WS. our card
  10. Christos Baziotis, Nikos Pelekis, and Christos Doulk-eridis. 2017. DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 747–754, Vancouver, Can… our card
  11. Santiago Castro, Luis Chiruzzo, and Aiala Rosá. 2018. Overview of the HAHA Task: Humor Analysis Based on Human Annotation at IberEval 2018. In IberEval@ SEPLN, pages 187–194. find
  12. Luis Chiruzzo, Santiago Castro, Mathias Etcheverry, Diego Garat, Juan José Prada, and Aiala Rosá. 2019. Overview of HAHA at IberLEF 2019: Humor Analysis based on Human Annotation. In IberLEF@ SE-PLN, pages 132–144. find
  13. Akshay Gugnani, Brian Zylich, Gabriel Brookman, and Nicholas Samoray. 2021. Amherst685 at SemEval-2021 Task 7: Joint Modeling of Classification and Regression for Humor and Offense. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International… find
  14. Aishwarya Gupta, Avik Pal, Bholeshwar Khurana, Lakshay Tyagi, and Ashutosh Modi. 2021. Humor@IITK at SemEval-2021 Task 7: Large Language Models for Quantifying Humor and Offensiveness. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internation… find
  15. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. A… find
  16. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. arXiv preprint arXiv:2006.03654. our card
  17. Tim Highfield. 2015. Tweeted Joke Lifespans and Appropriated Punchlines: Practices around Topical Humor on Social Media. International Journal of Communication, 9:22. find
  18. Jennifer Hofmann, Tracey Platt, Chloe Lau, and Jorge Torres-Marı́n. 2020. Gender Differences in Humor-Related Traits, Humor Appreciation, Production, Comprehension,(Neural) Responses, Use, and Correlates: A Systematic Review. Current Psychology, pages 1–14. find
  19. J. A. Meaney. 2020. Crossing the line: Where do demographic variables fit into humor detection? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 176–181, Online. Association for Computational Linguistics. find
  20. Nabil Hossain, John Krumm, Michael Gamon, and Henry Kautz. 2020. SemEval-2020 Task 7: Assessing humor in Edited News Headlines. arXiv preprint arXiv:2008.00304. find
  21. Rada Mihalcea and Carlo Strapparava. 2005. Making Computers Laugh: Investigations in Automatic Humor Recognition. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 531–538. our card
  22. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. arXiv preprint arXiv:1801.06146. find
  23. Alexandros Karasakalidis, Dimitrios Effrosynidis, and Avi Arampatzis. 2021. DUTH at SemEval-2021 Task 7: Is Conventional Machine Learning for Humorous and Offensive Tasks enough in 2021? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internati… find
  24. Giselinde Kuipers. 2015. The Humor Divide: Class, Age and Humor Styles. In Good Humor, Bad Taste, pages 71–101. De Gruyter Mouton. find
  25. Roberto Labadie, Mariano Rodriguez, Reynier Ortega, and Paolo Rosso. 2021. Dual Transformer for Sentiment Analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. find
  26. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv preprint arXiv:1909.11942. find
  27. Renyuan Liu and Xiaobing Zhou. 2021. Grenzlinie at SemEval-2021 Task 7: HaHackathon Detecting and Rating Humor and Offense. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. find
  28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  29. Sharon Lockyer and Michael Pickering. 2005. Beyond a Joke: The Limits of Humour. Springer. find
  30. Jian Ma, ShuYi Xie, Jiang Lianxin, Ryan Stark, Mo Yang, and Jianping Shen. 2021. MagicPai at SemEval-2021 Task 7: Method for Detecting and Rating Humor Based on Multi Task Adversarial Training. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In… find
  31. Rada Mihalcea and Carlo Strapparava. 2006. Learning to Laugh (Automatically): Computational Models for Humor Recognition. Computational Intelligence, 22(2):126–142. find
  32. Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2016. Adversarial Training Methods for Semi-supervised Text Classification. arXiv preprint arXiv:1605.07725. our card
  33. Terufumi Morishita, Gaku Morio, Shota Horiguchi, Hiroaki Ozaki, and Toshinori Miyoshi. 2020. Hitachi at SemEval-2020 task 8: Simple but effective modality ensemble for meme emotion recognition. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1126–1134, Barcelona (online). Int… find
  34. Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9–14. our card
  35. Chao Pang, Xiaoran Fan, Weiyue Su, Xuyi Chen, Shuohuan Wang, Jiaxiang Liu, Xuan Ouyang, Shikun Feng, and Yu Sun. 2021. abcbpc at SemEval-2021 Task 7: ERNIE-based Multi-task Model for Detecting and Rating Humor and Offense. In Proceedings of the 59th Annual Meeting of the Association for Computationa… find
  36. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:282… find
  37. Peter Potash, Alexey Romanov, and Anna Rumshisky. 2017. SemEval-2017 Task 6: \#Hashtagwars: Learning a Sense of Humor. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 49–57. find
  38. Tathagata Raha, Ishan Sanjeev Upadhyay, Radhika Mamidi, and Vasudeva Varma. 2021. IIITH at SemEval-2021 Task 7: Leveraging Transformer-based Humourous and Offensive Text Detection Architectures using Lexical and Hurtlex Features along with Task Adaptive Pretraining. In Proceedings of the 59th Annual… find
  39. J. T. Rayz. 2017. In Pursuit of Human-Friendly Interaction with a Computational System: Computational Humor. In 2017 IEEE 15th International Symposium on Applied Machine Intelligence and Informatics (SAMI), pages 000015–000020. find
  40. Willibald Ruch. 2010. The Sense of Humor: Explorations of a Personality Characteristic, volume 3. Walter de Gruyter. find
  41. Mayukh Sharma, Ilanthenral Kandasamy, and Vasantha W B. 2021. YoungSheldon at SemEval-2021 Task 7: Fine-tuning Is All You Need. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. find
  42. Leandro Silva, Mainack Mondal, Denzil Correa, Fabrı́cio Benevenuto, and Ingmar Weber. 2016. Analyzing the Targets of Hate in Online Social Media. In Proceedings of the International AAAI Conference on Web and Social Media, volume 10. find
  43. Răzvan-Alexandru Smădu, Dumitru-Clementin Cercel, and Mihai Dascalu. 2021. UPB at SemEval-2021 Task 7: Adversarial Multi-Task Learning for Detecting and Rating Humour and Offence. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International… find
  44. Bingyan Song, Chunguang Pan, Shengguang Wang, and Zhipeng Luo. 2021. DeepBlueAI at SemEval-2021 Task 7: Detecting and Rating Humor and Offense with Stacking Diverse Language Model-Based Methods. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th I… find
  45. Christina A Sue and Tanya Golash-Boza. 2013. ‘It Was Only a Joke’: How Racial Humour Fuels Colour-Blind Ideologies in Mexico and Peru. Ethnic and Racial Studies, 36(10):1582–1598. find
  46. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: A Continual Pre-training Framework for Language Understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8968–8975. our card
  47. Julia Taylor and S Attardo. 2017. Computational Treatments of Humor. The Routledge Handbook of the Linguistics of Humor. New York: Routledge, pages 456–471. find
  48. Julia M Taylor and Lawrence J Mazlack. 2004. Computationally Recognizing Wordplay in Jokes. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 26. find
  49. Yubo Xie, Junze Li, and Pearl Pu. 2021. HumorHunter at SemEval-2021 Task 7: Humor and Offense Recognition with Disentangled Attention. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing… find
  50. Renxian Zhang and Naishi Liu. 2014. Recognizing Humor on Twitter. In Proceedings of the 23rd ACM international conference on conference on information and knowledge management, pages 889–898. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/meaney-2021-semeval-2021-task-7-hahackathon/